Architecture and Python use

ACMAD Upload Helper combines a staged ETL pipeline with a ports-and-adapters architecture. Data moves from extraction, through transformation, to loading; dependencies point towards domain concepts and abstract ports rather than towards Excel, HTTP, or a particular user interface.

        classDiagram
    direction LR

    class UserInterfaces {
        <<component>>
        CLI
        Desktop GUI
    }
    class Application {
        <<component>>
        EtlPipeline
        Loaders
        Package orchestration
    }
    class Sources {
        <<component>>
        Filesystem discovery
        Excel extraction
        Source models
    }
    class Transformers {
        <<component>>
        Schema validation
        Canonical mapping
    }
    class Domain {
        <<component>>
        Canonical records
        Provenance
        Structured issues
    }
    class Ports {
        <<interfaces>>
        Extractor
        Transformer
        ClinicalSourceProvider
        Loader
        Repositories
    }
    class RestAdapters {
        <<component>>
        ACMAD repositories
        OIDC and JSON API client
    }
    class ExternalSystems {
        <<external>>
        Package files
        ACMAD REST API
        Identity provider
    }

    UserInterfaces --> Application : invokes
    UserInterfaces --> Sources : assembles
    Application --> Ports : depends on
    Application --> Domain : returns results and issues
    Sources ..|> Ports : implements Extractor
    Sources --> Domain : records provenance
    Sources --> ExternalSystems : reads package files
    Transformers ..|> Ports : implements Transformer
    Transformers --> Domain : creates canonical records
    RestAdapters ..|> Ports : implements repositories
    RestAdapters --> Domain : consumes canonical records
    RestAdapters --> ExternalSystems : HTTP and OIDC

Components, interfaces, and dependency direction

ETL stages

Extraction

The acmad_uploader.sources package is the boundary between package files and the rest of the application. It discovers relevant workbooks and motion-capture artefacts, reads vertical Excel forms, and produces source models that retain values together with file, worksheet, cell, and field locations. Extraction describes what was present; it does not decide how an API record should look or make network requests.

Each source class satisfies the generic Extractor protocol. Consequently, the pipeline needs only an extract(source) operation and does not need to know whether values came from Excel, a different file format, or a test double.

Transformation

The acmad_uploader.transformers package converts extracted source models into canonical records from acmad_uploader.domain. This is where template-version checks, required-field validation, controlled-vocabulary mappings, numeric coercion, cross-workbook joins, and site consistency checks belong. Problems are returned as structured issues with the provenance captured during extraction.

Transformers satisfy the Transformer protocol. They depend on source and domain values, but not on HTTP clients or destination-specific response shapes. A package can therefore be fully validated without authentication or access to the ACMAD service.

Loading

Loaders in acmad_uploader.application apply canonical records to a destination. They select the appropriate repository operation, aggregate created, updated, unchanged, or skipped outcomes, and translate expected repository failures into pipeline issues. The package-wide orchestration also runs record types in dependency order—for example, patient before gait assessment and motion-capture data before biomechanics artefacts.

Loaders depend on repository protocols from acmad_uploader.ports, not on the concrete REST implementation. The classes in acmad_uploader.adapters.api implement those protocols. They own OIDC authentication, HTTP requests, JSON response validation, natural-key lookup, destination payloads, and upsert behaviour. These details remain outside the domain and application layers.

Cross-cutting components

Domain

The acmad_uploader.domain package contains the canonical record types, controlled vocabularies, source provenance, site overrides, and structured issues shared by the stages. Domain objects describe ACMAD concepts rather than spreadsheets or REST resources. They do not import Excel or HTTP libraries, which keeps the central data model usable by every interface and adapter.

Application orchestration

EtlPipeline coordinates the extract, transform, and optional load stages. It stops after a failed stage, keeps validation as the safe default, and returns each stage’s result for inspection. Package orchestration composes the record-specific pipelines and applies their dependency order. A ClinicalSourceProvider supplies the source value, extractor, and transformer for every slice. The built-in ExcelPackageSourceProvider preserves the workbook and filesystem behaviour, while an institution-specific provider can supply database row models and mappings without changing the loaders or REST adapters. The command-line and desktop interfaces are thin composition roots: they choose concrete sources, transformers, loaders, and adapters, then present progress and issues to the user.

Ports and adapters

The protocols in acmad_uploader.ports define the capabilities that the application requires. Extractor, transformer, and loader protocols describe the generic stage boundaries; the source-provider protocol describes package-wide source composition; and record-specific repository protocols describe destination operations. The REST repositories are outbound adapters, while the command-line and desktop applications are inbound adapters that invoke the use cases.

Alternative source providers

An alternative provider binds its extractor and transformer together for every name in PACKAGE_SLICE_NAMES. Package orchestration validates the set, restores the core dependency order, and attaches the existing loaders for uploads. A provider may therefore use a connection or query selection as its source value and institution-specific row objects as its extracted value; it does not need to manufacture VerticalForm instances.

from acmad_uploader.application.package_upload import run_provider_validation
from acmad_uploader.ports.source_provider import SliceBinding

class HospitalDatabaseProvider:
    def __init__(self, connection_factory):
        self.connection_factory = connection_factory

    def bindings(self, context):
        return (
            SliceBinding(
                name="patient",
                source=self.connection_factory,
                extractor=PatientDatabaseExtractor(),
                transformer=PatientDatabaseTransformer(
                    site_override=context.site_override,
                ),
            ),
            # Bind the remaining package slices here.
        )

result = run_provider_validation(HospitalDatabaseProvider(connect))

Providers are ordinary trusted Python components installed and composed by an institution-specific application. Configuration should select from known providers rather than contain arbitrary import paths. Credentials belong in an environment variable, operating-system credential store, or another institution-managed secret mechanism rather than the package YAML file. See Customising the ETL pipeline for a complete implementation and deployment guide.

Loose coupling

The architecture is loosely coupled because components share small, typed contracts instead of concrete infrastructure:

A complete source provider can be replaced without changing package orchestration or loading. Its extractors and transformers may use entirely different source models.
A repository can be replaced with an in-memory implementation, another API, or a test double without changing its loader.
Canonical domain records remain independent of workbook layouts and REST payloads, so change at either boundary is localised.
Validation and upload use the same extraction and transformation path, preventing a second set of validation rules from drifting out of sync.
Record types are vertical slices. Adding one normally means adding its own source, domain model, transformer, port, loader, and adapter rather than modifying the internals of unrelated slices.

The result is not zero coupling: adjacent stages deliberately agree on typed values and protocols. It is controlled coupling, with infrastructure details kept at the edges and business meaning kept in the domain. This makes each stage independently testable and limits how far a workbook, API, or interface change can propagate.

Running an ETL pipeline

Components can be composed without the command-line interface. This example validates patient records locally; no repository is required because loader is omitted:

from pathlib import Path

from acmad_uploader.application.pipeline import EtlPipeline
from acmad_uploader.sources.patient import PatientWorkbookExtractor
from acmad_uploader.transformers.patient import PatientTransformer

pipeline = EtlPipeline(
    extractor=PatientWorkbookExtractor(),
    transformer=PatientTransformer(),
)
result = pipeline.run(Path("/path/to/package"))

if result.succeeded:
    assert result.transformation is not None
    for patient in result.transformation.value or ():
        print(patient.site_patient_id)
else:
    for issue in result.issues:
        print(issue.display())

To upload, supply the corresponding loader configured with an implementation of its repository protocol. The REST API classes in acmad_uploader.adapters.api are the built-in implementations.

Results and issues

Each stage returns StageResult rather than using exceptions for expected data-quality failures. A PipelineResult retains extraction, transformation, and loading outcomes, and combines their structured PipelineIssue values. Source locations carry file, worksheet, and cell context for packages, or resource, record, and field context for external sources such as database relations.