Architecture and Python use =========================== ACMAD Upload Helper combines a staged ETL pipeline with a ports-and-adapters architecture. Data moves from extraction, through transformation, to loading; dependencies point towards domain concepts and abstract ports rather than towards Excel, HTTP, or a particular user interface. .. mermaid:: :caption: Components, interfaces, and dependency direction classDiagram direction LR class UserInterfaces { <> CLI Desktop GUI } class Application { <> EtlPipeline Loaders Package orchestration } class Sources { <> Filesystem discovery Excel extraction Source models } class Transformers { <> Schema validation Canonical mapping } class Domain { <> Canonical records Provenance Structured issues } class Ports { <> Extractor Transformer Loader Repositories } class RestAdapters { <> ACMAD repositories OIDC and JSON API client } class ExternalSystems { <> Package files ACMAD REST API Identity provider } UserInterfaces --> Application : invokes UserInterfaces --> Sources : assembles Application --> Ports : depends on Application --> Domain : returns results and issues Sources ..|> Ports : implements Extractor Sources --> Domain : records provenance Sources --> ExternalSystems : reads package files Transformers ..|> Ports : implements Transformer Transformers --> Domain : creates canonical records RestAdapters ..|> Ports : implements repositories RestAdapters --> Domain : consumes canonical records RestAdapters --> ExternalSystems : HTTP and OIDC ETL stages ---------- Extraction ~~~~~~~~~~ The :mod:`acmad_uploader.sources` package is the boundary between package files and the rest of the application. It discovers relevant workbooks and motion-capture artefacts, reads vertical Excel forms, and produces source models that retain values together with file, worksheet, cell, and field locations. Extraction describes what was present; it does not decide how an API record should look or make network requests. Each source class satisfies the generic :class:`~acmad_uploader.ports.pipeline.Extractor` protocol. Consequently, the pipeline needs only an ``extract(source)`` operation and does not need to know whether values came from Excel, a different file format, or a test double. Transformation ~~~~~~~~~~~~~~ The :mod:`acmad_uploader.transformers` package converts extracted source models into canonical records from :mod:`acmad_uploader.domain`. This is where template-version checks, required-field validation, controlled-vocabulary mappings, numeric coercion, cross-workbook joins, and site consistency checks belong. Problems are returned as structured issues with the provenance captured during extraction. Transformers satisfy the :class:`~acmad_uploader.ports.pipeline.Transformer` protocol. They depend on source and domain values, but not on HTTP clients or destination-specific response shapes. A package can therefore be fully validated without authentication or access to the ACMAD service. Loading ~~~~~~~ Loaders in :mod:`acmad_uploader.application` apply canonical records to a destination. They select the appropriate repository operation, aggregate created, updated, unchanged, or skipped outcomes, and translate expected repository failures into pipeline issues. The package-wide orchestration also runs record types in dependency order—for example, patient before gait assessment and motion-capture data before biomechanics artefacts. Loaders depend on repository protocols from :mod:`acmad_uploader.ports`, not on the concrete REST implementation. The classes in :mod:`acmad_uploader.adapters.api` implement those protocols. They own OIDC authentication, HTTP requests, JSON response validation, natural-key lookup, destination payloads, and upsert behaviour. These details remain outside the domain and application layers. Cross-cutting components ------------------------ Domain ~~~~~~ The :mod:`acmad_uploader.domain` package contains the canonical record types, controlled vocabularies, source provenance, site overrides, and structured issues shared by the stages. Domain objects describe ACMAD concepts rather than spreadsheets or REST resources. They do not import Excel or HTTP libraries, which keeps the central data model usable by every interface and adapter. Application orchestration ~~~~~~~~~~~~~~~~~~~~~~~~~ :class:`~acmad_uploader.application.pipeline.EtlPipeline` coordinates the extract, transform, and optional load stages. It stops after a failed stage, keeps validation as the safe default, and returns each stage's result for inspection. Package orchestration composes the record-specific pipelines and applies their dependency order. The command-line and desktop interfaces are thin composition roots: they choose concrete sources, transformers, loaders, and adapters, then present progress and issues to the user. Ports and adapters ~~~~~~~~~~~~~~~~~~ The protocols in :mod:`acmad_uploader.ports` define the capabilities that the application requires. Extractor, transformer, and loader protocols describe the generic stage boundaries; record-specific repository protocols describe destination operations. The REST repositories are outbound adapters, while the command-line and desktop applications are inbound adapters that invoke the use cases. Loose coupling -------------- The architecture is loosely coupled because components share small, typed contracts instead of concrete infrastructure: * A source extractor can be replaced without changing transformation or loading, provided it returns the expected source model. * A repository can be replaced with an in-memory implementation, another API, or a test double without changing its loader. * Canonical domain records remain independent of workbook layouts and REST payloads, so change at either boundary is localised. * Validation and upload use the same extraction and transformation path, preventing a second set of validation rules from drifting out of sync. * Record types are vertical slices. Adding one normally means adding its own source, domain model, transformer, port, loader, and adapter rather than modifying the internals of unrelated slices. The result is not zero coupling: adjacent stages deliberately agree on typed values and protocols. It is controlled coupling, with infrastructure details kept at the edges and business meaning kept in the domain. This makes each stage independently testable and limits how far a workbook, API, or interface change can propagate. Running an ETL pipeline ----------------------- Components can be composed without the command-line interface. This example validates patient records locally; no repository is required because ``loader`` is omitted: .. code-block:: python from pathlib import Path from acmad_uploader.application.pipeline import EtlPipeline from acmad_uploader.sources.patient import PatientWorkbookExtractor from acmad_uploader.transformers.patient import PatientTransformer pipeline = EtlPipeline( extractor=PatientWorkbookExtractor(), transformer=PatientTransformer(), ) result = pipeline.run(Path("/path/to/package")) if result.succeeded: assert result.transformation is not None for patient in result.transformation.value or (): print(patient.site_patient_id) else: for issue in result.issues: print(issue.display()) To upload, supply the corresponding loader configured with an implementation of its repository protocol. The REST API classes in :mod:`acmad_uploader.adapters.api` are the built-in implementations. Results and issues ------------------ Each stage returns :class:`~acmad_uploader.application.results.StageResult` rather than using exceptions for expected data-quality failures. A :class:`~acmad_uploader.application.results.PipelineResult` retains extraction, transformation, and loading outcomes, and combines their structured :class:`~acmad_uploader.domain.issues.PipelineIssue` values. Source locations carry file, worksheet, cell, and field context where available.