Commerce Data Pipeline

Demo scenario, not a company architecture. Multiple sources use different formats and fail independently. The proposed pipeline separates collection, validation, and delivery so a failed source can be retried independently.

Example scope: implement collection jobs, queue retries, record normalization, duplicate detection, and a review queue for invalid records. Edit the scope to reflect what you actually built.

Example tradeoff: asynchronous jobs isolate source failures, but require idempotency and operational visibility. Invalid records are quarantined instead of silently dropped, adding review work in exchange for traceability.

Illustrative outcome: a consistent record format and a visible path for failed records to be reviewed and retried. Throughput, accuracy, and savings are intentionally not asserted in this demo.