Medallion Architecture — Data Platform Architecture Pattern

Problem

Dumping all data into a single lake layer with no quality tiers turns it into a 'data swamp' of unreliable, hard-to-query information. Analysts lose trust, pipelines break on schema drift, and there is no clean raw history to replay or audit when errors surface downstream.

Solution

Structure the data lake into three logical zones: Bronze (raw, append-only history), Silver (cleansed, conformed, filtered), and Gold (business-level aggregations optimized for analytics).

Cloud Paradigm

Multi-Tier Data Refinement (raw/cleansed/curated zones)
Immutable Append-Only Ingestion
Schema-on-Read at Landing, Schema-on-Write Downstream
Progressive Data Quality Enforcement
Replayable Lineage-Preserving Reprocessing
Separation of Storage and Compute

Solution Flow

Source systems (transactional databases, SaaS APIs, IoT streams, flat files) emit records that are ingested as-is with no transformation.
The ingestion service lands every event into the Bronze layer as an append-only, immutable copy — schema-on-read, partitioned by ingest date, with full lineage and source metadata preserved.
A Silver transformation job reads Bronze incrementally, deduplicates, enforces schema, casts types, resolves late-arriving data, and conforms entities against reference/master data into cleansed, queryable tables.
The Gold aggregation job joins conformed Silver tables into business-level marts — star schemas, pre-aggregated KPIs, and feature tables tuned for read performance.
BI tools, ML pipelines, and analysts consume Gold (and occasionally Silver for exploratory work), while Bronze remains the replayable system of record.
If logic changes or a bug is found, engineers reprocess downstream layers from Bronze without re-ingesting from sources.

When to Use

You ingest heterogeneous sources at varying quality and need a single reliable analytics platform.
Auditability and full data replay from raw history are requirements.
Multiple consumer types (BI, data science, reverse ETL) need different fidelity levels.
You run on a lakehouse engine (Delta Lake, Iceberg, Hudi) supporting ACID and time travel.

When NOT to Use

Small, single-source datasets where three layers add needless overhead.
Pure low-latency operational serving that a transactional store handles better.
Real-time sub-second use cases where batch layer hops introduce unacceptable lag.

Trade-offs

Reliable, replayable data vs. 3x storage and duplicated copies of each dataset.
Clear quality contracts per layer vs. more pipelines and orchestration to maintain.
Flexible reprocessing from raw vs. added end-to-end latency across hops.
Decoupled producers and consumers vs. governance discipline to prevent Silver/Gold sprawl.

Real-World Example

A national grocery retailer syncing point-of-sale, e-commerce, loyalty, and supplier feeds lands all events into Bronze on Delta Lake. Silver deduplicates transactions, standardizes store IDs against master data, and repairs late-arriving refunds. Gold produces daily sales-by-region marts and a churn feature table. When a currency-rounding bug surfaces, engineers correct the Silver logic and reprocess only the affected partitions from Bronze — dashboards refresh accurately without re-pulling anything from source systems.

Additional Details

Idempotent reprocessing: Silver/Gold jobs must be idempotent and keyed for safe replay — use MERGE/upsert on natural keys rather than blind appends, or a partition-overwrite strategy, so reprocessing a Bronze range doesn't duplicate or double-count records.
Bronze schema-on-read drift: Since Bronze captures raw payloads, store source schema/version metadata alongside each record; enforce schema at the Silver boundary and route rejects to a quarantine table rather than failing the whole batch.
Table maintenance: ACID lakehouse formats accumulate small files and stale snapshots — schedule compaction/OPTIMIZE, Z-ordering or clustering on common filter keys, and vacuum/expire-snapshots. Set retention deliberately, as vacuuming too aggressively breaks time-travel replay.
Cost drivers: Bronze storage grows unbounded (append-only), so tier or expire cold partitions; compute cost is dominated by full-table Silver rebuilds — prefer incremental change-data processing and watermarks over reprocessing everything.
Late/out-of-order data: Define explicit watermarks and reprocessing windows for late-arriving updates; corrections landing after a Gold partition is published require a documented backfill trigger downstream.
Lineage and observability: Track per-layer row counts, dedup/reject rates, schema-violation counts, and freshness lag per hop. Emit batch/commit IDs so a Gold anomaly can be traced to the exact Bronze ingest and transformation version.

Security Controls

Layer-scoped access control: Grant analysts read-only Gold access while restricting raw Bronze to data-engineering service principals.
Encryption at rest and in transit: Enforce storage-account encryption and TLS on all ingestion and query paths across every layer.
PII masking in Silver: Apply column-level masking, tokenization, or dynamic views so sensitive fields never propagate unprotected to Gold.
Immutable Bronze retention: Use write-once/append-only policies and object-lock to guarantee an auditable, tamper-evident raw history.
Lineage and audit logging: Capture end-to-end lineage and query audit trails to satisfy compliance and trace data provenance.
Data quality gates: Fail pipelines on schema or constraint violations before promoting data from Bronze to Silver to Gold.