Data PlatformReference ArchitectureData EngineeringCloudData Governance

Data Platform Reference Architecture

The data platform reference architecture covers ingestion, storage, processing and consumption, with governance, security, DataOps and cloud examples.

David TirabassiUpdated
Enterprise/Solution Architect · Almost 30 years' experience, specialising in integration · TOGAF 9.1 certifiedLinkedInGitHub

Data Platform Component Model

Hover over components to highlight, and click any item to see capability summaries and cloud vendor services.

Data Consumers
Data Consumption Layer
Data Processing Layer
Data Storage Layer
Data Ingestion Layer
Infrastructure Layer
Data Sources
Governance
Security
Supporting (DataOps)

From architecture to patterns

These components have a pattern in the library that shows how they are built and where they fit.

Components of the architecture and the patterns that implement them
ComponentPatterns
Data Queries
Data Visualization
Model Serving & Advanced Analytics
Data APIs
Stream Processing
Data Lake
Enterprise Data Warehouse (EDW)
Purpose-Built Databases
Vector Store
Stream & Messaging Ingestion
API Ingestion
Data Masking & Encryption
Master & Reference Data Management

Overview

A data platform is a governed, shared foundation that acquires data from operational systems, applications, files and event streams, lands it once, processes it, and serves it to analytical and decision-support workloads. It supports reporting and analytics, operational analytics, and machine learning and AI from one governed source of data.

This reference architecture specifies the components, what each is responsible for, and the relationships between them. It also defines the governance, security and operational controls that apply across all components. The architecture is vendor-neutral. Each capability is mapped to example services on AWS, Azure and Google Cloud and linked to the implementation patterns that realise it.

Scope

In scope. The platform is responsible for:

  • Acquisition. Ingesting data from source systems through the integration patterns.
  • Governance and security. Cataloguing, quality, lineage, access control and protection of data across its lifecycle.
  • Storage and processing. Retaining, transforming and curating data for analytical use.
  • Serving. Publishing data products, features, models and scores to consumers and decision systems.

Out of scope.

  • Not an integration platform. Moving business transactions between operational systems is the responsibility of the Enterprise Integration Platform reference architecture and the integration patterns. The data platform receives data from, and publishes data back through, those patterns. It does not replace them.
  • Not an application stack. Operational applications own their transactional databases. Platform stores support analytical, reporting and decision workloads only. An application that needs its own database has one in its own architecture, fed from the platform.
  • Not only a reporting warehouse. Reporting is one consumer of platform data, alongside models, operational analytics, and data published through APIs and event streams.

Why the data goes in

TimescaleTypical decisionWhat the platform supplies
Strategic (months)Where to add capacity, which contracts to renewGoverned history and reporting
Tactical (days)Staffing for next week, which claims to reviewForecasts, trained models and data marts
Operational (seconds to minutes)Which route to take, whether to approve a paymentFeatures and scores, published to the system that makes the decision

In every case the platform informs the decision and an operational system acts on it.

How data gets in and out

Data reaches the platform, and leaves it, through integration patterns. Each integration pattern comes in three versions, one for each boundary the data crosses: cloud, external and internal. The Enterprise Integration Platform reference architecture shows how they fit together, and the integration patterns library lists them all.

Data in

PatternCloudExternalInternal
Change data capturecloudexternalinternal
Event streamingcloudexternalinternal
Secure file transfercloudexternalinternal

Webhooks have a single pattern and no boundary versions: Webhooks (Event-Push Ingestion).

Data out

PatternCloudExternalInternal
API managementcloudexternalinternal
Event streamingcloudexternalinternal
Master data management, publishing changes back to source systemscloudexternalinternal

Processing and serving inside the platform

Component Model Layers

1. Data Sources Layer

This foundational layer represents the diverse origins of data that exist before entering the platform. These sources generate the raw information that feeds the enterprise.

COMPONENTDESCRIPTIONEXAMPLES
Operational DatabasesCore transactional relational and document engines backing the primary operational products of the enterprise.Amazon RDS / Aurora (AWS); Azure SQL / PostgreSQL (Azure); Cloud SQL / Cloud Spanner (GCP); PostgreSQL, MySQL, MongoDB (Open Source / Other)
ApplicationsBusiness enterprise software suites, cloud SaaS applications, CRM, ERP, and general line-of-business software platforms.SAP on AWS (AWS); SAP on Azure (Azure); SAP on Google Cloud (GCP); Odoo, ERPNext (Open Source / Other)
StreamsUnbounded telemetry feeds, clickstream trails, and internet-connected devices transmitting active continuous sensor measurements.AWS IoT Core (AWS); Azure IoT Hub (Azure); Pub/Sub with a partner IoT platform (e.g. ClearBlade) (GCP); Eclipse Mosquitto, EMQX (Open Source / Other)
Structured DataHighly structured and tabular data formatted files sent by internal teams or external vendors (such as monthly CSV summaries).Amazon S3 Uploads (AWS); Blob Storage Inbound (Azure); GCS Bucket Uploads (GCP); SFTP, Samba (Open Source / Other)
Unstructured DataBinary, heavy, and media files containing rich operational data lacking pre-defined relational structures (e.g. PDFs, images).Amazon S3 Object Store (AWS); Azure Blob Storage (Azure); Google Cloud Storage (GCP); MinIO Object Storage (Open Source / Other)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

2. Infrastructure Layer

The underlying physical or virtualized hardware and cloud services that host the data platform. This layer provides the foundational compute, storage, and networking capabilities on which all other layers depend.

COMPONENTDESCRIPTIONEXAMPLES
Cloud InfrastructurePrimary scalable virtual machines, bare metal components, and block storage platforms serving as the core host of the platform.Amazon EC2 & EBS (AWS); Azure VMs & Managed Disk (Azure); Compute Engine & Persistent Disk (GCP); OpenStack (Open Source / Other)
Container OrchestrationSchedules, scales, and manages containerized applications and transformation tasks consistently across hybrid environments.Amazon EKS (AWS); Azure Kubernetes Service (AKS) (Azure); Google Kubernetes Engine (GKE) (GCP); Kubernetes, Nomad (Open Source / Other)
Private NetworkingEnsures isolation of cloud workloads using virtual firewalls, private end-points, and secure virtual private tunnels to block exfiltration.AWS VPC & PrivateLink (AWS); Azure Virtual Network & Private Link (Azure); Google Cloud VPC & PSC (GCP); Cilium, Calico (Open Source / Other)
On-Premises / HybridSecure on-premises physical data centres linked via high-speed dedicated tunnels for low-latency or strict data residency systems.AWS Outposts / Direct Connect (AWS); Azure Local / ExpressRoute (Azure); Google Distributed Cloud / Interconnect (GCP); VMware, Nutanix (Open Source / Other)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

3. Data Ingestion Layer

This layer extracts data from diverse source systems, acting as the primary consumer of the Enterprise Integration Platform for event-driven and API-based feeds. Note on Data Contracts: Across all ingestion methods, this layer acts as the enforcement point for Data Contracts—ensuring incoming data adheres to agreed-upon schemas and SLAs before it enters the platform.

COMPONENTDESCRIPTIONEXAMPLES
Database IngestionSafely reads transactional logs in real-time (Change Data Capture) or executes batches without impacting database performance.AWS Database Migration Service (AWS); Azure Database Migration Service (Azure); Google Datastream (GCP); Debezium, Fivetran, Airbyte (Open Source / Other)
Stream & Messaging IngestionDirectly ingests, aggregates, and pipes high-velocity messaging topics and streaming data sources straight into storage.Amazon Kinesis Data Streams (AWS); Azure Event Hubs (Azure); Google Pub/Sub (GCP); Apache Kafka, Logstash (Open Source / Other)
API IngestionConnects securely to third-party SaaS tools and web endpoints to pull or push data assets via REST/GraphQL or webhooks.Amazon AppFlow (AWS); Azure Logic Apps (Azure); Google Cloud Data Fusion (GCP); Meltano, Singer (Open Source / Other)
File IngestionManaged secure data transfer services to safely receive, move, and store external file exports, flat archives, and XMLs.AWS Transfer Family (AWS); Azure Data Factory (Azure); GCP Storage Transfer (GCP); rclone, Apache NiFi (Open Source / Other)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

4. Data Storage Layer

This layer provides scalable, decoupled storage for all data types. It supports everything from raw object storage to highly structured analytical models and specialized operational formats.

COMPONENTDESCRIPTIONEXAMPLES
Data LakeHighly cost-effective object storage holding all historical records. Modern architectures overlay open table formats (Iceberg, Delta) for ACID reliability.Amazon S3 + Apache Iceberg (AWS); ADLS Gen2 + Delta Lake (Azure); Google Cloud Storage + Iceberg (GCP); MinIO + Apache Iceberg (Open Source / Other)
Enterprise Data Warehouse (EDW)Structured analytical relational storage optimized for complex reporting queries, running highly performant mass analytical schemas.Amazon Redshift (AWS); Azure Synapse / Fabric (Azure); Google BigQuery (GCP); Snowflake, ClickHouse (Open Source / Other)
Streaming BrokerDistributed, immutable commit log with configurable retention (Kafka-style) that durably stores high-velocity event streams, enabling replay, decoupled consumers, and multi-subscriber fan-out.Amazon MSK (AWS); Azure Event Hubs (Azure); Google Pub/Sub (GCP); Apache Kafka, Redpanda (Open Source / Other)
Purpose-Built DatabasesSpecialized non-relational database services optimized for document, graph, key-value, or low-latency lookup needs. These stores serve analytical and decision workloads, and applications keep their own databases.Amazon Neptune / DocumentDB (AWS); Azure Cosmos DB (Azure); Google Firestore / Spanner (GCP); MongoDB, Neo4j, PostgreSQL (Open Source / Other)
Vector StoreSpecialized databases designed to store, index, and query high-dimensional vector embeddings—crucial for modern semantic search, RAG, and generative AI pipelines.Amazon OpenSearch / RDS pgvector (AWS); Azure Cosmos DB (Vector Search) / AI Search (Azure); Vector Search / Cloud SQL pgvector (GCP); pgvector, Qdrant, Milvus, Chroma (Open Source / Other)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

5. Data Processing Layer

This layer encompasses the compute engines responsible for transforming, cleaning, enriching, and modeling the data. It also includes the heavy computational lifting required for artificial intelligence.

COMPONENTDESCRIPTIONEXAMPLES
Distributed Batch ProcessingScalable parallel compute clusters executing scheduled transformations on vast historical datasets.Amazon EMR, AWS Glue (AWS); Azure Databricks, Synapse Spark (Azure); Google Managed Service for Apache Spark (GCP); Apache Spark, dbt (Open Source / Other)
Stream ProcessingReal-time stream computations transforming unbounded data flows instantly for alerting and rapid-response dashboarding.Amazon Managed Service for Apache Flink (AWS); Azure Stream Analytics (Azure); Google Dataflow (GCP); Apache Flink, Spark Streaming (Open Source / Other)
ML Model TrainingHigh-performance GPU-backed compute used to develop, train, evaluate, and version machine learning and deep learning models — distinct from inference, which is served via the Consumption layer.Amazon SageMaker (AWS); Azure Machine Learning (Azure); Agent Platform Managed Training (GCP); Kubeflow, PyTorch (Open Source / Other)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

6. Data Consumption Layer

This layer exposes the processed data and trained models to the end consumers. Note on Data Contracts: Interfaces exposed here (like APIs and outbound feeds) are governed by outbound Data Contracts, guaranteeing reliability for downstream consumers.

COMPONENTDESCRIPTIONEXAMPLES
Data QueriesAd-hoc, federated SQL query engines that allow interactive querying directly on the data lake without moving it to a warehouse.Amazon Athena (AWS); Synapse Serverless (Azure); BigQuery (External) (GCP); Trino, Presto (Open Source / Other)
Semantic / Metrics LayerOne definition of each business metric, used by dashboards, models and AI agents.Metric definitions held in a BI model or a standalone semantic service
Data VisualizationBusiness Intelligence tools used to construct dynamic, interactive dashboards and scheduled, paginated reports.Amazon Quick Sight (AWS); Microsoft Power BI (Azure); Google Looker / Data Studio (GCP); Apache Superset, Metabase (Open Source / Other)
Model Serving & Advanced AnalyticsHosted, highly scalable inference endpoints serving low-latency predictions and generative model results to client systems, alongside complex statistical analysis and exploratory data science workloads.SageMaker Endpoints (AWS); Azure ML Endpoints (Azure); Agent Platform Endpoints (GCP); BentoML, KServe (Open Source / Other)
Data Activation / Reverse ETLPushes curated analytical profiles (e.g. customer churn risk scores) back to operational business applications through integration patterns, so the system that acts on the decision receives it.Amazon AppFlow (AWS); Azure Data Factory (Azure); BigQuery EXPORT DATA (Pub/Sub, Spanner, Bigtable) (GCP); Hightouch, Census (Open Source / Other)
Operational Data Stores (ODS)Read-optimized serving stores that cache curated, pre-aggregated data products for low-latency retrieval by the operational systems that act on a decision, bridging the analytical platform and those systems. These stores serve analytical and decision workloads, and applications keep their own databases.Amazon DynamoDB / ElastiCache (AWS); Azure Managed Redis (Azure); Google Cloud Bigtable (GCP); Apache Cassandra, Redis (Open Source / Other)
Data APIsExposes secure GraphQL or REST APIs on top of data warehouses or operational caches to grant programmatic access to application developers.Amazon API Gateway (AWS); Azure API Management (Azure); Google Apigee (GCP); Hasura, PostgREST (Open Source / Other)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

7. Data Consumers Layer

The topmost layer, representing the end-users and systems that derive business value from the data platform.

COMPONENTDESCRIPTIONEXAMPLES
Business UsersAnalysts, executives, and operational business staff relying on visual dashboards, metrics, and BI reports for daily strategic decisions.Amazon Quick Sight (AWS); Microsoft Power BI (Azure); Google Looker / Data Studio (GCP); Apache Superset (Open Source / Other)
Data ScientistsPower users building machine learning models, conducting advanced predictive analytics, and running exploratory data experiments.SageMaker Notebooks (AWS); Azure ML Studio (Azure); Agent Platform Workbench (GCP); JupyterHub, MLflow (Open Source / Other)
ApplicationsInternal or custom line-of-business applications consuming operational or analytical data products to drive workflows.Operational systems (receive data through integration patterns)
External 3rd PartiesThird-party clients, external vendors, or industry partners consuming platform data safely via secure outbound B2B feeds or API gateways.AWS Data Exchange (AWS); Azure Data Share (Azure); Analytics Hub (GCP); Delta Sharing (Open Source / Other)
Automated SystemsAutonomous systems, enterprise robotic process automation (RPA), and automated algorithms consuming insights with no human in the loop.AWS Step Functions (AWS); Azure Logic Apps (Azure); GCP Workflows (GCP); Node-RED, temporal.io (Open Source / Other)
Technical OperationsData engineers, platform support teams, and SRE professionals monitoring pipeline stability, data quality alerts, and costs.AWS CloudWatch (AWS); Azure Monitor (Azure); Google Cloud Observability (GCP); Grafana, Prometheus (Open Source / Other)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.


Integration carries data in and out of the platform

Data reaches the platform through integration patterns, and results leave it the same way. The platform lands the data once and governs it. Integration moves it in and out.

Diagram. Data Sources (databases, event streams, files and SaaS applications) feed integration in (change data capture, event streaming, file transfer and webhooks). Integration in feeds the data platform: Data Ingestion, where contracts are checked, then Data Storage, Data Processing and Data Consumption, with Governance, Security and Supporting (DataOps) alongside them. Integration out (API management and event streaming) carries results to Data Consumers: people, models and operational systems.

The diagram is wide, so open it at full size to read the labels.


Cross-Cutting Layers

These layers span horizontally across the platform, ensuring data remains secure, compliant, understandable, and operations run smoothly.

Data Governance Layer

COMPONENTDESCRIPTIONEXAMPLES
Data Discovery & MetadataA searchable database of data schemas, terminology definitions, and ratings allowing users to find data fast.AWS Glue Data Catalog (AWS); Microsoft Purview (Azure); Google Knowledge Catalog (GCP); Amundsen, DataHub, Apache Atlas (Open Source / Other)
Data QualityContinuously asserts quality parameters (freshness, completeness, shape) on active pipelines and alerts on deviation.AWS Glue Data Quality (AWS); Purview Data Quality (Azure); Google Knowledge Catalog (auto data quality) (GCP); Great Expectations, Soda, Deequ (Open Source / Other)
Data LineageDraws visual execution pathways of data from original source, through processing transformations, to visualization dashboards.Amazon SageMaker Catalog (data lineage) (AWS); Microsoft Purview Lineage (Azure); Google Knowledge Catalog (data lineage) (GCP); OpenLineage, DataHub (Open Source / Other)
Master & Reference Data ManagementReconciles and creates a single golden master record for core business entities (like 'Customer' or 'Product').AWS Entity Resolution / partner MDM (AWS); Profisee on Azure (Azure); Partner MDM (e.g. Reltio, Semarchy) (GCP); TIBCO EBX, Talend MDM (Open Source / Other)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

Data Security Layer

COMPONENTDESCRIPTIONEXAMPLES
Access ControlEnforces row-level, column-level, or cell-level access constraints on queries depending on corporate role clearance.AWS Lake Formation, IAM (AWS); Microsoft Purview Access (Azure); BigQuery Policy Tags, IAM (GCP); Apache Ranger, Immuta (Open Source / Other)
Data Masking & EncryptionObfuscates sensitive parameters like credit card numbers dynamically and encrypts data both in transit and at rest.AWS KMS, Redshift Masking (AWS); Azure Key Vault, SQL Masking (Azure); Google Cloud KMS, Sensitive Data Protection (GCP); HashiCorp Vault, Apache Ranger (Open Source / Other)
Secrets ManagementCentral repository holding, rotating, and tracking programmatic access credentials and server connection strings safely.AWS Secrets Manager (AWS); Azure Key Vault Secrets (Azure); Google Secret Manager (GCP); HashiCorp Vault (Open Source / Other)
Data Classification & Sensitive Data DiscoveryScans new data objects to identify and tag PII (Personally Identifiable Information), health tags, or banking numbers.Amazon Macie (AWS); Microsoft Purview Information Protection (Azure); Google Cloud Sensitive Data Protection (GCP); Apache Ranger Classification (Open Source / Other)
Audit LoggingTamper-evident system capture documenting every single query, configuration alteration, and file download action.AWS CloudTrail (AWS); Azure Monitor Logs (Azure); Google Cloud Audit Logs (GCP); Elasticsearch, OpenSearch (Open Source / Other)
Identity & Access Management (IAM)Federates user profiles, handles identity authentication, and issues safe machine credential tokens to control who and what can access platform resources.AWS IAM & Identity Center (AWS); Microsoft Entra ID (Azure); Google Cloud IAM (GCP); Keycloak, Okta (Open Source / Other)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

Supporting Services (DataOps) Layer

COMPONENTDESCRIPTIONEXAMPLES
Data Orchestration & Workflow ManagementPipes execution dependency commands across tasks, ensuring batches complete sequentially and retry on failure.Amazon MWAA (Airflow) (AWS); Azure Data Factory (Azure); Google Managed Service for Apache Airflow (GCP); Apache Airflow, Prefect, Dagster (Open Source / Other)
Monitoring & Observability (incl. FinOps)Tracks resource workloads, queries execution cost metrics, system crashes, and platform performance latency in real-time.Amazon CloudWatch & Cost Explorer (AWS); Azure Monitor & Cost Management (Azure); Google Cloud Observability & FinOps (GCP); Grafana, Prometheus, Datadog (Open Source / Other)
CI/CD & Source ControlAutomates testing code integrations, model versioning, and environment container migrations directly from Git.AWS CodePipeline (AWS); Azure Pipelines (Azure); Google Cloud Build (GCP); GitHub Actions, GitLab CI (Open Source / Other)
Infrastructure as Code (IaC)Represents data warehouses and infrastructure configurations as code structures, allowing rapid environment duplication.AWS CloudFormation (AWS); Azure Bicep (Azure); GCP Infrastructure Manager (Terraform) (GCP); Terraform, OpenTofu, Pulumi (Open Source / Other)
Data Modeling ToolsVisual and programmatic systems to map relational schemas, relational keys, warehouse stars, and data vault architectures.AWS Glue Schema Registry (AWS); Synapse Database Designer (Azure); Dataform (GCP); dbdiagram.io, SqlDBM, Erwin (Open Source / Other)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

Examples

Four examples show data informing a decision that a person or an operational system then acts on. They are not equally well sourced, so each one says what kind of source it rests on.

Illustrative Scenario

A hypothetical scenario showing how the pattern applies. Not a client case study.

Logistics routing. Flooding closes the main route of a regional freight carrier. Traffic and weather feeds, depot records and shipment data already sit in the platform, and a delay model has been trained on past disruptions. Within minutes the model estimates that one alternative route adds about an hour to a delivery and another adds about five hours. The platform publishes those estimates, with the shipments each one affects, to the routing system. The routing system weighs them against delivery windows and vehicle availability, chooses a route, and tells the drivers. The platform does not reroute anything. A dispatcher can still override the choice, and the platform has no say in that either. Afterwards the actual delays go back into the platform, so the estimates can be compared with what happened and the model can be checked and retrained. The decision about the route stays with the operational system and the people who run it. The platform's part is to have the data ready, to turn it into estimates quickly, and to keep a record of what it published and what followed.

Benefit fraud screening: third-party reporting

Sources: the National Audit Office report of 22 October 2025, which is the primary source, and The Register's report of 27 October 2025 on it.

The NAO reports that, since May 2022, the UK Department for Work and Pensions has used a machine learning model to flag potentially fraudulent Universal Credit advance claims. The model refers the claims it assesses as high risk to a caseworker, and department staff, not the model, decide whether the advance is paid. The NAO reports the department's estimate that the model saved about £4.4 million from 2022-23 to 2024-25. It also reports fairness findings: the department's published analysis showed older claimants and non-UK nationals being referred for review more often than expected, and the department could analyse only one of the nine protected characteristics (age) because it had limited data. The NAO adds that the department's IT systems are not fully integrated and that a lack of common data standards makes fraud and error harder to identify, and The Register reports that this limits how far the work can scale. The sources do not describe the department's data architecture, so the example shows the decision pattern, not a platform design.

Card fraud scoring: vendor reference design

Source: a Databricks engineering blog post, 19 May 2026. It is the vendor's own design for a reference implementation, not a customer case.

In the design, card transactions arrive as a stream. The pipeline computes features for each card, such as how many transactions it made in the last minute, and keeps them in a separate low-latency store. A model scores each transaction, the score routes it as approved, flagged for review, or blocked, and the decision is published to topics that downstream systems act on. The example shows the split this page describes: the platform computes the features and trains the model, and a store built for fast reads lets the decision be made at speed.

Utility storm planning: vendor blog, self-reported

Source: a Databricks blog post, 18 February 2025, about Alabama Power's use of the platform. It is the vendor's own account, with comments from the utility's staff.

According to the post, Alabama Power, working with the consultancy E Source, built two applications on Databricks. One forecasts how severe weather will affect its grid, including the number of incidents, the resources needed and the estimated time to restore supply, to help its storm centre plan. The other monitors reliability, so that underperforming equipment can be maintained or replaced. The post also gives savings estimates. They are projections, so this page does not repeat them.