Integration PlatformReference ArchitectureCloudAPIsMiddleware
Integration Reference Architecture
A comprehensive reference architecture for building a modern, cloud-native integration platform — covering API management, middleware services, event streaming, data integration, governance, and security with product recommendations across all major cloud vendors.
David TirabassiUpdated
Integration Platform Component Model
Select any capability tile or layer to drill down into specifications and multi-cloud recommendations.
Consumers
Integration Layer
Information Layer
Governance Layer
Management and Monitoring
Development and Testing Tools
Security Layer
Producers
Hover over any capability to learn more.
Overview
This reference architecture defines a comprehensive Integration Platform designed to establish an infrastructure that facilitates the seamless connection between data and service producers and the corresponding consumers. It outlines the essential capabilities required to manage APIs, route messages, transform data, and orchestrate services, wrapped in robust governance, security, and operational pipelines.
Component Model Layers
1. Consumers Layer
The Consumers Layer represents the various end-users and systems that consume an organization's data and services via the integration platform.
| COMPONENT | DESCRIPTION | EXAMPLES |
|---|---|---|
| Internal IT Systems | Internal operational systems and services residing within the primary enterprise corporate network. | Internal App Integration (AWS); Private Virtual Network (Azure); Private VPC Access (GCP); Core APIs (Open Source) |
| External IT Systems | Systems belonging to the organization but situated outside the local high-security network perimeter. | AWS Client VPN (AWS); Azure Bastion / VPN (Azure); Google Cloud VPN (GCP); OpenVPN (Open Source) |
| External Partners | Third-party organizations, vendors, and business partners exchanging business documents or services safely. | AWS Transfer Family (AWS); Azure B2B Connectivity (Azure); Analytics Hub (GCP); AS2 / SFTP Protocols (Open Source) |
| Web Sites | Internal or external web applications that query transactional endpoints or API gateways. | CloudFront CDN (AWS); Azure Front Door (Azure); GCP Cloud CDN (GCP); Next.js Web Apps (Open Source) |
| Mobile Devices | Native iOS or Android applications consuming secure organizational REST or GraphQL services. | AWS AppSync (AWS); Azure Mobile Apps (Azure); Firebase Mobile SDK (GCP); React Native Integration (Open Source) |
| Field Devices | Remote sensors, automated machines, or telemetry endpoints sending data streams from the field. | AWS IoT Core (AWS); Azure IoT Hub (Azure); GCP IoT Core (Partner) (GCP); MQTT Broker (Open Source) |
2. Integration Layer
The Integration Layer comprises a collection of technology components organized based on their shared functionality to bridge data consumers and providers.
| COMPONENT | DESCRIPTION | EXAMPLES |
|---|---|---|
| API Management | Create, document, secure, throttle, and govern APIs. Exposes digital assets cleanly to internal and external developers. | Amazon API Gateway (AWS); Azure API Management (Azure); Google Apigee (GCP); Kong, 3scale (Red Hat), WSO2 (Open Source) |
| Middleware Services (ESB) | Enterprise Service Bus providing message brokering, protocol translation, routing, data transformation, service orchestration, and adapter connectivity. | AWS EventBridge / Step Functions (AWS); Azure Service Bus / Logic Apps (Azure); GCP Application Integration (GCP); Red Hat Integration, MuleSoft, WSO2, Camel (Open Source) |
| ETL / Batch Processing | Batch data movement extracting data from sources, transforming it into compatible formats, and loading it into targets. | AWS Glue (AWS); Azure Data Factory (Azure); GCP Cloud Data Fusion (GCP); Talend Data Integration, Airbyte (Open Source) |
| Change Data Capture (CDC) | Captures and tracks changes made at a data source in real-time or near real-time, ensuring synchronization. | AWS Database Migration Service (DMS) (AWS); Azure SQL CDC / ADF (Azure); GCP Datastream (GCP); Debezium, Fivetran (Open Source) |
| Master Data Management (MDM) | Centralized reference repository ensuring critical enterprise data elements remain consistent, synchronized, and high quality. | Amazon Neptune / partner MDM (AWS); CluedIn MDM / Profisee (Azure); GCP Semarchy / partner MDM (GCP); Talend MDM, Pimcore (Open Source) |
| Secure File Transfer (SFT) | Robust managed file transfer (MFT) securing files in transit and at rest with comprehensive auditing. | AWS Transfer Family (AWS); Azure Storage (SFTP) (Azure); GCP SFTP Gateway (GCP); rclone, Apache NiFi, SFTP (Open Source) |
| Event Streaming | Ingests, durably stores, and delivers high-velocity event streams to independent consumers, decoupling producers from consumers and absorbing load spikes. (Processing those streams is a data-platform concern — see the Kappa pattern.) | Amazon MSK (Kafka) (AWS); Azure Event Hubs (Azure); Google Pub/Sub (GCP); Apache Kafka, Redpanda (Open Source) |
| Workflow & Orchestration | Automation of business processes and rules. Coordinates task sequences, escalations, human-in-the-loop steps, and decision tables. | AWS Step Functions (AWS); Azure Logic Apps (Azure); GCP Workflows (GCP); Camunda BPM, jBPM, temporal.io (Open Source) |
| Artificial Intelligence (AI) | Empowers the integration layer with intelligent decision-making, running transaction flows through machine learning models for predictive routing. | Amazon SageMaker (AWS); Azure Machine Learning (Azure); Google Vertex AI (GCP); TensorFlow, PyTorch (Open Source) |
| Native / Connector Integration | Pre-built, vendor-supplied connectors embedded within enterprise SaaS and COTS platforms, used to exchange data without building or hosting custom integration code. Trades control and portability for speed of delivery. | Amazon AppFlow (AWS); Power Platform Connectors (Azure); Application Integration connectors (GCP); Airbyte, n8n (Open Source) |
| Manual / Ad-hoc Transfer | Governed human-operated data movement for infrequent, low-volume, or exception-path transfers where automation cannot be justified. Executed from a hardened, ephemeral desktop with full audit capture — the controlled fallback, not an absence of pattern. | Amazon WorkSpaces + Transfer Family (AWS); Azure Virtual Desktop (Azure); Cloud Workstations (GCP); Apache Guacamole jump host, audited SFTP (Open Source) |
3. Producers Layer
The Producers Layer encompasses diverse data and service providers within an organization. These sources represent the origin of the data or the backend services being exposed through the Integration Platform.
| COMPONENT | DESCRIPTION | EXAMPLES |
|---|---|---|
| Internal IT Systems | Internal operational systems and services residing within the primary enterprise corporate network. | Internal App Integration (AWS); Private Virtual Network (Azure); Private VPC Access (GCP); Core APIs (Open Source) |
| External IT Systems | Systems belonging to the organization but situated outside the local high-security network perimeter. | AWS Client VPN (AWS); Azure Bastion / VPN (Azure); Google Cloud VPN (GCP); OpenVPN (Open Source) |
| External Partners | Third-party organizations, vendors, and business partners exchanging business documents or services safely. | AWS Transfer Family (AWS); Azure B2B Connectivity (Azure); Analytics Hub (GCP); AS2 / SFTP Protocols (Open Source) |
| Web Sites | Internal or external web applications that query transactional endpoints or API gateways. | CloudFront CDN (AWS); Azure Front Door (Azure); GCP Cloud CDN (GCP); Next.js Web Apps (Open Source) |
| Mobile Devices | Native iOS or Android applications consuming secure organizational REST or GraphQL services. | AWS AppSync (AWS); Azure Mobile Apps (Azure); Firebase Mobile SDK (GCP); React Native Integration (Open Source) |
| Field Devices | Remote sensors, automated machines, or telemetry endpoints sending data streams from the field. | AWS IoT Core (AWS); Azure IoT Hub (Azure); GCP IoT Core (Partner) (GCP); MQTT Broker (Open Source) |
Cross-Cutting Layers
These layers span horizontally across the platform, ensuring integration services remain secure, governed, and operationally sound.
Information Layer
Provides a consolidated perspective of information assets to facilitate their utilization across integration services.
| COMPONENT | DESCRIPTION | EXAMPLES |
|---|---|---|
| Data Definition & Modelling | Defines reusable schemas and objects utilized in service definitions and database schemas. | AWS Glue Schema Registry (AWS); Azure Schema Registry (Azure); GCP Schema Registry (GCP); Apicurio Registry, Avro / JSON Schema (Open Source) |
| Common Vocabulary | Centralized repository for common business objects and fields, allowing domain teams to selectively define integration contract content. | AWS Glue Data Catalog (AWS); Azure Purview Data Catalog (Azure); GCP Dataplex Catalog (GCP); Backstage.io Catalog, Confluent Schema Registry (Open Source) |
Governance Layer
Includes processes and tools to manage artifacts, policies, and the lifecycle of services and APIs.
| COMPONENT | DESCRIPTION | EXAMPLES |
|---|---|---|
| API Developer Portal | A central web-based hub where internal or external developers can discover, explore, test, and access APIs. Includes documentation, SDKs, and analytics. | Amazon API Gateway Developer Portal (AWS); Azure API Management Developer Portal (Azure); Apigee Integrated Developer Portal (GCP); Backstage, Kong Developer Portal (Open Source) |
| Service Catalogue | Centralized registry cataloging all deployed APIs, microservices, structures, dependencies, versions, and deprecations. | AWS Glue Catalog (AWS); Azure Purview (Azure); GCP Analytics Hub (GCP); Backstage.io, Apicurio Registry (Open Source) |
| Service Registry | A runtime registry that manages active services, their endpoints, states, metadata, and dynamic routing configurations. | AWS Cloud Map (AWS); Azure Resource Graph (Azure); GCP Service Directory (GCP); Eureka, Consul (Open Source) |
Security Layer
Ensures data protection, confidentiality, and access control across all integration components.
| COMPONENT | DESCRIPTION | EXAMPLES |
|---|---|---|
| Authentication & Authorisation | Verifies user/client identity (OAuth2, OIDC) and executes granular, role-based access control (RBAC) to APIs and admin functions. | Amazon Cognito / IAM (AWS); Azure Entra ID (Active Directory) (Azure); Google Identity Platform (GCP); Keycloak, Authelia (Open Source) |
| Data Security | Enforces data encryption at rest and in transit, tokenization, data masking, and sensitive data protection policies. | AWS KMS / Macie (AWS); Azure Information Protection / SQL Encryption (Azure); GCP Cloud KMS / DLP API (GCP); Apache Ranger, HashiCorp Vault (Open Source) |
| Secrets Management | Securely stores, encrypts, and rotates sensitive credentials, connection strings, certificates, and API tokens. | AWS Secrets Manager / SSM (AWS); Azure Key Vault (Azure); GCP Secret Manager (GCP); HashiCorp Vault (Open Source) |
| Transport Security | Establishes secure, encrypted point-to-point data transmission channels using mutual TLS (mTLS) and SSL. | AWS Certificate Manager (ACM) (AWS); Azure App Gateway TLS (Azure); GCP Load Balancing TLS (GCP); NGINX mTLS, Linkerd Service Mesh (Open Source) |
| Policy Enforcement | Active edge protection enforcing rate limiting, IP whitelists, request sanitization, and SQL injection shielding. | AWS WAF / Shield (AWS); Azure WAF (Azure); Google Cloud Armor (GCP); Kong Plugins, Open Policy Agent (OPA) (Open Source) |
Management and Monitoring Layer
Oversees runtime activities, identifying deviations from acceptable thresholds and alerting support teams.
| COMPONENT | DESCRIPTION | EXAMPLES |
|---|---|---|
| Service Management | Lifecycle management controls executing service deployments, scaling, version rollbacks, and active containers management. | AWS Systems Manager / ECS (AWS); Azure Automation / Container Apps (Azure); GCP Cloud Run / GKE (GCP); Kubernetes, Ansible (Open Source) |
| Metrics Monitoring | Aggregates and tracks real-time platform performance metrics (latency, CPU, traffic) to trigger alerts for out-of-threshold metrics. | Amazon CloudWatch (AWS); Azure Monitor (Azure); GCP Cloud Operations (GCP); Prometheus, Grafana (Open Source) |
| Logging & Auditing | Collects, structures, and stores log files from all systems for compliance audit trails, threat detection, and search. | AWS CloudTrail / CloudWatch Logs (AWS); Azure Activity Log / Log Analytics (Azure); GCP Cloud Logging (GCP); ELK Stack (Elasticsearch), Grafana Loki (Open Source) |
| Error Handling | Standardizes platform exceptions, redirects failures to Dead Letter Queues (DLQ), and alerts operations for manual resolution. | AWS SQS DLQ (AWS); Azure Service Bus DLQ (Azure); GCP Pub/Sub Dead Lettering (GCP); Apache Camel Error Handler (Open Source) |
| Job Scheduling | Governs and schedules background scripts, automated tasks, and batch ETL jobs to run on cron intervals. | AWS EventBridge Scheduler / Batch (AWS); Azure Automation / Event Grid (Azure); GCP Cloud Scheduler (GCP); Apache Airflow, temporal.io (Open Source) |
Development and Testing Tools Layer
Encompasses the essential capabilities for modelling, designing, testing, and deploying services effectively.
| COMPONENT | DESCRIPTION | EXAMPLES |
|---|---|---|
| Integrated Development Environment | Local or hosted environments loaded with extensions to construct integration flows, mappings, and schemas. | AWS Cloud9 (AWS); VS Code Online (Azure); GCP Cloud Workstations (GCP); VS Code, IntelliJ IDEA, Eclipse (Open Source) |
| Testing Tools | Automates API contract testing, unit mockings, and volumetric load testing to guarantee quality prior to deployment. | AWS Device Farm (AWS); Azure DevTest Labs (Azure); GCP Firebase Test Lab (GCP); Postman, SoapUI, Apache JMeter, Pact (Open Source) |
| CI / CD | Automates packaging, static analysis scanning, testing, and multi-environment deployment pipelines. | AWS CodePipeline (AWS); Azure DevOps Pipelines (Azure); GCP Cloud Build (GCP); GitLab CI, GitHub Actions, Jenkins (Open Source) |
| Configuration Management & Automation | Ensures declarative Infrastructure-as-code (IaC) and system configurations are tracked in source control and deployed predictably. | AWS CloudFormation / CDK (AWS); Azure ARM Templates / Bicep (Azure); GCP Deployment Manager (GCP); Terraform, Ansible, Git (Open Source) |
Solution Patterns and Decision Framework
The Integration Reference Architecture needs to be tailored for each organisation in order for it to be relevant and gain adoption. After tailoring the reference architecture, the next step is to develop solution patterns and a decision framework.
- Solution Patterns are reusable solutions to commonly occurring problems. The Enterprise Integration Patterns (https://www.enterpriseintegrationpatterns.com/) are well known in the integration domain.
- A Decision Framework is a framework that guides an architect in selecting the most appropriate solution pattern for the problem at hand.
Read the following article which describes a set of Solution Patterns and Decision Framework that support this Integration Reference Architecture: /articles/integration-patterns-and-decision-framework