Integration PlatformReference ArchitectureCloudAPIsMiddleware

Integration Reference Architecture

The integration platform reference architecture covers API management, middleware, events and data integration, with security and cloud examples.

David TirabassiUpdated
Enterprise/Solution Architect · Almost 30 years' experience, specialising in integration · TOGAF 9.1 certifiedLinkedInGitHub

Integration Platform Component Model

Select any capability tile or layer to drill down into specifications and multi-cloud recommendations.

Consumers

Integration Layer

Information Layer

Governance Layer

Management and Monitoring

Development and Testing Tools

Security Layer

Producers

Hover over any capability to learn more.

From architecture to patterns

Each component below is carried by a pattern in the library, one for each boundary the integration crosses. If you are not sure which one fits, the integration patterns and decision framework chooses for you.

Components of the architecture and the patterns that implement them
ComponentPatterns
API Management
Integration Middleware
Event Streaming
ETL / Batch Processing
Secure File Transfer (SFT)
Change Data Capture (CDC)
Master Data Management (MDM)
Workflow & Orchestration
Artificial Intelligence (AI)
Native / Connector Integration
Manual / Ad-hoc Transfer

Overview

An integration platform carries data and events between systems that were designed separately, across internal, external and cloud boundaries. It exists so that a new system joins the estate by implementing patterns already in use, instead of a new point-to-point link for every neighbour.

This reference architecture specifies the components, what each is responsible for, and the relationships between them. It also defines the governance, security and operational controls that apply across all components. The architecture is vendor-neutral, with example services on AWS, Azure and Google Cloud and links to the implementation patterns that realise each capability.

Scope

In scope. The platform is responsible for:

  • Boundary crossing. Exposing and consuming services and events across internal, external and cloud boundaries.
  • API management. Publishing, securing, versioning and monitoring the interfaces other systems call.
  • Messaging and events. Routing, transforming and delivering messages and event streams between producers and consumers.
  • Brokering and orchestration. Sequencing calls across systems, and managing long-running interactions.
  • File and batch movement. Moving bulk data on a schedule where a message or a call is the wrong instrument.

Out of scope.

  • Not a data platform. Landing data once and governing it for analytics is the responsibility of the Data Platform reference architecture.
  • Not a system of record. Business transactions stay with the applications that own them. Where master data management or a rules engine runs on the platform, it serves the integration; the source applications remain the owners of their data.

Principles. No single product implements every capability: an API gateway, an event broker, an iPaaS and a managed file transfer service each cover part of the estate, and the pattern library names a product only as an example. Nor does one pattern serve the whole estate. Two systems can need different interfaces for different purposes (a lookup by API, a change by event, a nightly extract by file), so the pattern is chosen per interface, from volume, timing and data classification.

Component Model Layers

1. Consumers Layer

The Consumers Layer represents the various end-users and systems that consume an organization's data and services via the integration platform.

COMPONENTDESCRIPTIONEXAMPLES
Internal IT SystemsInternal operational systems and services residing within the primary enterprise corporate network.Internal App Integration (AWS); Private Virtual Network (Azure); Private VPC Access (GCP); Core APIs (Open Source)
External IT SystemsSystems belonging to the organization but situated outside the local high-security network perimeter.AWS Client VPN (AWS); Azure Bastion / VPN (Azure); Google Cloud VPN (GCP); OpenVPN (Open Source)
External PartnersThird-party organizations, vendors, and business partners exchanging business documents or services safely.AWS Transfer Family (AWS); Azure B2B Connectivity (Azure); Analytics Hub (GCP); AS2 / SFTP Protocols (Open Source)
Web SitesInternal or external web applications that query transactional endpoints or API gateways.CloudFront CDN (AWS); Azure Front Door (Azure); GCP Cloud CDN (GCP); Next.js Web Apps (Open Source)
Mobile DevicesNative iOS or Android applications consuming secure organizational REST or GraphQL services.AWS AppSync (AWS); Azure Mobile Apps (Azure); Firebase Mobile SDK (GCP); React Native Integration (Open Source)
Field DevicesRemote sensors, automated machines, or telemetry endpoints sending data streams from the field.AWS IoT Core (AWS); Azure IoT Hub (Azure); GCP IoT Core (Partner) (GCP); MQTT Broker (Open Source)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

2. Integration Layer

The Integration Layer comprises a collection of technology components organized based on their shared functionality to bridge data consumers and providers.

COMPONENTDESCRIPTIONEXAMPLES
API ManagementCreate, document, secure, throttle, and govern APIs. Exposes digital assets cleanly to internal and external developers.Amazon API Gateway (AWS); Azure API Management (Azure); Google Apigee (GCP); Kong, 3scale (Red Hat), WSO2 (Open Source)
Middleware Services (ESB)Enterprise Service Bus providing message brokering, protocol translation, routing, data transformation, service orchestration, and adapter connectivity.AWS EventBridge / Step Functions (AWS); Azure Service Bus / Logic Apps (Azure); GCP Application Integration (GCP); Red Hat Integration, MuleSoft, WSO2, Camel (Open Source)
ETL / Batch ProcessingBatch data movement extracting data from sources, transforming it into compatible formats, and loading it into targets.AWS Glue (AWS); Azure Data Factory (Azure); GCP Cloud Data Fusion (GCP); Talend Data Integration, Airbyte (Open Source)
Change Data Capture (CDC)Captures and tracks changes made at a data source in real-time or near real-time, ensuring synchronization.AWS Database Migration Service (DMS) (AWS); Azure SQL CDC / ADF (Azure); GCP Datastream (GCP); Debezium, Fivetran (Open Source)
Master Data Management (MDM)Centralized reference repository ensuring critical enterprise data elements remain consistent, synchronized, and high quality.Amazon Neptune / partner MDM (AWS); CluedIn MDM / Profisee (Azure); GCP Semarchy / partner MDM (GCP); Talend MDM, Pimcore (Open Source)
Secure File Transfer (SFT)Robust managed file transfer (MFT) securing files in transit and at rest with comprehensive auditing.AWS Transfer Family (AWS); Azure Storage (SFTP) (Azure); GCP SFTP Gateway (GCP); rclone, Apache NiFi, SFTP (Open Source)
Event StreamingIngests, durably stores, and delivers high-velocity event streams to independent consumers, decoupling producers from consumers and absorbing load spikes. (Processing those streams is a data-platform concern — see the Kappa pattern.)Amazon MSK (Kafka) (AWS); Azure Event Hubs (Azure); Google Pub/Sub (GCP); Apache Kafka, Redpanda (Open Source)
Workflow & OrchestrationAutomation of business processes and rules. Coordinates task sequences, escalations, human-in-the-loop steps, and decision tables.AWS Step Functions (AWS); Azure Logic Apps (Azure); GCP Workflows (GCP); Camunda BPM, jBPM, temporal.io (Open Source)
Artificial Intelligence (AI)Empowers the integration layer with intelligent decision-making, running transaction flows through machine learning models for predictive routing.Amazon SageMaker (AWS); Azure Machine Learning (Azure); Google Vertex AI (GCP); TensorFlow, PyTorch (Open Source)
Native / Connector IntegrationPre-built, vendor-supplied connectors embedded within enterprise SaaS and COTS platforms, used to exchange data without building or hosting custom integration code. Trades control and portability for speed of delivery.Amazon AppFlow (AWS); Power Platform Connectors (Azure); Application Integration connectors (GCP); Airbyte, n8n (Open Source)
Manual / Ad-hoc TransferGoverned human-operated data movement for infrequent, low-volume, or exception-path transfers where automation cannot be justified. Executed from a hardened, ephemeral desktop with full audit capture — the controlled fallback, not an absence of pattern.Amazon WorkSpaces + Transfer Family (AWS); Azure Virtual Desktop (Azure); Cloud Workstations (GCP); Apache Guacamole jump host, audited SFTP (Open Source)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

3. Producers Layer

The Producers Layer encompasses diverse data and service providers within an organization. These sources represent the origin of the data or the backend services being exposed through the Integration Platform.

COMPONENTDESCRIPTIONEXAMPLES
Internal IT SystemsInternal operational systems and services residing within the primary enterprise corporate network.Internal App Integration (AWS); Private Virtual Network (Azure); Private VPC Access (GCP); Core APIs (Open Source)
External IT SystemsSystems belonging to the organization but situated outside the local high-security network perimeter.AWS Client VPN (AWS); Azure Bastion / VPN (Azure); Google Cloud VPN (GCP); OpenVPN (Open Source)
External PartnersThird-party organizations, vendors, and business partners exchanging business documents or services safely.AWS Transfer Family (AWS); Azure B2B Connectivity (Azure); Analytics Hub (GCP); AS2 / SFTP Protocols (Open Source)
Web SitesInternal or external web applications that query transactional endpoints or API gateways.CloudFront CDN (AWS); Azure Front Door (Azure); GCP Cloud CDN (GCP); Next.js Web Apps (Open Source)
Mobile DevicesNative iOS or Android applications consuming secure organizational REST or GraphQL services.AWS AppSync (AWS); Azure Mobile Apps (Azure); Firebase Mobile SDK (GCP); React Native Integration (Open Source)
Field DevicesRemote sensors, automated machines, or telemetry endpoints sending data streams from the field.AWS IoT Core (AWS); Azure IoT Hub (Azure); GCP IoT Core (Partner) (GCP); MQTT Broker (Open Source)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.


Cross-Cutting Layers

These layers span horizontally across the platform, ensuring integration services remain secure, governed, and operationally sound.

Information Layer

Provides a consolidated perspective of information assets to facilitate their utilization across integration services.

COMPONENTDESCRIPTIONEXAMPLES
Data Definition & ModellingDefines reusable schemas and objects utilized in service definitions and database schemas.AWS Glue Schema Registry (AWS); Azure Schema Registry (Azure); GCP Schema Registry (GCP); Apicurio Registry, Avro / JSON Schema (Open Source)
Common VocabularyCentralized repository for common business objects and fields, allowing domain teams to selectively define integration contract content.AWS Glue Data Catalog (AWS); Azure Purview Data Catalog (Azure); GCP Dataplex Catalog (GCP); Backstage.io Catalog, Confluent Schema Registry (Open Source)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

Governance Layer

Includes processes and tools to manage artifacts, policies, and the lifecycle of services and APIs.

COMPONENTDESCRIPTIONEXAMPLES
API Developer PortalA central web-based hub where internal or external developers can discover, explore, test, and access APIs. Includes documentation, SDKs, and analytics.Amazon API Gateway Developer Portal (AWS); Azure API Management Developer Portal (Azure); Apigee Integrated Developer Portal (GCP); Backstage, Kong Developer Portal (Open Source)
Service CatalogueCentralized registry cataloging all deployed APIs, microservices, structures, dependencies, versions, and deprecations.AWS Glue Catalog (AWS); Azure Purview (Azure); GCP Analytics Hub (GCP); Backstage.io, Apicurio Registry (Open Source)
Service RegistryA runtime registry that manages active services, their endpoints, states, metadata, and dynamic routing configurations.AWS Cloud Map (AWS); Azure Resource Graph (Azure); GCP Service Directory (GCP); Eureka, Consul (Open Source)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

Security Layer

Ensures data protection, confidentiality, and access control across all integration components.

COMPONENTDESCRIPTIONEXAMPLES
Authentication & AuthorisationVerifies user/client identity (OAuth2, OIDC) and executes granular, role-based access control (RBAC) to APIs and admin functions.Amazon Cognito / IAM (AWS); Azure Entra ID (Active Directory) (Azure); Google Identity Platform (GCP); Keycloak, Authelia (Open Source)
Data SecurityEnforces data encryption at rest and in transit, tokenization, data masking, and sensitive data protection policies.AWS KMS / Macie (AWS); Azure Information Protection / SQL Encryption (Azure); GCP Cloud KMS / DLP API (GCP); Apache Ranger, HashiCorp Vault (Open Source)
Secrets ManagementSecurely stores, encrypts, and rotates sensitive credentials, connection strings, certificates, and API tokens.AWS Secrets Manager / SSM (AWS); Azure Key Vault (Azure); GCP Secret Manager (GCP); HashiCorp Vault (Open Source)
Transport SecurityEstablishes secure, encrypted point-to-point data transmission channels using mutual TLS (mTLS) and SSL.AWS Certificate Manager (ACM) (AWS); Azure App Gateway TLS (Azure); GCP Load Balancing TLS (GCP); NGINX mTLS, Linkerd Service Mesh (Open Source)
Policy EnforcementActive edge protection enforcing rate limiting, IP whitelists, request sanitization, and SQL injection shielding.AWS WAF / Shield (AWS); Azure WAF (Azure); Google Cloud Armor (GCP); Kong Plugins, Open Policy Agent (OPA) (Open Source)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

Management and Monitoring Layer

Oversees runtime activities, identifying deviations from acceptable thresholds and alerting support teams.

COMPONENTDESCRIPTIONEXAMPLES
Service ManagementLifecycle management controls executing service deployments, scaling, version rollbacks, and active containers management.AWS Systems Manager / ECS (AWS); Azure Automation / Container Apps (Azure); GCP Cloud Run / GKE (GCP); Kubernetes, Ansible (Open Source)
Metrics MonitoringAggregates and tracks real-time platform performance metrics (latency, CPU, traffic) to trigger alerts for out-of-threshold metrics.Amazon CloudWatch (AWS); Azure Monitor (Azure); GCP Cloud Operations (GCP); Prometheus, Grafana (Open Source)
Logging & AuditingCollects, structures, and stores log files from all systems for compliance audit trails, threat detection, and search.AWS CloudTrail / CloudWatch Logs (AWS); Azure Activity Log / Log Analytics (Azure); GCP Cloud Logging (GCP); ELK Stack (Elasticsearch), Grafana Loki (Open Source)
Error HandlingStandardizes platform exceptions, redirects failures to Dead Letter Queues (DLQ), and alerts operations for manual resolution.AWS SQS DLQ (AWS); Azure Service Bus DLQ (Azure); GCP Pub/Sub Dead Lettering (GCP); Apache Camel Error Handler (Open Source)
Job SchedulingGoverns and schedules background scripts, automated tasks, and batch ETL jobs to run on cron intervals.AWS EventBridge Scheduler / Batch (AWS); Azure Automation / Event Grid (Azure); GCP Cloud Scheduler (GCP); Apache Airflow, temporal.io (Open Source)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

Development and Testing Tools Layer

Encompasses the essential capabilities for modelling, designing, testing, and deploying services effectively.

COMPONENTDESCRIPTIONEXAMPLES
Integrated Development EnvironmentLocal or hosted environments loaded with extensions to construct integration flows, mappings, and schemas.AWS Cloud9 (AWS); VS Code Online (Azure); GCP Cloud Workstations (GCP); VS Code, IntelliJ IDEA, Eclipse (Open Source)
Testing ToolsAutomates API contract testing, unit mockings, and volumetric load testing to guarantee quality prior to deployment.AWS Device Farm (AWS); Azure DevTest Labs (Azure); GCP Firebase Test Lab (GCP); Postman, SoapUI, Apache JMeter, Pact (Open Source)
CI / CDAutomates packaging, static analysis scanning, testing, and multi-environment deployment pipelines.AWS CodePipeline (AWS); Azure DevOps Pipelines (Azure); GCP Cloud Build (GCP); GitLab CI, GitHub Actions, Jenkins (Open Source)
Configuration Management & AutomationEnsures declarative Infrastructure-as-code (IaC) and system configurations are tracked in source control and deployed predictably.AWS CloudFormation / CDK (AWS); Azure ARM Templates / Bicep (Azure); GCP Deployment Manager (GCP); Terraform, Ansible, Git (Open Source)

Products are listed as examples to show how a component can be implemented. Inclusion is not an endorsement of the vendor, and omission is not a judgement about the product.

Solution Patterns and Decision Framework

The Integration Reference Architecture needs to be tailored for each organisation in order for it to be relevant and gain adoption. After tailoring the reference architecture, the next step is to develop solution patterns and a decision framework.

  • Solution Patterns are reusable solutions to commonly occurring problems. The Enterprise Integration Patterns (https://www.enterpriseintegrationpatterns.com/) are well known in the integration domain.
  • A Decision Framework is a framework that guides an architect in selecting the most appropriate solution pattern for the problem at hand.

Read the following article which describes a set of Solution Patterns and Decision Framework that support this Integration Reference Architecture: /articles/integration-patterns-and-decision-framework