Integration / ESP
ESPInternalIntegration Pattern

Int-ESP-Internal

Problem

A backend workload requires asynchronous, decoupled, and potentially real-time data exchange or event notification with one or more other internal backend workloads or microservices within the private subnet.

Solution

Implement a centralized Event Streaming Platform (ESP) or Message Broker to facilitate asynchronous communication between internal backend workloads. Producers publish events or messages to designated topics or streams, while consumers subscribe to these topics, utilizing consumer groups or queues to ensure efficient and scalable message consumption.

Cloud Paradigm

  • Event-Driven Architecture (EDA)
  • Asynchronous Messaging
  • Microservices Communication
  • Stream Processing and Analytics
  • Publish-Subscribe Pattern
  • Loose Coupling

Implementation Guidelines

Solution Flow

Producer Flow (Publishing Events):

  1. Backend Workload (Producer): An internal microservice or backend workload generates an event or message requiring asynchronous communication.
  2. Event Streaming Platform (ESP): The producer publishes this event to a designated topic or stream within the ESP, ensuring proper authentication and adherence to schema.
  3. Event Persistence & Distribution: The ESP ingests, persists, and distributes the event to all authorized and subscribed consumers.

Consumer Flow (Consuming Events):

  1. Backend Workload (Consumer): An internal microservice or backend workload subscribes to one or more topics or streams on the ESP, often as part of a consumer group to ensure scalable and fault-tolerant processing.
  2. Event Retrieval: The consumer retrieves new events from the subscribed topic, processing them according to its business logic. The ESP manages offsets and ensures message delivery guarantees.
  3. Acknowledgement: Upon successful processing, the consumer acknowledges the message, allowing the ESP to update its state and ensure that messages are not reprocessed unnecessarily in a fault-tolerant manner.

Additional Details

  • Protocol & Format: Utilize modern, efficient streaming protocols (e.g., Kafka protocol, AMQP, MQTT) for communication. Data formats should be standardized (e.g., Avro, Protobuf, JSON, CloudEvents) to ensure interoperability and schema validation.
  • Schema Management: Implement a centralized schema registry to govern message schemas, ensuring producers adhere to defined structures and consumers can parse messages reliably. This is critical for evolving data contracts.
  • Quality of Service (QoS): Configure the ESP to provide appropriate QoS levels, such as at-least-once or exactly-once delivery semantics, based on the criticality of the data and business requirements.
  • Message Durability: Ensure events are durably stored within the ESP, allowing consumers to process messages even after temporary outages or scaling events.
  • Event Processing Patterns: This pattern supports various event processing patterns, including simple publish-subscribe, stream processing for real-time analytics, event sourcing, and Command Query Responsibility Segregation (CQRS) implementations.
  • Scalability & Resilience: The ESP itself must be highly available, fault-tolerant, and scalable to handle varying message volumes and maintain low latency.
  • Observability: Implement comprehensive monitoring for the ESP, tracking metrics like message throughput, latency, consumer lag, and error rates. Centralized logging and distributed tracing (e.g., OpenTelemetry) across producers, ESP, and consumers are essential for diagnosing issues.
  • Developer Experience: Provide clear API/SDKs for interacting with the ESP and comprehensive documentation (e.g., AsyncAPI) for all topics and event structures to accelerate developer adoption and integration.

Security Controls

  • Transport Security: Enforce strict Transport Layer Security (TLS 1.2 or higher) on all connections between producers, the Event Streaming Platform, and consumers.
  • Authentication & Authorization:
    • Authenticate all producers and consumers interacting with the ESP using service accounts, managed identities, or Mutual TLS (mTLS).
    • Implement robust authorization policies (e.g., Role-Based Access Control - RBAC) to control which producers can publish to specific topics/streams and which consumers can subscribe to them.
  • Data Encryption: Ensure data at rest within the Event Streaming Platform is encrypted using platform-managed or customer-managed keys.
  • Network Segmentation: Deploy the Event Streaming Platform within a private subnet (workloads), isolating it from direct public internet exposure. Access should be restricted to authorized internal backend workloads.
  • Auditing and Logging: Enable comprehensive logging of all producer/consumer interactions, configuration changes, and access attempts. Integrate logs with a centralized security information and event management (SIEM) system for anomaly detection and auditing.