Unified Batch & Stream Processing (Kappa)

1. Producers emit every state change as an immutable event to a durable, append-only log topic—there is no separate batch feed and no in-place update, only new records appended in order.
2. The Event Log (Kafka) retains these events as the single system of record, with retention configured long enough to replay the full history rather than a short buffer.
3. A single Stream Processing Engine (Flink, Kafka Streams, or Spark Structured Streaming) subscribes to the log and applies one codebase of transformation logic to compute aggregates, joins, and enrichments continuously.
4. For real-time consumption, the Engine reads from the log tail and writes low-latency results to a serving store.
5. For a historical recompute or logic change, the Operator deploys a new job version that resets its offset to the beginning of the log and reprocesses every event through the same code path, materializing a fresh output table.
6. The Serving Store (a key-value or columnar view) is atomically swapped or versioned so downstream Consumers—dashboards, APIs, ML features—query a consistent view without dual-write reconciliation.

• Single codebase and consistent logic vs the cost of retaining a long, replayable event log.
• On-demand historical reprocessing vs reprocessing latency and throughput limits when replaying billions of events.
• Simpler operations (one engine) vs stream frameworks being harder to reason about for complex batch-style joins.
• Immutable audit trail vs increased storage and the need for compaction/tiering strategies.

A ride-hailing company streams every trip lifecycle event—requested, matched, started, completed—into Kafka as the authoritative log. A single Flink application computes driver payout aggregates and surge-pricing signals in real time. When finance discovers a rounding error in the payout formula, engineers deploy a corrected job that resets to offset zero and replays six months of trip events through the identical operator graph, regenerating accurate historical payouts into a new serving table—no separate Spark batch pipeline, no reconciliation between two divergent implementations.

• Ordering & keyed state: Guarantees hold only per-partition; events sharing a key must be partitioned together or windowed aggregates and joins produce wrong results. Cross-key ordering is never guaranteed, so design keys around your consistency boundary.
• Replay determinism: A recompute is only correct if operators are deterministic—wall-clock timestamps, external lookups, and random IDs break replay. Use event-time semantics with watermarks and pin enrichment data to versioned snapshots.
• Schema evolution: Old and new events flow through one code path during a full replay, so consumers must decode every historical schema. Use a registry with backward-compatible rules and never repurpose field meanings.
• Reprocessing cost & throughput: Replaying billions of events saturates source-topic read bandwidth and serving-store write capacity; throttle parallelism and provision the swap table separately from live traffic. Retention length directly drives storage bill—tier cold segments to object storage.
• State backend maintenance: Large keyed state needs checkpoint/savepoint tuning, incremental snapshots, and periodic state TTL to avoid unbounded growth. Log compaction on keyed topics reclaims space but destroys full-history replay—keep compaction and reprocessing topics separate.
• Observability: Track consumer lag, watermark skew, checkpoint duration/failure rate, and replay progress by offset; version output tables and record input offset ranges as lineage so any served number is traceable to a code version.