Webhooks (Event-Push Ingestion)

1. The source system (SaaS app, payment processor, or partner service) detects a state change and issues an HTTP POST carrying the event payload to a pre-registered callback URL.
2. The API Gateway terminates TLS, validates the HMAC signature header against a shared secret, checks the request against a per-source rate limit, and rejects malformed or unauthenticated calls before they reach any compute.
3. A lightweight ingestion function acknowledges the delivery immediately with a 200 response, then writes the raw envelope—headers, body, and a generated event ID—to a durable landing store, decoupling acceptance from processing.
4. The event broker (a queue or streaming topic) fans the buffered event out to downstream consumers, providing back-pressure and replay so a slow transform cannot drop deliveries.
5. A stream processor deduplicates on the event ID, applies schema validation, enriches, and lands typed records into the curated serving tier.
6. Analytics and downstream apps query the curated tables with near-real-time freshness, no polling loop required.

• Low latency and zero wasted polls vs. you must expose and defend a public internet-facing endpoint.
• Loose coupling via the broker vs. added operational surface (dead-letter queues, replay tooling).
• Scales with event volume, not schedule vs. unpredictable spikes demand autoscaling and rate limits.
• Provider owns delivery timing vs. you inherit their retry semantics and at-least-once duplicates.

An online marketplace registers a webhook with its payment provider so that every charge.succeeded and refund.created event is pushed to its API Gateway the instant it occurs. Previously a five-minute polling job scanned the payments API and returned mostly empty pages; now the gateway verifies the Stripe signature, drops the raw event onto a Kafka topic, and a stream processor reconciles it against open orders within seconds—letting fraud dashboards and settlement reports run on genuinely fresh data while the payments API sees zero polling traffic.

• At-least-once delivery, dedup by event ID: Providers retry on any non-2xx or timeout, so the same event arrives multiple times; make the event ID persistent and enforce idempotency at the dedup step—collapsing on payload hash alone breaks when providers resend identical-but-reissued events.
• Ordering is not guaranteed: Webhooks fire per-state-change and can arrive out of sequence (a refund before its charge); carry the source-side timestamp or sequence number and reconcile on that, not on arrival order.
• Acknowledge fast, process later: Most providers enforce a short delivery timeout (often 3–10s) and disable endpoints after repeated failures. Return 200 only after the raw envelope is durably persisted—never after full transform—or you risk silent endpoint suspension.
• Secret and signature rotation: HMAC secrets expire and providers rotate signing keys; support overlapping active secrets during rotation and reject on clock skew for timestamp-signed payloads.
• Schema drift: Providers add fields and version event types without notice; validate loosely on ingest, store the raw envelope for replay, and pin a schema version per event type so downstream transforms fail loudly rather than silently dropping fields.
• Observability and DLQ hygiene: Track signature-rejection rate, dedup hit rate, broker lag, and DLQ depth; keep a replay path from the landing store, since a provider's retry window (often hours to days) is your only backfill source if a consumer bug corrupts curated data.