Event-Driven Microservices Architecture with Apache Kafka
An event-driven architecture communicates via immutable state change events published to an append-only log, enabling asynchronous processing, zero temporal coupling, and horizontal consumer scaling.
Live Architecture Studio
Edit components, modify labels, add databases, or redraw connections directly on this canvas:
Loading Kafka Event Stream Blueprint...
Mounting vector diagram elements, nodes, and capacity metrics
Loading Kafka Event Stream Blueprint...
Mounting vector diagram elements, nodes, and capacity metrics
1. Problem & Challenge
Synchronous REST calls between 15 microservices create tight temporal coupling: if one downstream service times out, checkout fails. The system needs asynchronous event distribution with high throughput, guaranteed ordering per entity, and replayability.
2. Core Building Blocks & Responsibilities
👉 Desliza la tabla para ver roles y responsabilidades| Component | Role | Plain-English Explanation |
|---|---|---|
| Event Producer Microservice | Domain Event Publisher | Executes local business transactions and emits state change events (e.g., OrderPlacedEvent). |
| Transactional Outbox Table | Dual-Write Guarantee | Relational table committed in the same ACID transaction as the business entity to guarantee at-least-once publishing. |
| Apache Kafka Broker Cluster | Distributed Partitioned Log | High-throughput append-only disk commit log partitioned across brokers with sequential I/O and zero-copy transfer. |
| Schema Registry | Contract Evolution & SerDe | Enforces backwards-compatible Avro or Protobuf schemas to prevent breaking changes across decoupled services. |
| Consumer Groups (Payments, Inventory) | Parallel Event Processors | Independent groups of microservices consuming partitions concurrently, tracking their own committed offsets. |
| Dead-Letter Queue (DLQ) | Poison Pill Quarantine | Isolates unprocessable or corrupt messages without blocking the main partition consumer offset progression. |
3. Step-by-Step Request Flow
Atomic Database Mutation
Order Service commits order record and writes OrderPlaced event to local outbox table in one ACID transaction.
CDC / Outbox Relay
Change Data Capture (Debezium) reads outbox table log and streams message to Kafka topic orders.v1.
Key Hashing & Partitioning
Kafka hashes event order_id to assign it to Partition 3, guaranteeing strict sequential ordering for that specific order.
Independent Parallel Consumption
Payment Consumer Group and Inventory Consumer Group read simultaneously from Partition 3 at independent processing speeds.
Idempotent Execution & Commit
Consumers verify idempotency keys against local datastore, process business logic, and commit partition offsets.
4. Architectural Trade-offs
Partition-level Ordering vs Global Ordering
Chosen: Partition-level by Entity ID
Rationale: Global ordering requires a single partition, bottlenecking throughput to ~10k QPS. Partitioning by customer or order ID scales horizontally across hundreds of brokers while guaranteeing strict causality per entity.
At-Least-Once Delivery vs Exactly-Once Semantics (EOS)
Chosen: At-Least-Once + Idempotent Consumers
Rationale: Full two-phase commit Kafka transactions (EOS) add 15-30% latency overhead. Combining high-speed At-Least-Once delivery with database unique constraints or Redis idempotency keys provides bulletproof consistency at maximum throughput.
Interview Tip
Highlight the Transactional Outbox Pattern: Never publish directly to Kafka inside an application database transaction block. If the network write to Kafka succeeds but the local DB commit fails, you emit phantom events. Writing the event to an outbox table in the local DB guarantees 100% atomicity.
Explore Related System Blueprints
TinyURL Shortener
A URL shortener converts a long link (like a 100-character article URL) into a compact 7-character key (like tinyurl.com/xyz123) and redirects visitors in under 15 milliseconds.
API Rate Limiter
A rate limiter acts as a digital bouncer at the door of your API, ensuring each client stays within their allowed request limits (e.g. 100 requests per minute) and blocking abusive traffic.
Video Streaming CDN
Streaming high-definition video to millions of smart TVs and mobile phones requires breaking large 10GB video files into tiny 5-second chunks, encoding each into 20 different resolutions, and caching them right inside local ISP networks.
Uber Dispatch Engine
A real-time geospatial dispatch system matches riders with the most optimal nearby drivers using 64-bit H3 hexagonal indexing and 2-second batch optimization, minimizing city-wide pickup ETA and driver idle time.
Stripe Payments Ledger
A resilient financial payments architecture guarantees strict consistency (CP system) using cryptographic idempotency reservation, double-entry balanced postings, and sharded balance locks.
Figma Multiplayer Engine
A real-time multiplayer document engine uses stateful sticky session routing and server-authoritative operational ordering to sync 2D scene graphs across worldwide collaborators without locking.