Distributed systems: CAP, quorum, replication, and consensus
System Evolution
Streaming & Event-Driven Systems
Synchronous calls chain latency. Event-driven systems decouple producers and consumers β permanently.
Why the previous solution stopped being enough
Your system is balanced and resilient. But your checkout service synchronously calls payments, inventory, notifications, and analytics in sequence. Under load, any one slow call blocks the entire checkout.
Architecture Diagram
Service A
Service B
Mobile App
Event Stream
Kafka / Kinesis
Analytics
Notifications
ML Pipeline
Producers and consumers are decoupled. Each evolves independently.
In synchronous systems, Service A calls Service B and waits. Service A calls Service C and waits. If B is slow, A is slow. If C fails, A fails. The latency and reliability of every service is multiplied across the call chain.
Event-driven systems break this coupling. Instead of calling B and C directly, A publishes an event: 'order.placed'. Services B, C, D, E β whatever is interested β subscribe to that event and process it independently, at their own pace.
The event stream (Kafka, Kinesis, Pulsar) is a durable, ordered log. Events are stored reliably and replayed if a consumer fails. This gives you exactly-once or at-least-once processing semantics, depending on configuration.
The benefits are profound: producers don't wait for consumers (lower latency), consumers can scale independently, new consumers can be added without touching the producer, and you get a full audit trail of everything that happened.
The tradeoffs are equally real: operations that span multiple services (like a transaction) can't be wrapped in a single atomic unit. Eventual consistency must be embraced. Debugging a chain of events is harder than tracing a synchronous call stack.
Tradeoffs
Engineering Edge Cases
The Saga pattern handles distributed transactions in event-driven systems: each step publishes events, and failure triggers compensating events to roll back.
Consumer group lag is your main health metric β if consumers can't keep up with event production, you have a backpressure problem (see step 4).
π
Module Complete
You've traced how distributed systems evolved β from a single server through every mechanism that keeps them alive. Each concept exists because the previous one introduced a new failure mode.
