Distributed systems: CAP, quorum, replication, and consensus
System Evolution
Heartbeat
Nodes die silently. Heartbeats let orchestrators know which nodes are actually alive.
Why the previous solution stopped being enough
Your system is stable under load. But last week a worker node ran out of disk space and stopped processing jobs β silently. Nobody noticed for six hours. Traffic was still routed to it, and requests were lost.
Architecture Diagram
Health Monitor
orchestrator
Node A
alive
Node B
alive
Node C
silent
Node C timeout β marked dead β traffic redirected
In distributed systems, processes don't announce when they crash. A node can fail for many reasons β hardware failure, OOM kill, network partition, disk full β and in most cases, no explicit failure message is sent. It just goes quiet.
A heartbeat is a periodic signal β typically a small ping every few seconds β that a node sends to a central monitor or coordinator. As long as heartbeats arrive, the node is considered alive.
When heartbeats stop, the monitor starts a countdown. After a timeout period (usually 2β3 missed heartbeats), the node is declared dead. Traffic is rerouted, replacement jobs are scheduled, and alerts fire.
Heartbeat intervals involve a tradeoff. Short intervals (1 second) detect failures quickly but generate significant background traffic across a large cluster. Long intervals (30 seconds) are quieter but leave you routing traffic to dead nodes for longer.
In Kubernetes, this is the liveness probe. In Zookeeper, service nodes maintain ephemeral znodes that expire when the connection dies. In databases, replication monitors use heartbeats to detect primary failure and trigger failover.
Tradeoffs
Engineering Edge Cases
Set the timeout to at least 2β3 heartbeat intervals, not 1. A single missed heartbeat is often a transient delay, not a failure.
The heartbeat monitor itself is a single point of failure β it must be replicated.
What comes next
You know which nodes are alive. But as services scale out, how does Service A know where to find Service B? IPs change constantly as nodes come and go.
