Distributed systems: CAP, quorum, replication, and consensus
System Evolution
Service Discovery
In dynamic systems, service locations change constantly. Discovery replaces hardcoded IPs.
Why the previous solution stopped being enough
You have heartbeats telling you what's alive. But your API service is hardcoded to call the database at 10.0.1.5 β and last week, a rolling deployment changed that IP. Services were calling dead addresses for minutes.
Architecture Diagram
Service A
Service Registry
A β 10.0.1.4:8080
B β 10.0.1.9:8080
Service B
Service C
needs B
gets back: 10.0.1.9:8080
In early-stage systems, it's common to hardcode the address of a database or downstream service. This works when IPs are stable. But in any modern cloud environment β with auto-scaling groups, container orchestration, and rolling deployments β IP addresses change constantly.
Service discovery solves this by introducing a registry. When Service B starts up, it registers itself: 'I'm up, I'm Service B, I'm at 10.0.1.9:8080.' When Service A needs to call Service B, it asks the registry: 'Where is Service B right now?' and gets the current address.
There are two main models. Client-side discovery: the client queries the registry and makes the call directly. Server-side discovery: the client calls a load balancer that queries the registry and routes the request.
Consul, Eureka, etcd, and Kubernetes DNS are all service registries. In Kubernetes specifically, every service gets a stable DNS name (service-name.namespace.svc.cluster.local) that resolves to the current pod IPs automatically.
Service discovery also enables health-aware routing. The registry knows which instances are healthy (via heartbeats or health checks) and only returns healthy addresses. Unhealthy instances are automatically removed from the pool.
Tradeoffs
Engineering Edge Cases
Register with a short TTL and use heartbeats to refresh. If you crash without deregistering, the TTL expires and removes the stale entry.
Cache registry results locally to avoid a registry round-trip on every request β but with a short TTL to stay fresh.
What comes next
Services can now find each other dynamically. But your database is becoming a read bottleneck β a single server can't handle 50,000 read queries per second.
