πŸ’¬
βœ‰οΈ
πŸ“Έ
▢️

Loading…

Distributed systems: CAP, quorum, replication, and consensus

System Evolution

System Evolution

Step 6 of 14
πŸ”

Service Discovery

In dynamic systems, service locations change constantly. Discovery replaces hardcoded IPs.

Why the previous solution stopped being enough

You have heartbeats telling you what's alive. But your API service is hardcoded to call the database at 10.0.1.5 β€” and last week, a rolling deployment changed that IP. Services were calling dead addresses for minutes.

Architecture Diagram

Service A

register

Service Registry

A β†’ 10.0.1.4:8080
B β†’ 10.0.1.9:8080

register

Service B

Service C

needs B

asks registry: "where is B?"
gets back: 10.0.1.9:8080

In early-stage systems, it's common to hardcode the address of a database or downstream service. This works when IPs are stable. But in any modern cloud environment β€” with auto-scaling groups, container orchestration, and rolling deployments β€” IP addresses change constantly.

Service discovery solves this by introducing a registry. When Service B starts up, it registers itself: 'I'm up, I'm Service B, I'm at 10.0.1.9:8080.' When Service A needs to call Service B, it asks the registry: 'Where is Service B right now?' and gets the current address.

There are two main models. Client-side discovery: the client queries the registry and makes the call directly. Server-side discovery: the client calls a load balancer that queries the registry and routes the request.

Consul, Eureka, etcd, and Kubernetes DNS are all service registries. In Kubernetes specifically, every service gets a stable DNS name (service-name.namespace.svc.cluster.local) that resolves to the current pod IPs automatically.

Service discovery also enables health-aware routing. The registry knows which instances are healthy (via heartbeats or health checks) and only returns healthy addresses. Unhealthy instances are automatically removed from the pool.

Tradeoffs

Dynamic routingTraffic automatically routes around failed or replaced instances.
Registry availabilityIf the registry goes down, services can't discover each other. It must be highly available.
Stale entriesCrashed services may remain in the registry briefly before TTL expiry. Callers get dead addresses.
ComplexityAdds an operational component that must be maintained, monitored, and scaled.

Engineering Edge Cases

β–Έ

Register with a short TTL and use heartbeats to refresh. If you crash without deregistering, the TTL expires and removes the stale entry.

β–Έ

Cache registry results locally to avoid a registry round-trip on every request β€” but with a short TTL to stay fresh.

What comes next

Services can now find each other dynamically. But your database is becoming a read bottleneck β€” a single server can't handle 50,000 read queries per second.