Service Discovery
- Pradeep P
- 4 days ago
- 3 min read
Layer 5 · Post 3 of 12
← Previous: Kubernetes → Next: Service Meshes
Layer 5 — Modern infrastructure · Post 53 of 88
Service discovery answers 'where is this service right now?' so clients do not need hardcoded IPs in a world where pods come and go.
What you'll learn
Why hardcoded IPs die the moment you autoscale or reschedule
Client-side vs server-side discovery, in plain language
How Kubernetes DNS, load balancers, and registries fit together
The idea in one minute
Instances appear and vanish. IPs are rented, not owned. Service discovery is the lookup: given a logical name (payments), return healthy addresses now.
Checkout --"where is payments?"--> [ Discovery ] | v 10.1.2.8, 10.1.2.9 (healthy)
Without it you put IPs in config and page people when a deploy moves a box.
Why it matters
Layer 1 load balancers already hide many backends behind one VIP. Discovery is the same idea when the set of backends is dynamic and large: containers, multi-region pools, ephemeral CI.
In interviews, "microservices" without discovery means every service hardcodes every other service. That design does not survive the first autoscaling event.
How it works
Two patterns show up everywhere.
Server-side discovery
The client always talks to a stable name or VIP. A load balancer, Kubernetes Service, or DNS name resolves to whoever is healthy. The client stays dumb. Kubernetes CoreDNS plus kube-proxy (or a replacement) is this: payments.default.svc → ClusterIP → endpoints.
Cloud load balancers do the same at the edge: register instance IPs, health-check, remove the dead.
Client-side discovery
The client queries a registry (Consul, Eureka, etcd), gets a list, and picks an instance (round robin, least requests). Netflix OSS popularized this. You gain flexibility (custom load balancing, zone awareness). You pay with a library in every language and a registry that must stay up.
Registration
Self-registration: the instance announces itself on boot and heartbeats. Fail to heartbeat, you are gone.
Third-party registration: the orchestrator does it. Kubernetes watches Pods and updates Endpoints/EndpointSlices. You do not call a registry API from your app. That is the usual modern default.
DNS is discovery with a catch: TTLs. Short TTLs mean more lookups. Long TTLs mean stale IPs after a scale-down. Kubernetes in-cluster DNS is designed around this; public DNS as your only service registry often is not.
A simple example
Checkout calls payments. In Kubernetes you use http://payments. The Service keeps an endpoint list. A rolling deploy adds new Pods, drops old ones. Checkout never sees Pod IPs.
On VMs, payments processes register in Consul on start. Checkout's client library asks Consul, caches the list for a few seconds, and skips instances that fail a health check. Same problem, different registry.
Common mistakes
Caching the list forever. You scale to zero in a region and clients hammer ghosts. Bound the cache; honor deregistration.
Discovery without health. A process that is up but not ready still gets traffic. Pair discovery with readiness, not just "port is open."
Using public DNS as a microservice phone book. You will fight TTL, cache in resolvers you do not control, and split-horizon pain. Use in-cluster DNS, a mesh, or an internal load balancer.
One global registry, no failure domain. If discovery is down, the whole mesh is blind. Replicate it; prefer the orchestrator's built-in list when you are already on k8s.
How this shows up in real systems
Kubernetes Services + CoreDNS: the default inside a cluster.
AWS Cloud Map, Consul, Eureka: registries when you are not all-in on one orchestrator.
Service meshes: discovery plus retries and mTLS — next post. The mesh still needs a source of truth for endpoints.
API gateways: north-south discovery for clients outside the cluster. Multi-region adds a layer: which region first, then which instance.
Recap
Discovery maps a logical name to healthy instances that exist right now.
Server-side (VIP / k8s Service) keeps clients simple; client-side needs a registry and a library.
Let the orchestrator register instances when you can, and never skip health.
Once you can find a service, you still have to talk to it safely — that is the mesh.
Layer 5 · Post 3 of 12
← Previous: Kubernetes → Next: Service Meshes
Comments