Autoscaling
- Pradeep P
- 3 days ago
- 3 min read
Layer 5 · Post 9 of 12
← Previous: Secrets Management → Next: Multi-region Architecture
Layer 5 — Modern infrastructure · Post 59 of 88
Autoscaling adds or removes capacity from demand signals. Done well it saves money. Done poorly it oscillates or scales too late.
What you'll learn
Horizontal vs vertical autoscaling, and which signal to scale on
Why cooldown, warmup, and min replicas matter more than the CPU chart
How HPA, cluster autoscaling, and scheduled scale interact
The idea in one minute
Autoscaling watches a signal (CPU, QPS, queue depth, custom SLI) and changes how many replicas (or how large a machine) you run.
Load up --> more Pods / nodes Load down --> fewer (after cooldown)
It is a control loop: measure, compare to a target, act, wait. If the loop is naive, you oscillate — scale up, overshoot, scale down, miss SLA, repeat.
Why it matters
Traffic is not a flat line. Sales, launches, and retries (Layer 3) create cliffs. Overprovisioning 24/7 works until the bill arrives. Underprovisioning works until the outage.
Interviews: "we'll autoscale" is incomplete. Say on what metric, min/max, and how slow the new replica is to become ready. A 3-minute image pull cannot save a 10-second spike.
How it works
Horizontal: more copies. Kubernetes HPA (Horizontal Pod Autoscaler) is the textbook: target CPU 70%, or requests per second, or a Prometheus metric. It updates a Deployment's replica count.
Vertical: bigger CPU/memory on the same instance (VPA). Useful for right-sizing; awkward if it restarts Pods to apply. Most user-facing services prefer horizontal.
Cluster / node autoscaling: HPA wants 20 Pods; the cluster has room for 12. A cluster autoscaler (or Karpenter, or cloud ASG) adds nodes. Scale-down drains nodes. This loop is slower than HPA. Design for pending Pods.
Scheduled / predictive: you know Monday 9:00. Scale before the metric moves. Reactive-only scaling always lags.
The control loop details
Target: 50% CPU sounds safe and can still be wrong if the app is queue-bound or I/O-bound. Scale on the bottleneck (queue lag, p99, concurrent inflight).
Warmup: a new Pod is not capacity until readiness passes. Include pull time + JIT + cache fill.
Cooldown: after a scale-up, wait before scale-down. After scale-down, wait before another cut. Stops flapping.
Min replicas: never 0 for a latency-critical service unless you accept cold starts (and designed for them).
Max replicas: a runaway retry storm should not infinitely multiply. Caps are a safety rail.
A simple example
Checkout CPU sits at 40% all day and 90% from 18:00–21:00. HPA: min 4, max 20, target 60% CPU. At 18:05 replicas climb. If the image is cached on nodes, you are fine. If every scale-up pulls 1 GB and the cluster is also adding nodes, you spend 8 minutes overloaded. Fix: pre-pull, min 8 before the peak, or scale on RPS with a scheduled bump at 17:45.
A worker pool should scale on queue depth / consumer lag, not CPU. Idle workers are 0% CPU and 2 million messages behind.
Common mistakes
Scaling on CPU for an I/O-wait service. CPU is calm; latency is not. Wrong signal.
Min replicas = 1, max = 1000, no budget. The first thundering herd becomes a cloud bill and a mesh outage.
Ignoring downstream. You scaled checkout 10× and melted payments, which did not scale. Scale the bottleneck, or add backpressure.
Scale-to-zero without a warmup path. First request pays cold start; users feel it. Fine for async workers; rude for checkout.
No load test of the loop. You tested 10 replicas static, never the 4→20 transition.
How this shows up in real systems
K8s HPA + cluster autoscaler / Karpenter: the usual pair.
AWS ASG, Cloud Run, Lambda, Autoscale groups: same loop, different unit (VM, request, function).
KEDA: scale on queue length, Kafka lag, cron — when CPU is the wrong knob.
Autoscaling is one region getting taller. Next: running that idea in more than one place on the map.
Recap
Autoscaling is a control loop on a signal that matches the bottleneck.
Budget warmup, cooldown, min, and max or you flap and miss spikes.
HPA without node capacity just pending-Pods; include cluster scaling in the design.
One region can still vanish. Next: putting the system in more than one region.
Layer 5 · Post 9 of 12
← Previous: Secrets Management → Next: Multi-region Architecture
Comments