Hot Keys and Hot Partitions

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 1 — The building blocks

Layer 1 · Post 21 of 21

← Previous: Cache Stampede → Next: Delivery Guarantees

Layer 1 — The building blocks · Post 94 of 119

A single popular key or uneven partition can saturate one shard while the rest sit idle.

What you'll learn

Why a "perfectly sharded" cluster still melts on one ID
Salting, splitting, local caches, and "don't shard by celebrity"
How this is different from a cache stampede

The idea in one minute

Sharding spreads keys across machines. It does not spread traffic unless traffic is spread across keys.

A hot key is one ID that takes a huge fraction of reads or writes (user:celebrity, product:iphone, tenant:bigcorp). A hot partition is a shard that owns too many of those, or one range that grew (all of today's timestamps, all users in one city).

Shard 1  ████████████████  90% CPU   <-- celebrity user id
Shard 2  ██                8%
Shard 3  █                 2%

Amdahl: for that tenant, N = 1. Consistent hashing will happily pin the celebrity to one vnode forever.

Why it matters

This is the usual reason "we sharded and it still falls over." Interviewers want a key design story, not another box on the diagram.

It is also a product constraint: a social network cannot put GET /feed for a mega-account on one Redis key and hope. You split the work, not only the keyspace.

How it works

Pick a shard key that matches load, not only uniqueness. userId is fine if users are similar. It is bad if 1% of users are 90% of QPS. Time-only keys (yyyy-mm-dd) dump a day onto one partition.

Split the hot key.

Read: local/process cache, CDN, or replicate that one row to many caches. The source of truth can stay one row; serving should not.
Write: salting — store likes:postId:{0..N} and sum on read; or shard the write by hash(actorId) so each liker hits a different counter. You trade read complexity for write spread.
Queue: partition Kafka by something other than the celebrity id if that partition cannot keep up; or add keys.

Isolate tenants. A noisy neighbor gets a dedicated shard / cell (cell-based architecture, later) instead of sitting on the same partition as everyone else.

Detect. Per-partition CPU, per-key QPS, Kafka consumer lag by partition. Averages hide a hot shard.

Do not "fix" it with a bigger single node forever. That is vertical scaling of the celebrity. It works until it does not. Have a split plan.

Stampede (previous) is synchronized emptiness. Hot key is sustained load on one place, cache full or not.

A simple example

A ticketing site: event:world-cup-final inventory in one row, one shard. 200,000 people refresh. The row is the correct source of truth for "how many seats," but the machine cannot take 200k writes.

Patterns that actually ship:

Hold inventory in many buckets (sector-A, sector-B) so contention is per sector.
Serialize purchases through a queue per event and make the write rate match what one row can do — and tell the product that.
Cache sold out aggressively so reads stop hitting the row.

You cannot consistent-hash your way out of one event ID. You change the key or the access pattern.

Common mistakes

Sharding on tenant_id when one tenant is the business. You built a dedicated slow database for your biggest customer.

Auto-increment IDs as the Kafka key. All new records go to the "high" partition if you range-shard; hash the id instead, unless you want time-order on one partition.

Read replicas for a hot write key. Replicas do not split writes. They can split reads if you accept lag.

Fixing only Redis. The hot key often continues in the database, the lock table, and the search index. Follow the ID.

How this shows up in real systems

DynamoDB adaptive capacity / split for hot keys: AWS moving extra throughput to a partition — still has limits; salt if you must.
Kafka: one partition = one consumer in a group. A hot key hashed to that partition caps throughput.
Instagram / Twitter-style fanout: celebrity keys get special paths (dedicated caches, pull vs push).
Postgres: one heavily updated row = lock convoy. Same problem without "shards" in the name.

Recap

Shards spread keys; hot keys concentrate load.
Split reads with cache; split writes with salts / smaller keys; isolate noisy tenants.
Next: delivery guarantees — what queues can actually promise.