Designing a Feature Flag System

Pradeep P
3 days ago
4 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 5 of 26

← Previous: Designing a Recommendation System → Next: Designing a Distributed Scheduler

Layer 6 — Modern systems · Post 67 of 88

A feature flag system lets you turn code paths on or off for specific users without redeploying, which is how modern teams ship and experiment safely.

What you'll learn

Why flags are a distributed config problem (evaluation close to the request, not a DB hit per check)
How targeting, percentage rollouts, and a kill switch actually get evaluated
What goes wrong with consistency, stale SDKs, and flags that never die

The idea in one minute

A feature flag is a named switch plus rules: "on for employees," "on for 5% of users in India," "off everywhere." Your code asks if flags.on("new_checkout", user) and takes a path. Changing the flag does not require a deploy.

Admin UI / Git / API | v [ Flag store + audit log ] --publish--> [ CDN / streaming config ] | +-------------------------------------+ | SDK poll / stream | v v [ App servers / clients ] [ Edge / workers ] | | evaluate locally evaluate locally | store unreachable? last-known-good snapshot, fail closed for risky flags

The interview design is not SELECT enabled FROM flags on every request. That couples your p99 to the flag service and will take you down when you need the kill switch most.

Why it matters

Modern shipping is: merge behind a flag, deploy dark, open to 1%, watch errors, then 100% or rollback the flag — not the binary. Experimentation (A/B) is the same machinery with a variant instead of a boolean.

If the flag service is in the request path and it dies, you either freeze the world on old values or accidentally enable a half-finished checkout. Interviewers want that failure story.

How it works

Control plane. Product and eng set flags: key, type (bool, string, JSON), rules (user IDs, attributes, percentage), and environments (dev/stage/prod). Writes are audited. This store is small.
Distribution. A publisher pushes a snapshot (or a stream of diffs) to every evaluator: server SDKs, mobile, edge. Snapshots are versioned. SDKs poll every few seconds or subscribe (streaming).
Evaluation. The SDK hashes flag_key + user_key into a bucket (0–99) for percentage rollouts so the same user is stable. Rules are ordered: kill switch / targeting overlays beat the default. Evaluation is CPU-local and microseconds.
Contexts. You pass a context: user_id, country, plan, app_version. Never send secrets you would not log. Client-side flags are visible to the user — do not hide a pricing algorithm in a mobile flag.
Failure. SDK uses last known good. For a dangerous feature, fail closed (treat missing config as off). For a performance optimization, fail open might be fine. Name the default in the interview.
Experiments. A flag with variants plus an assignment that you log. The experiment platform is analytics on top of the same assignment function.

Clients (admin) write flags. APIs of your product only read via SDK. Stores are the flag DB plus a cache/CDN of snapshots. Failure is partition from the control plane — local snapshot must still evaluate.

A simple example

You ship a new search ranker behind search_v2. Rules: on for employee=true; then 5% of users whose hash falls in 0–4. A user in the 5% hits any of your 40 app servers and always gets v2 because the hash is deterministic. Error rate spikes. You flip the flag off in the UI. Within the poll interval (or immediately on stream), SDKs evaluate false. You did not roll back the deploy.

An intern checks flags.on("search_v2") with an extra Postgres call "to be sure." During a flag-service blip, checkout waits on that query. That is the anti-pattern.

Common mistakes

Evaluating in a remote API on every if. Flags must be local. The control plane can be down.

Non-sticky percentages (random() each request). Users flicker between old and new UI. Hash the user.

Flags that live forever. Every flag is a bit of extra cyclomatic complexity. Expiry dates and "remove after 100%" are process, but mention them.

Sensitive logic only in client flags. Attackers read the binary. Server-side enforcement still required.

No audit log. "Who turned on payments_v3 in prod?" is an incident question.

How this shows up in real systems

LaunchDarkly, Split, Unleash, Flagsmith, ConfigCat: control plane + SDKs.
Facebook/Gatekeeper, Uber/experiments: in-house, same snapshot idea.
Kubernetes / app config: related, but flags add per-user targeting, not only per-cluster YAML.

Layer 2's "config reload" meets Layer 4's consistent hashing (sticky buckets). Next: work that must run at a time, not on a user request — schedulers.