Configuration Management

Pradeep P
3 days ago
3 min read

Series: Modern System Design · Layer 5 — Modern infrastructure

Layer 5 · Post 7 of 12

← Previous: Observability → Next: Secrets Management

Layer 5 — Modern infrastructure · Post 57 of 88

Configuration management is how settings reach running services safely: what changes, who can change it, and how a bad change is rolled back.

What you'll learn

What belongs in config versus code versus secrets
Static files, env vars, and dynamic config — and when each is safe
Why a config change is a deploy, even when no binary moved

The idea in one minute

Your service needs a database URL, a feature on/off, a rate-limit number, and a timeout. Configuration management is how those values get to the process, who can edit them, and how you undo a bad edit.

git / console --> [ store ] --> running replica | audit + rollback

A surprising fraction of outages are "someone flipped a flag" or "prod still has staging's URL." Treat config as production.

Why it matters

You already know deploys can break prod. Config changes ship faster and often without review. That is the point and the hazard.

Interviews: when you say "we'll tune the cache TTL" or "kill switch for the new matcher," you are designing a config system. If the answer is "edit a YAML on the box," you have no audit trail and no fleet consistency.

How it works

Layer the kinds of config.

Build-time / image: rarely changes (compiler flags). Baked in. Rollback = old image.

Deploy-time: env vars, Kubernetes ConfigMaps, files mounted at start. Changing them usually restarts the process. Good for connection strings' non-secret parts, replica counts adjacent to the Deployment, log level if you accept a bounce.

Runtime / dynamic: a client polls or watches a store (Consul, etcd, AppConfig, a feature-flag service). No restart. Good for kill switches, experiment weights, "stop calling vendor X." This is the sharpest knife.

GitOps: the store of truth is git. A controller applies it to the cluster. You get PRs, reviews, and git revert. Ansible/Chef on VMs is the older cousin: converge the box to a declared state.

Config vs flags vs secrets

Config: behavior the app is allowed to know (timeouts, URLs of public endpoints, pool sizes).

Feature flags: config with targeting (user, percentage) and a product workflow. Same delivery pipes; different UX.

Secrets: next post. Do not put them in ConfigMaps, git, or Slack.

Schema and defaults matter. Unknown keys should fail loud in staging. Missing keys should not silently mean "unlimited."

A simple example

You need to cut off a flaky SMS vendor in two minutes. If the vendor URL is compiled in, you need an emergency deploy. If it is a ConfigMap, you need a rolling restart of every sender. If it is a dynamic flag sms.vendor = none, each replica picks it up in one poll interval. You still want: who changed it, a default of "on," and a rollback that does not require finding the previous Slack message.

A worse example: DATABASE_HOST copied from a .env.example into Kubernetes by hand. Staging host in prod. GitOps with overlays (base, staging, prod) makes that diff visible in the PR.

Common mistakes

ConfigMaps for passwords. They are not secret; they show up in etcd and in kubectl describe. Use a secrets manager.

Hot-reloading everything. A new DB URL mid-request is a split brain. Some keys must bounce; some must not. Document which.

No versioning. The live value is "whatever is in the console." You cannot answer "what was checkout's timeout at 14:02?"

One giant JSON blob. Every change invalidates everything and everyone conflicts. Split by service and by blast radius.

Forgetting clients cache config. You rolled back the store; instances still have the poison value for 5 minutes. Set TTLs and a force-refresh path.

How this shows up in real systems

Kubernetes ConfigMaps + GitOps (Argo CD, Flux): the default for deploy-time config.
LaunchDarkly, Unleash, homegrown flags: runtime targeting.
AWS AppConfig, Consul, etcd: dynamic key-value with watch APIs.
Ansible, Puppet, Chef, Nix: still how many VM fleets converge.

Recap

Config is how settings reach processes, with review and rollback — not a file on a laptop.
Prefer git and restarts for dangerous keys; dynamic flags for kill switches.
Keep secrets out of this store. That is the next post.