Secrets Management
- Pradeep P
- 3 days ago
- 3 min read
Layer 5 · Post 8 of 12
← Previous: Configuration Management → Next: Autoscaling
Layer 5 — Modern infrastructure · Post 58 of 88
Secrets management stores passwords, tokens, and keys outside the codebase, with rotation, access control, and audit trails.
What you'll learn
What counts as a secret, and why git and ConfigMaps are the wrong home
How injection (env, files, sidecars) and identity (IAM, service accounts) fit
Rotation, least privilege, and what to say when an interviewer asks about leaks
The idea in one minute
A secret is a credential the app needs and an attacker would love: DB passwords, API tokens, TLS private keys, signing keys. Secrets management keeps them out of source, out of Slack, and out of world-readable files — and records who fetched them.
App identity (IAM / k8s SA) | v [ Vault / cloud SM ] --short-lived creds--> process memory / tmpfs | audit log
The store encrypts at rest. The app proves who it is, not "who knows this one shared password we emailed."
Why it matters
The most common design fail is DATABASE_PASSWORD in git, in a Docker image, or in a Kubernetes Secret that is just base64. Base64 is encoding, not encryption.
A leaked repo or a kubectl get secret -o yaml in a ticket should not be a company-ending event. Rotation and blast-radius limits are how you make that true.
How it works
Store: HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, a KMS-backed cluster store. Someone (or a CI role) writes. Runtimes read with identity, not with a long-lived master password checked into Helm.
Identity: cloud instance roles, Kubernetes service accounts + IRSA/Workload Identity, Vault AppRole or JWT auth. The machine or Pod authenticates. Humans use SSO + break-glass.
Delivery:
Env vars: simple; easy to leak via /proc and crash dumps. OK for some apps.
Mounted files on a tmpfs: slightly better, rotatable by rewriting the file.
Sidecar / agent that refreshes: Vault agent, CSI driver. The process reads a file that updates.
Encryption: KMS for the envelope. The secrets manager uses it; you rarely invent AES in the app.
Rotation: short-lived DB creds (Vault dynamic secrets, IAM auth to RDS) beat a password last changed in 2019. For static secrets, a job rotates, rolls the fleet, then invalidates the old value. Both sides of a handshake must accept overlap, or you downtime yourself.
Kubernetes Secrets are a convenience object. Enable encryption at rest, RBAC them hard, and prefer an external manager as the source of truth. Do not treat "it's a Secret" as "it's safe."
A simple example
Payments needs a card-processor API key. Bad: in .env committed to GitHub, copied into the image. Better: the key lives in Secrets Manager. The payments Pod uses a service account that may GetSecretValue on that one ARN. At start, the CSI driver mounts the file. A leak of the repo does not leak the key. When an engineer leaves, you rotate the key, not "hope they forgot it."
Dynamic DB creds: Vault issues payments-readonly with a 1-hour TTL. The leaked credential self-expires. That is the interview-grade answer for high-value data.
Common mistakes
Git, CI logs, and Docker layers. COPY .env, echo $TOKEN in a build, a secret in an image history. Scan; use build-time secret mounts.
One vault policy for the whole cluster. Checkout can read the signing key for payouts. Least privilege per identity.
Never rotating because "it would break." Then a leak is forever. Design overlap windows.
Logging config dumps. Frameworks that print env on boot will print the password. Redact.
Sharing prod secrets to staging "just to test." Now staging is a prod credential store with weaker controls.
How this shows up in real systems
Vault, cloud secret managers, SOPS + KMS in git (encrypted files, still reviewed in PRs).
Sealed Secrets / External Secrets Operator: Kubernetes objects synced from a real manager.
Cert-manager: TLS certs as a secrets problem with automatic rotation.
Recap
Secrets live in a manager with identity, encryption, and audit — not in git or base64 YAML.
Prefer short-lived, least-privilege credentials and a rotation story.
Delivery is mount or inject; never bake into the image.
With images, config, and secrets in place, capacity can change with load. That is autoscaling.
Layer 5 · Post 8 of 12
← Previous: Configuration Management → Next: Autoscaling
Comments