Sidecar Pattern

Pradeep P
2 days ago
3 min read

Series: Modern System Design · Layer 5 — Modern infrastructure

Layer 5 · Post 13 of 15

← Previous: Bloom Filters → Next: Cell-Based Architecture

Layer 5 — Modern infrastructure · Post 112 of 119

Run a companion container (proxy, agent) beside the app so mesh, secrets, and telemetry are not baked into the binary.

What you'll learn

What a sidecar is allowed to do, and what must stay in the app
Why service meshes standardized on this (and why "ambient" meshes exist)
The cost: extra CPU, extra hop, extra failure domain

The idea in one minute

A sidecar is a process (usually a container in the same pod) that shares fate with the app: same host network namespace (or loopback), same lifecycle. It is not a remote dependency. It is a helper bolted to the bike.

Typical jobs: proxy all traffic (Envoy), ship logs, refresh certs, talk to a vault. The app speaks localhost. The sidecar speaks mTLS, retries, and the control plane.

[ Pod ]
  app  <-->  127.0.0.1:envoy  <-->  network
  log-shipper -->  disk / stdout

Post 54 (service meshes) is the famous sidecar. The pattern is older: logstash next to the JVM, Consul agent next to the binary.

Why it matters

You cannot rewrite every service in every language to get consistent TLS and metrics. A sidecar is polyglot infrastructure. Interviews: "how does the mesh work?" — sidecar intercepts; control plane programs it.

It is also how you get a new outage shape: Envoy config bad → all pods in the namespace fail together. The sidecar is on the request path. Treat it like production software, not a daemon you ignore.

How it works

Same pod, shared volumes / network. Kubernetes: two containers, one Pod. Shutdown order matters: stop taking traffic (proxy drain) then kill the app — or the opposite, depending on inbound vs outbound. Get this wrong and you drop in-flight requests.

Traffic hijack. iptables / eBPF redirect, or the app is configured to proxy explicitly. Implicit hijack is convenient and surprising.

Not a sidecar: a node agent (DaemonSet) shared by all pods on the box (CNI, log agent). Cheaper; less isolation; one agent bug hits every app. Library instrumentation (OpenTelemetry SDK) is in the process — no extra hop, language-specific.

Ambient / sidecar-less mesh. Some meshes move the proxy to the node or to waypoints to cut per-pod cost. Same pattern at a different granularity: still "not in the app binary."

What not to put in the sidecar. Business rules, user authz that needs app state, "the checkout discount." If the helper needs to understand orders, it is a BFF or a service, not a sidecar.

A simple example

Java payments service. You want mTLS to users and retries on 503.

Without sidecar: Java HTTP client config, cert reload in Java, every language copies it. Python team does it differently. One team forgets timeouts.

With Envoy sidecar: Java calls http://users. Envoy adds mTLS, retries, timeouts from mesh config. A new Go service gets the same policy by joining the mesh. Cost: ~50–100 MB RAM × thousands of pods, and a second process to page on.

When Envoy's cluster config points at the wrong EDS, Java is fine and traffic is dead. Debug the sidecar.

Common mistakes

Sidecar per feature (one for logs, one for metrics, one for mesh, one for vault) until the pod is a zoo. Combine where you can; DaemonSet for node-level.

No resource limits on Envoy. The helper OOMs the app via cgroup. Limit and budget.

Ignoring drain. K8s preStop and terminationGracePeriodSeconds exist for this pattern.

Sidecar that must be up before the app, with a circular wait. Startup probes and depends need a story; startupProbe on the app should not assume the proxy is already intercepting unless it is.

How this shows up in real systems

Istio, Linkerd, Consul Connect, AWS App Mesh: Envoy (or similar) sidecars.
Cloud SQL / RDS proxy, Vault agent injector, OpenTelemetry Collector as sidecar.
Service Fabric / classic IIS "sidecar" executables on VMs — same idea without pods.
Knative / service meshes "ambient": the backlash when sidecar cost dominates.