Designing a Distributed Scheduler

Pradeep P
3 days ago
4 min read

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 6 of 26

← Previous: Designing a Feature Flag System → Next: Designing a Payment System

Layer 6 — Modern systems · Post 68 of 88

A distributed scheduler runs jobs at the right time, exactly once (or close), even when workers crash and clocks disagree.

What you'll learn

Why cron on one box is not a scheduler, and what "lease + visible job" replaces it with
How a job store, dispatchers, and workers coordinate without double-running
Why "exactly once" is a lie you approximate with idempotent handlers and fencing

The idea in one minute

A distributed scheduler is: persist "run this handler at time T with payload P," then have many workers compete to run due jobs without two of them doing the side effect twice.

API / crontab-like UI | v [ Job store: due_at, status, attempts, lease ] | v [ Dispatcher / poller shard ] --claim (lease)--> [ Workers ] | | | heartbeat lease +--> handler (HTTP, queue, script) v worker death? lease expires, job becomes visible again clock skew? store uses DB time / logical due, not worker wall clock alone

Recurring jobs are "on success, insert the next due_at." One-shots are deleted or marked terminal.

Why it matters

Billing runs, reminder emails, report generation, ML batch kicks, and "delete expired sessions" all need this. Interviewers listen for at-least-once delivery, idempotency, and what happens when a worker dies after doing the work but before acking.

If you say "we'll use Kubernetes CronJobs" that can be a valid piece — still explain duplicate runs when two replicas schedule, or when a job is retried.

How it works

Clients / APIs create jobs: handler, payload, due_at, idempotency_key, optional cron expression. The store is the source of truth (Postgres with FOR UPDATE SKIP LOCKED, or a queue with delayed messages, or ZooKeeper/etcd for small control planes).
Partitioning. You shard by job ID or by time buckets so pollers do not all scan the same rows. Each dispatcher owns a slice.
Claim. A worker (or dispatcher) transactionally picks due jobs: set status=running, locked_until=now+lease, locked_by=worker_id. SKIP LOCKED lets others skip in-flight rows.
Execute. Worker runs the handler. Heartbeats extend the lease if the job is long. On success: status=done (and enqueue next occurrence). On failure: increment attempts, backoff due_at, or dead-letter.
Failure. Process killed: lease expires, another worker claims. If the first worker actually completed (sent the email) but died before done, you run twice. Handlers must be idempotent (same idempotency_key into the notification or payment system).
Clocks. Do not trust datetime.now() on a random VM for "is it due." Use store time, or NTP-disciplined clocks plus slack. For "run once globally," a lease in the store beats three crons in three regions.

Exactly-once across a crash is not something you get from the scheduler alone. You get at-least-once + idempotent work, sometimes with a fencing token so a late worker cannot commit after losing the lease.

A simple example

You schedule "dunning email for invoice_55 at 2026-09-01 10:00 UTC." At 10:00 a poller claims the row with a 60s lease. The worker calls the notification service with key dunning_invoice_55. It succeeds, then the VM is preempted before status=done. At 10:01 the lease expires. Another worker claims, calls notifications again, and the notification system no-ops on the same key. The scheduler shows two attempts; the user got one email.

A naive crontab on two app servers both fire 0 10 * * * — two emails unless the handler is idempotent. The distributed design makes the duplicate a defined retry, not a surprise.

Common mistakes

Cron on every replica. You get N executions. Use a leader or a shared job store.

No lease heartbeat on long jobs. Another worker starts a second copy at locked_until.

Using the worker's clock for due time across regions with skew. The store should decide "due."

Promising exactly-once without idempotent downstream. Say at-least-once clearly.

One hot row for "the global cron." Shard, or you serialize the whole company on one lock.

How this shows up in real systems

Celery Beat, Sidekiq Cron, Quartz, Temporal/Cadence, Airflow: different layers (task queues vs workflow vs DAG), same claim/lease ideas.
Cloud: EventBridge Scheduler, Cloud Scheduler, Cloud Tasks, SQS delay: delayed messages as the store.
Kubernetes CronJob: fine for coarse jobs; still design idempotency.

Layer 3 queues often are the worker side. Payments (next) are the handlers you most want idempotent.

Recap

Persist jobs, claim with a lease, run, then ack — duplicates on crash are expected.
Idempotent handlers (and fencing if you need it) approximate exactly-once.
Shard polling and do not trust a single box's crontab or clock.

When the job is "charge this card," the scheduler is the easy part. The ledger is not.

Series: Modern System Design · Layer 6 — Modern systems

Layer 6 · Post 6 of 26

← Previous: Designing a Feature Flag System → Next: Designing a Payment System