Design a Distributed Job Scheduler

System Design

Problem

0/17

Clarified job types (one-off vs recurring/cron) and execution guarantee (at-most-once) before designing
Captured functional scope: create/reschedule/cancel, retries, SLA/timeout detection, status/log query, failure notifications
Stated concrete non-functional targets: scale (millions of jobs), dispatch latency, durability/retention

Separated job *definition* storage (relational, strongly consistent) from job *execution* path (queue + worker fleet)
Included a Scheduler Fleet that polls due jobs via an indexed query rather than holding schedule state in memory
Used a partitioned/durable task queue to decouple scheduling from execution
Chose an appropriate store for run history (high write throughput, append-mostly) distinct from the job definitions store
Defined REST APIs covering create, reschedule/cancel, status query, run history query, and worker heartbeat/complete callbacks

Explained at-most-once execution via distributed locking on dispatch + lease/TTL fencing on execution, with idempotency pushed to run_id/attempt
Addressed recurring jobs: efficient next-run computation/indexing and avoiding cadence drift (recompute at dispatch, not completion)
Covered missed SLA/timeout detection and crash/restart recovery via lease expiry scans, with no reliance on in-memory scheduler state
Designed retry/backoff policy with a hard cutoff and dead-letter/final-failure notification (not per-attempt spam)
Discussed horizontal scaling approach (sharding jobs/schedulers, partitioned queue, caching hot status) to reach millions of tasks
Articulated at least 3-4 explicit trade-offs (e.g., exactly-once impossibility, polling vs push, SQL vs NoSQL choice)

Delivered a clear, structured opening statement framing the design approach
Managed time across requirements/high-level/deep-dive/wrap-up phases appropriately for 45 minutes
Proactively named failure modes (crash, restart, poison job) rather than waiting to be asked

Loading canvas…