Job Scheduler System Design Interview Guide: Queues, Retries, and Exactly-Once Execution

A distributed job scheduler needs two separate guarantees: durable knowledge of which work should run, and recoverable execution when workers or networks fail. Start with an authoritative schedule store, immutable logical runs, a transactional enqueue intent, ready queues and time-bounded worker ownership. Expect duplicate delivery and explain how each handler contains it.

“Exactly once” needs a boundary. A queue cannot independently guarantee that an arbitrary external action happens once. A worker may complete that action and crash before recording or acknowledging it. The design below demonstrates that window with a stable execution ID, changing lease epochs and an explicit reconciliation state.

Use the system design practice collection to change the requirements after your first walkthrough. This original scheduler exercise is preparation material, not a vendor reference architecture or a claimed company interview question.

Clarify which failures the product can tolerate

Ask about immediate jobs, one-time schedules, recurring jobs, timing precision, execution deadlines, maximum runtime, time zones, dependencies, cancellation and tenant limits. Is a late occurrence still useful? Would duplicate work be harmless, expensive or irreversible? Those answers determine recovery policy.

Consider these fictional scale inputs: ten million active schedules, 100,000 runs due per minute, a tenfold top-of-hour burst and two-second average runtime. Average arrival rate is about 1,667 runs/second; the burst is about 16,667. At 80% target slot utilization, a simple steady-state estimate needs roughly 4,167 average or 41,667 burst execution slots. This is a planning calculation, not a measured capacity limit. Runtime variation, retries, downstream quotas and startup time can change it substantially.

Do not solve that scale problem by placing every future job in an in-memory timer. Durable scheduling and ready-work distribution have different responsibilities. Google SRE's distributed-cron discussion also distinguishes the consequences of missed and duplicate launches; a single policy does not fit every job class.

Separate definitions, runs and attempts

Record	Example identity	Owns
Job definition	invoice_57, schedule version v3	Recurrence, handler, tenant, permissions and policy
Logical run	run_57_v3_20261008T120000Z	One intended occurrence and its business outcome
Worker attempt	attempt_A8 or attempt_B9	Worker, lease epoch, timing and error history
Enqueue outbox	enqueue_o100	Durable intention to publish this run
Completion inbox	The logical run ID	Deduplication of a local business mutation
Completion outbox	complete_o101	Durable intention to publish its result

The logical run ID stays the same across retries; the attempt ID and lease epoch change. A queue message ID is transport identity, not business identity. Two published messages can refer to the same run.

For recurrence, enforce a unique occurrence key such as (job_id, schedule_version, scheduled_instant_utc). Keep dispatched occurrences immutable when an administrator edits the definition. Specify whether cancellation applies to future materialization, queued runs, running work or all three.

A useful run state machine is SCHEDULED → READY → LEASED → SUCCEEDED, with RETRY_WAIT, DEAD, CANCELLED and RECONCILING branches. Keep the attempt history when a run returns to READY. A transport timeout alone does not establish business failure.

How does due work reach a worker safely?

The schedule API commits the definition to durable storage. A materializer finds due occurrences and commits the Run plus enqueue outbox row in one transaction. A dispatcher publishes the outbox row, then records its publication. If it crashes after publishing but before recording success, it may publish again. Consumers must tolerate that duplicate.

For a modest system, use an indexed due-time query with short transactions. PostgreSQL documents SKIP LOCKED for queue-like consumers while warning that it gives an inconsistent view. It helps competing consumers avoid a locked row; it does not establish that every due occurrence was discovered, or replace a unique run constraint.

At larger scale, partition the due-time index by time range and stable job/tenant hash. Promote a bounded near-term horizon instead of scanning every definition. A heap or timing wheel may help local dispatch, but its contents must be recoverable from authoritative storage. Monitor the oldest due occurrence, not just scan throughput.

The transactional outbox addresses the local database-to-message gap. AWS's pattern guidance explicitly calls out duplicate publication and idempotent consumption. It does not make a separate provider's side effect atomic with your scheduler database.

Crash-and-redelivery trace: one run, two owners

The following is a fictional supplied timeline for one scheduled invoice action. The external mock destination supports durable deduplication by logical run ID with parameter matching; assume its retention covers the entire recovery period. That assumption will be removed in the next section. Times are relative to 12:00:00 UTC, and the lease authority uses its own clock. Epoch numbers illustrate increasing ownership generations, not counts of attempts.

Time	Event and identifier	Authoritative state	Consequence
0.000 s	Commit Run + enqueue_o100	Run READY, epoch 7	Work exists even if dispatch stops
0.100 s	Publish queue_m1; dispatcher crashes	Outbox still unconfirmed	Same enqueue intent can publish again
1.000 s	A claims attempt_A8, epoch 8, expiry 5.000	Run LEASED by A	Lease acquired conditionally
2.000 s	Duplicate queue_m2 delivered to B	A's lease still valid	B cannot acquire another active lease
4.800 s	Destination commits receipt receipt_900, key = run ID	Provider has one effect; local completion absent	Response is lost
5.100 s	A observes timeout after its lease expires	Local outcome unresolved	A cannot assert success from a timeout
5.200 s	B receives redelivery and claims attempt_B9, epoch 9, expiry 9.200	B owns current lease	Same run; new attempt
5.300 s	A gets a late receipt and tries completion with epoch 8	Conditional completion rejects A	Old worker cannot overwrite B's state
5.400 s	B retries the same destination operation	Provider returns receipt_900	No second external effect under stated contract
5.500 s	B commits inbox, local update, success + complete_o101	Run SUCCEEDED, receipt stored	Local completion durable
5.600 s	B crashes before acknowledging its delivery	Message remains unacknowledged	Queue may deliver again
6.000 s	C receives another delivery	Terminal Run already SUCCEEDED	Acknowledge this delivery; do not rerun handler

At 5.300 seconds, A's completion is rejected because its epoch is stale. The fence protects the local completion write. It did not stop A's earlier provider action, and it cannot stop a later external action unless that destination participates in the ownership protocol. The stable run key, not the local fence, contains duplicate external effects in this scenario.

What must be atomic in the completion transaction?

After B obtains the verified receipt, begin a local transaction and conditionally reserve/complete the still-current run. Require LEASED state, owner B, epoch 9 and an unexpired lease according to the authority's clock. If no row matches, stop and roll back; do not mutate business state first and check ownership afterwards.

Within that same transaction, insert the unique inbox key, apply the local business mutation once, store the provider receipt and final Run state, and create the completion outbox record. A duplicate inbox entry must be checked against the stored operation/result, not treated as arbitrary evidence of success. Commit all of those records together or none.

For example, marking the local invoice settled and inserting the inbox in different transactions reopens a crash window. The inbox can exist while the invoice remains unsettled, causing a retry to skip needed work. Reversing the order can duplicate the local mutation. The transaction is the guarantee; merely drawing two tables is not.

Use the successful transaction's durable state before acknowledging the queue delivery. If acknowledgement fails, a later consumer checks terminal state and acknowledges its own delivery without repeating the handler. If completion-outbox publication fails, retry that publication independently. Neither recovery should require executing the business action again.

These are intended transaction semantics for the interview design, not a tested database implementation. In an implementation review, request rollback/crash tests and actual conditional-write results before claiming they work across processes.

What changes if the external action has no idempotency contract?

Remove provider deduplication from the 4.800-second step. The destination might have sent an email or performed another non-queryable action before its response disappeared. B cannot safely infer “failed,” and a new attempt may repeat it. A local inbox written afterwards cannot retroactively prevent an external effect that already occurred.

Move the Run to RECONCILING. Retain the original execution ID, intended parameters, attempts, timestamps and any destination correlation ID. Choose the next action from the actual destination contract:

Destination capability	Recovery choice	Remaining limitation
Query authoritative status by operation ID	Query, confirm outcome, then complete locally	Handle a genuinely pending result
Durable idempotency with matching parameters	Retry the same operation identity	Recovery must remain within retention
Business operation and inbox share one database	Commit them atomically	Guarantee stays inside that transaction
No lookup, dedupe or shared transaction	Pause for reconciliation or accept declared duplicate risk	No universal exactly-once claim

“Not found” is not automatically permission to retry: a lookup can race an in-flight commit or read stale data. Ask whether the lookup is authoritative and whether it reports pending operations. Record the recovery decision and evidence. If an operator deliberately starts a new logical run, give it a new occurrence identity and audit its relationship to the original.

If the destination expires idempotency records, keep your local deduplication longer than the maximum redrive horizon. Even then, a local unresolved outcome cannot become safe to retry merely because your database retains the key. The external contract still matters.

Leases, heartbeats and cancellation: where does authority end?

A lease expiry permits recovery; it does not kill a paused process. Heartbeats extend ownership only while the worker still matches the current epoch and owner. Never let an old heartbeat revive an expired epoch after reassignment. Renewals and completions need conditional writes against the same authority.

SQS visibility timeout illustrates temporarily hidden deliveries and eventual redelivery, but its documentation still allows duplicates during visibility. Separate that delivery mechanism from your durable Run lease. If the queue and database timers disagree, the conditional Run claim decides which worker may commit local state.

Cancellation has a similar boundary. Check authorization and current state before starting irreversible work, but acknowledge the race between that check and an external action. If cancellation arrives after the action has begun, return a defined “cancellation requested” or “too late” result and reconcile. Do not promise that a final pre-call check makes an unrelated provider atomically cancellable.

For long tasks, decide whether a lost lease should stop computation immediately or allow a result to be discarded. If a downstream resource supports fencing, send and enforce the epoch there too. Tokens that are logged but never checked provide no protection.

Retries, fairness and recurring-job policy

Classify errors before retrying. Invalid payloads or revoked authorization need correction, not a fast loop. Transient capacity failures may use bounded exponential backoff and jitter, constrained by attempt count, total age and the occurrence's usefulness deadline. Ambiguous external outcomes take the reconciliation path instead of an ordinary blind retry.

A dead-letter queue needs an owner, alerts, retained attempt history and an audited redrive policy. Preserve the logical run ID during retry/redrive. Inspect previously completed or unresolved state before re-enqueueing; “move everything back” can repeat business effects.

Use per-tenant concurrency and rate budgets, reserved capacity for urgent classes and aging to prevent starvation. Queue length alone hides a tenant waiting hours behind someone else's burst. Measure oldest ready age and dispatch latency by tenant and job class. Include retry traffic in admission planning.

For recurrence, store the rule and original time zone, then materialize explicit UTC instants. Define handling for missing and repeated local times at daylight-saving transitions. After downtime, choose skip, one catch-up run, every occurrence or expiry. Bound a catch-up wave so recovery cannot overload the destination. For multi-region failover, require authority transfer and fencing before a second region expands the same shard's schedules.

A 45-minute walkthrough with evidence

This is a suggested practice allocation, not an employer's interview format. Spend five minutes clarifying duplicate/missed-run cost and scale; ten defining models and the due-work path; ten walking the epoch-eight/epoch-nine timeline; ten removing destination idempotency and explaining reconciliation; and ten covering retry limits, fairness, recurrence and metrics.

Keep five concrete outputs: an API/data model, the occurrence uniqueness constraint, the completed crash trace, the ambiguous-outcome recovery table and a failure-injection plan. That plan should kill a dispatcher after publication, resume a stale worker, crash completion before commit and lose acknowledgement after commit. State the durable records expected in each case.

Practise the existing prompts “Design a distributed job scheduler,” “Design distributed message queue service,” “Design delayed job scheduler (LLD),” “Design a Reliable Job Scheduler” and “Design a Reliable Scheduler for Payment Jobs.” Change one constraint each time; these titles represent practice topics, not a prediction of your assessment.

Frequently asked questions

Can the scheduler guarantee exactly-once external execution?

Only within a defined cooperating transaction or destination contract. Arbitrary external effects remain ambiguous after a lost response. Separate repeated attempts from one logical business outcome.

Does a lease prevent two workers from running?

It controls accepted ownership, but an expired worker may still be alive. Conditional completion and enforced destination fencing protect participating resources; a timer alone does not stop a process.

Is an outbox enough to prevent duplicate work?

No. It preserves publication intent alongside the local database change. Dispatch can still duplicate messages, so consumers and handlers need their own identity and recovery rules.

When should an unresolved run be redriven?

After its outcome has been reconciled or the business has explicitly accepted the documented risk. A new queue delivery is not evidence that the previous external attempt failed.

Sources and research scope

• Google SRE distributed periodic scheduling: differing missed-run and duplicate-run consequences.
• Amazon SQS visibility timeout: delivery visibility and duplicate-delivery boundaries.
• AWS transactional outbox pattern: local dual-write recovery and duplicate notifications.
• PostgreSQL SELECT documentation: SKIP LOCKED behavior and its consistency caveat.

Sources were checked October 8, 2026. The capacity inputs, identifiers, timelines and transaction design are original editorial exercises. They were reviewed for consistent ownership and recovery boundaries; no distributed scheduler, real payment or database fault-injection test was run for this article.