Celery Interview Questions: Retries, Acknowledgements, and Worker Failures

Celery interview questions about failures require three separate answers: what happened to the task attempt, what happened to the broker message, and what happened to the business operation. A failed task can already have committed its side effect. A late acknowledgement setting does not automatically turn an exception into a retry.

This guide follows a notification-recording task through controlled failures. Official Celery 5.6.3 documentation defines the contracts; the local experiment uses a real prefork worker and Redis rather than eager mode. No real notifications are sent. The questions are editorial preparation, not claims about a particular employer's interview.

What Happens When a Celery Task Fails?

First locate the failure. An ordinary Python exception, an explicit retry request, a terminated pool child, and loss of the entire worker are different events. They can produce different delivery behavior even when the task function begins with the same code.

An acknowledgement tells the broker that a delivery has been handled according to the consumer's policy. It is not a receipt proving the business operation happened exactly once. Result-backend state is another record again: seeing FAILURE tells you about an attempt's outcome, not that its database transaction was rolled back.

The official Tasks guide explains early and late acknowledgement and the child-termination exception. Read those together before summarizing the feature as “delete the message only after success.” That shorthand loses the failure policy that matters in this article.

Our task writes a harmless SQLite notification-intent row before triggering each fault. This intentionally creates evidence that durable work can precede a failure. In production, the corresponding boundary might be a committed order change or a provider accepting a request before the connection times out.

How Do retry and acks_late Differ?

self.retry() requests another execution by publishing a new message. Celery normally preserves the task ID for that retry. acks_late changes acknowledgement timing for the current delivery; it does not identify which application exceptions deserve another attempt.

The Celery FAQ discusses why retry and late acknowledgement serve different purposes. A useful answer names the recoverable error and the delivery policy separately. Combining the mechanisms can make sense, but neither substitutes for the other's responsibility.

A bound task can request one retry like this:

@app.task(bind=True, max_retries=1)
def notify(self, operation_key):
    try:
        record_notification(operation_key)
    except TemporaryProviderError as exc:
        raise self.retry(exc=exc, countdown=5)

This is a design snippet, not the exact failure injector used in the lab. The lab deliberately requests a retry after committing its local row. That harsher sequence tests whether the business identity survives another execution.

Celery's Retry exception is control flow used to signal retry handling. Do not catch it accidentally with a broad exception wrapper that converts it into an unrelated success or error. Likewise, avoid wrapping every exception in automatic retry: malformed input usually needs a permanent failure response rather than repeated load.

Fix the Version and Configuration Before Predicting

The local fixture pins Celery 5.6.3, Python 3.12, Redis server 7.4.2, and a prefork pool with concurrency one. It uses a dedicated loopback Redis port and separate broker/result databases. The SQLite store is a test artifact, not a production database or external notification system.

These settings are explicit:

app.conf.update(
    task_acks_late=True,
    task_acks_on_failure_or_timeout=True,
    task_reject_on_worker_lost=False,
    worker_prefetch_multiplier=1,
    worker_cancel_long_running_tasks_on_connection_loss=False,
    broker_transport_options={"visibility_timeout": 5},
    result_backend_transport_options={"visibility_timeout": 5},
    visibility_timeout=5,
)

The short visibility timeout makes the isolated failure experiment easier to observe; it is not a production recommendation. A task whose execution exceeds a transport's visibility window can create a different redelivery problem. Choose settings against actual workload behavior and the relevant transport documentation.

For every attempt, the fixture records task ID, operation key, mode, child PID, retry count, and the redelivered flag. It separately records the durable notification intent under a unique business key. Those two tables let us distinguish multiple attempts from multiple business rows.

Capture worker logs and broker observations too. A statement such as “it disappeared” is ambiguous until you say whether you inspected ready messages, unacknowledged deliveries, result state, or committed data.

Why Does an Ordinary Exception Not Retry?

The exception variant commits the intent and raises ValueError. It has no automatic-retry decorator and does not call retry. With the fixture's acknowledgement-on-failure setting, the attempt finishes as a failure rather than becoming another scheduled application attempt.

This is the first counterexample to success-only acknowledgement. Late acknowledgement is enabled, but the configured failure path still acknowledges. Increasing max_retries alone would not cause this function to invoke retry; a retry policy must actually select the error.

The explicit-retry variant behaves differently. Its first attempt commits, sees retry count zero, and requests one immediate retry. The next attempt uses the same task ID with an incremented retry count and returns successfully. The business-key constraint prevents a second intent row.

A result backend can eventually show SUCCESS for a task that had an earlier retry. Preserve attempt-level logs if your diagnosis depends on the intermediate event. A single final status is not a complete execution history.

What Changes When the Pool Child Exits?

The child-loss variant commits the row and calls os._exit(17) inside the task. The parent worker remains alive and reports WorkerLostError. That is different from losing the whole consumer process and its broker connection.

With task_reject_on_worker_lost=False, Celery's documented behavior allows acknowledgement despite late acknowledgement being enabled. Setting the rejection option changes that policy, but it can also create repeated delivery of a task that deterministically kills every child that runs it.

The configuration reference documents these separate controls. Do not prescribe the rejection flag without asking whether the task is idempotent, whether the failure is repeatable, and how a poisonous job will be bounded or investigated.

The lab does not claim to test every timeout or signal combination. A hard time limit, a graceful shutdown, a process kill, and an exception have distinct paths. Naming the exact fault is part of a strong answer, not unnecessary detail.

What If the Entire Worker Dies After Commit?

For the whole-worker variant, the first attempt commits and then waits. The harness observes the durable row and an unacknowledged delivery, kills only its own test worker process group, and starts a replacement worker. The Redis broker remains running.

That setup isolates a specific uncertainty window: the side effect exists, while the delivery has not been acknowledged. It does not simulate Redis losing its own data, a network partition with an old worker still executing, or every production shutdown policy.

The Redis transport guide explains visibility-timeout redelivery. The configured timeout is not a promise that an observer sees a new attempt at exactly that second; restoration and polling also matter. Retain a bounded observation window and actual timestamps rather than inventing a precise recovery SLA.

Connection loss without process death raises another question: can the old execution keep producing effects while another delivery runs elsewhere? The cancellation setting addresses part of that behavior, but cannot undo an already committed transaction. This branch is documentation-supported reasoning, not a network-partition experiment performed here.

Can the Same Task Produce Duplicate Effects?

Yes. A task ID identifies Celery work; a business operation key identifies the logical effect you want to deduplicate. A caller can submit the same business request twice with different task IDs. Conversely, a retry can reuse an ID while the underlying action executes again.

For our local effect, the essential schema is:

CREATE TABLE effects(op TEXT PRIMARY KEY);

The task performs INSERT OR IGNORE using the operation key inside its SQLite transaction. An attempts table still records every execution. Therefore the evidence can show two attempts and one durable intent without pretending the second attempt never happened.

This proves a narrow property of the local database operation. It does not prove exactly-once email, SMS, payment, or webhook delivery. If you insert the key and then call an external service, the process can fail between those actions. If you call the service first, an ambiguous response can also leave you unsure whether the effect occurred.

A production design may use a transactional outbox to store work with the domain change, a provider-supported idempotency key, and reconciliation for uncertain responses. Explain which boundary each mechanism covers. A database uniqueness constraint cannot make a remote provider participate in the same transaction by itself.

Choose the business key carefully. For a due-date reminder, a renewal may represent a different logical event. A key based only on patron ID would suppress legitimate notifications. Include the event identity or version that your business contract says should distinguish one operation from another.

Read the Failure Evidence Before Choosing a Fix

Use this diagnostic matrix as a prediction exercise before examining the recorded run. The outcomes are specific to the configuration above; transport and policy changes require another check.

Fault	Attempt-level evidence to inspect	Business question
Python exception	FAILURE, no requested retry, acknowledgement policy	Did work commit before the exception?
Explicit retry	Same task ID, higher retry count, another message	Does the same key suppress a duplicate effect?
Prefork child exit	WorkerLostError while parent survives	Does worker-loss rejection apply?
Whole worker killed after commit	Unacknowledged delivery and replacement-worker attempt	Can redelivery safely repeat committed work?

Do not infer acknowledgement solely from result state. Correlate the worker's fault log, transport state, and attempt ledger. Even a zero unacknowledged count observed after completion is a point-in-time observation, not a universal durability guarantee.

When reporting a fix, say what changed and what stayed uncertain. “We added a unique operation key and verified duplicate database inserts were suppressed” is supportable. “We enabled late ack and now nothing can be lost or duplicated” is not.

The completed run recorded attempt counts of 1, 2, 1, and 2 for exception, explicit retry, child loss, and whole-worker loss respectively. Final states were FAILURE, SUCCESS, FAILURE, and SUCCESS. Each operation retained one effect row. The replacement worker's attempt kept the same task ID and retry count zero, but its redelivered flag changed to true; the explicit retry instead incremented retry count to one.

These observations distinguish broker redelivery from application retry using actual attempt metadata. The whole-worker case had one unacknowledged delivery before the kill and zero after recovery. The first observation window was too short; extending it captured restoration. That is why the configured five-second visibility value is not presented as an exact recovery timer.

What Should You Practice Next?

Explain the exception and child-exit cases aloud without using the word retry loosely. Then introduce the post-commit worker failure and identify why a repeated execution is acceptable only if the effect boundary is designed for it.

Question	Practice focus
Design Idempotent Due-Date and Hold Notifications	Define one logical notification and changing business state.
Implement an Idempotent Versioned Database Update	Combine version checks and operation identity.
Design an async API with idempotency	Separate accepted work from completed effects.
Design a Reliable Scheduler for Payment Jobs	Reason about recovery and ambiguous outcomes.
Implement retry wrapper and interdependent validators	Select recoverable failures and bound retries.

Return to the notification design question with one concrete failure timeline: commit, lose the worker, receive another attempt, and decide whether the business effect should change.

Sources and Further Reading

• Celery Tasks — task execution, acknowledgement, and retry behavior.
• Celery FAQ — retry versus late acknowledgement.
• Celery configuration — failure and worker-loss settings.
• Celery with Redis — visibility timeout and transport limitations.