Webhook Delivery System Design Interview: Signatures, Retries, and Tenant Isolation

A webhook that returned 200 can still be wrong, and a webhook that timed out may already have succeeded. The design must make authenticity, acceptance, processing, and business effect independently observable. A good answer does not begin with a fashionable tool name. It begins with a contract, draws the state boundaries, and shows how failure changes the design.

This guide gives you a reusable interview method plus concrete traces for webhook delivery system design interview.

Webhook Delivery System Design Interview: Quick Answer

Sign canonical delivery bytes, store events durably before sending, assign stable event and attempt identities, retry with backoff and jitter, make receivers idempotent, and enforce tenant scope at every queue, secret, log, and replay boundary.

Official documentation: Stripe documents signature verification over the raw request body, recommends quick 2xx responses followed by asynchronous processing, retries failed deliveries, warns that event order is not guaranteed, and tells receivers to handle duplicate events. Those facts motivate an at-least-once design; exact schedules remain provider-specific.

Candidate reports: We did not find a reliable, same-cycle set of public candidate reports establishing a standard interview format for this topic.

Engineering inference: The interview structure and recommendations in this guide are derived from the documented semantics and common failure modes. They are not an employer-wide hiring rule.

Draw the state boundaries first

Boundary	What it contains	Interview consequence
Event	Stable business occurrence ID and tenant	Immutable payload version
Attempt	One HTTP delivery with timestamp and signature	New attempt ID and retry count
Receipt	Receiver durably accepted bytes	Return 2xx after persistence
Effect	Business handler changed state	Deduplicate by tenant and event identity

This table is the first article-specific asset. Point to a row whenever the interviewer changes the scenario. If a value crosses a boundary, say how it is identified, serialized, authorized, versioned, and cleaned up. If it does not cross, say why the process can safely reconstruct it.

A boundary is also where partial failure becomes visible. The caller may know it sent a request while the receiver knows it committed a change; neither observation alone proves the end-to-end outcome. Keep attempted action, durable result, and later verification as separate facts.

Turn the prompt into an explicit contract

Before proposing an implementation, ask what must be correct and what may be approximate, delayed, retried, or discarded. Name the unit of identity, the lifetime of the state, the isolation boundary, and the evidence that counts as success. These questions prevent a polished solution from answering a different problem.

A useful answer also names ownership. Who creates the state? Who may mutate it? Who cleans it up? What happens when the process, request, or connection ends early? Interviewers often add a failure after the happy path; an ownership model lets you extend the design without replacing it.

Explain the invariant before the mechanism

The invariant is the sentence that must remain true after every transition. State it before code. Then walk one normal case, one boundary case, and one failure case. This makes the answer reviewable: the interviewer can see why a line exists and which assumption would invalidate it.

Avoid claiming a mechanism is safe or fast by name alone. A cache can miss, a queue can duplicate, a lock can block, and an async task can fail unobserved. Connect the mechanism to the stated contract and include the metadata or measurement needed to prove it at runtime.

Work through a concrete scenario

The sender times out after the receiver commits an invoice update but before the 2xx reaches it. The sender retries with a fresh timestamp and signature but the same stable event ID. The receiver’s unique (tenant_id, provider, event_id) inbox key turns the second receipt into a safe no-op while preserving the attempt audit.

Do the arithmetic or state trace aloud. Label every number that is an assumption and every behavior that comes from documentation. If the interviewer changes scale, platform, or failure semantics, update only the affected term. This shows that you understand the model instead of memorizing a configuration.

The supporting visual is the second topic-specific asset. Read it left to right: establish identity and preconditions, perform one bounded transition, collect the result, and verify durable state. When the result is unknown, prefer a read or reconciliation step before repeating a side effect.

Show a small implementation skeleton

BEGIN;
INSERT INTO webhook_inbox(tenant_id, provider, event_id, raw_body)
VALUES (:tenant, :provider, :event_id, :raw)
ON CONFLICT (tenant_id, provider, event_id) DO NOTHING;
COMMIT;
-- enqueue processing only when the insert created a new inbox row

This is deliberately a skeleton, not production-ready code. In an interview, explain which functions are pure, which perform I/O, which values carry identity, and where errors cross the boundary. Mention tests before adding framework details.

The most important code review question is: “What fact makes the next transition safe?” If the answer is only “the previous call returned,” add validation. If the action can be retried, define whether the same identity repeats the same effect, resumes work, or creates a new attempt.

Rehearse one failure before adding complexity

Start with the smallest version of the system that can demonstrate the invariant. Record the state before the action, introduce parse json before verification, and observe the concrete failure: whitespace or encoding changes signed bytes. Apply the narrow repair—verify the exact raw body first—then repeat the same input. The before-and-after evidence should differ only at the repaired boundary.

Next inject a second, independent fault: use one secret for every tenant. The expected failure is a leak expands the blast radius. Repair it by following this rule: Store versioned per-endpoint secrets. This two-step exercise shows whether the design composes. If the first repair masks the second failure, add an observation point rather than assuming the second path ran.

In a live interview, narrate the distinction between prevention, detection, and recovery. Prevention rejects an invalid transition before it changes state. Detection proves that an attempted transition produced an unacceptable result. Recovery moves from that observed result to a known state. A single mechanism rarely supplies all three.

Also name what you would leave out of the first implementation. Extra brokers, caches, services, or abstractions create more state transitions and more failure combinations. Add one only when a stated throughput, isolation, ownership, or recovery requirement needs it. That trade-off demonstrates judgment without hiding behind an oversized architecture.

Diagnose the misleading answers

Tempting answer	Failure	Better interview response
Parse JSON before verification	Whitespace or encoding changes signed bytes	Verify the exact raw body first
Use one secret for every tenant	A leak expands the blast radius	Store versioned per-endpoint secrets
Retry immediately and forever	Outages create retry storms	Use capped exponential backoff and jitter
Require global event order	Distributed delivery can reorder	Order only within a defined aggregate when needed

These counterexamples are more useful than a long list of trivia. For each one, reproduce the smallest failing case, name the violated invariant, repair the narrow boundary, and rerun the check. This approach also keeps you from changing several variables at once.

Measure the behavior you actually promised

Track accepted, verified, rejected, delivered, retried, deduplicated, processed, dead-lettered, and replayed events by tenant. Measure time to first attempt, time to successful receipt, handler lag, attempt distribution, and backlog age. Alerts need tenant-safe labels and redacted payloads.

Do not report a percentage improvement without naming the workload, baseline, sample window, and measurement point. Separate average behavior from tail behavior and separate task failures from infrastructure failures. A design can lower average latency while making recovery or isolation worse.

For an interview experiment, use a deterministic fixture with a fixed seed and record inputs plus versions. Run the baseline and proposed design on the same fixture. Add one fault at a time—timeout, duplicate, stale state, invalid identity, cancellation, or capacity overflow—and assert both the immediate result and the durable state afterward.

Use this answer structure in the interview

1. Restate the contract. Define input, output, identity, lifetime, failure semantics, and scale.
2. Name the invariant. Give one sentence that must survive every transition.
3. Draw boundaries. Separate caller, worker or service, durable state, and external side effects.
4. Walk one example. Use concrete values and show the state before and after each step.
5. Inject failure. Choose timeout, crash, duplicate, stale data, or cancellation and recover without guessing.
6. Measure and test. Name observable counters, a baseline, and an adversarial test.
7. State the trade-off. Explain what your design optimizes and what it deliberately spends.

This sequence is an editorial recommendation based on the mechanics above. It is not an official interview rubric. Its advantage is that every claim can be challenged locally: the interviewer can change a constraint and you can revise the affected boundary instead of rebuilding the answer.

Questions interviewers may ask next

What should I clarify before writing code?

Ask about correctness, ordering, identity, concurrency, platform or provider version, memory and latency limits, cancellation, security scope, and how success is verified. If the prompt omits one, state a reasonable assumption and make it easy to change.

How do I distinguish a fact from an inference?

A fact should point to current official documentation or a reproducible program result. A candidate report describes one person’s prompt or experience. An inference connects those facts to a design recommendation. Label the category when the claim first appears, especially for changing platform behavior.

What makes an answer senior-level?

Senior answers make ownership and failure explicit. They identify the source of truth, define what happens after an unknown result, protect tenant or user boundaries, and propose measurements that could disprove the design. More components do not automatically make the answer stronger.

Should I memorize every API detail?

No. Memorize the state model and a few diagnostic examples. State version-sensitive details carefully, cite current documentation when writing, and say how you would verify an uncertain API during real work.

Final preparation checklist

• Define the unit of identity and the source of truth.
• Separate current observation from durable state.
• State the main invariant before code.
• Trace one normal, boundary, and failure case.
• Explain ownership, cleanup, cancellation, and retries.
• Preserve tenant or user authorization at every boundary.
• Name version-sensitive claims and their official source.
• Measure the promised outcome and its failure surface.
• Avoid turning a candidate-reported prompt into an employer-wide claim.
• End with one clear trade-off and one test that could falsify your design.

Sources and Further Reading

• Stripe: Receive webhook events
• Stripe: Resolve signature verification errors