cswork / libraryOpen practice workspace

Interview guides · Axa Partners

Axa Partners Software Engineer Interview Questions & Guide 2026

Software engineer interview prep for Axa Partners: original coding, system design and debugging practice, each with a worked solution.

The 60-second brief

A Software Engineer at Axa Partners plays a pivotal role in maintaining and evolving the digital infrastructure that supports the company’s global insurance and assistance services. You are not just writing code; you are building robust, scalable solutions that ensure seamless experiences for millions of customers who rely on Axa Partners during critical moments.

Browser-facing seats are not covered by algorithm practice. Scope in state ownership, what the page does on a slow or failed request, and how you would diagnose something that renders correctly but feels slow.

Focus areas

  • Scope every query by tenant below application code
  • Evolve schemas expand-contract across un-upgradable client deployments
  • Bound credential lifetime by the engagement end date

Overview

The work here is characterized by a blend of stability and technical modernization. You will contribute to product-focused teams, often collaborating across international borders to deliver features that improve operational efficiency and user outcomes. Whether you are working on backend services, internal tooling, or customer-facing applications, your contribution directly influences the technical health of the organization and the reliability of their service delivery.

The Axa Partners interview process

Axa Partners candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Application Review

Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.

What to demonstrate

  • Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
  • Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
  • Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled

How to prepare

  • Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
  • Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
  • Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market

HR Screening

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

  • Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
  • Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
  • Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

  • Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
  • Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
  • Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Hiring Manager Discussion

Expect the estimate to be probed harder than the outcome. A manager asking how long something will take is checking whether your number covers the parts that reliably go wrong: migrating the rows that already exist, and the second service that has to change at the same time. A single confident number is the weak answer. The strong one states what the estimate assumes, names the unknown that could double it, and says what you would cut first if the date were fixed. Scope is the variable you control; hours are not.

What to demonstrate

  • Whether an estimate arrives with its assumptions attached, including what it excludes and how much of it is waiting on review, environments or another team
  • Whether you can name work you cut and the person you told, since a cut nobody heard about surfaces later as a surprise rather than a decision
  • How you order two tasks that both look urgent, and whether that order survives being told the deadline moved in by a week
  • Whether you separate what must be done before launch from what must be done before the thing can be left running unattended

How to prepare

  • Take the last three things you shipped and write down, from memory, the estimate you gave and the time it actually took. Bring the largest miss and why it missed; a candidate who has never missed has never committed to a date.
  • Against your current queue, write who is blocked until each item lands. Anything with nobody behind it becomes your worked example of what you would drop and how you would say so.
  • Rehearse the fixed-date answer aloud: which slice ships, which slice is deferred and written down somewhere it will be found, and the one thing you would refuse to cut because removing it makes the release unsafe rather than smaller.

Where candidates lose points

Validating against a client sandbox and assuming production parity

Sandboxes typically carry smaller data, looser or absent rate limits, a schema version behind production, and sometimes synchronous behaviour where production is asynchronous. Code that passes there fails first in the client's production, during a change window you do not control and often cannot get a second one of. The mitigations are specific: assert the observed schema version at the boundary of every run, measure the production rate limit empirically rather than reading it from a document, and design the first production run to be a bounded, reversible slice rather than a full backfill.

Long-lived shared credentials for client systems

One static key used by the connector runtime, a debugging script and three engineers cannot be attributed to a person in an audit, cannot be expired at engagement close, and cannot be rotated without breaking an unknown number of consumers at once. The expensive part of the eventual incident is not the leak; it is that rotation requires finding every holder under time pressure, and nothing recorded who they were. Issue per-principal, per-target, short-lived grants from a broker so rotation is a no-op, attribution is automatic, and closure is a TTL rather than a search.

Naming no test cases at all

State what you would test before being asked: empty input, a single element, all elements equal, the maximum permitted size, and the input that exercises the branch you just wrote. It costs thirty seconds and is much of what separates someone who has shipped code from someone who has only solved puzzles.

Not asking what the system looks like if it dies halfway through

For any multi-step write, say what state remains if the process stops between step two and step three, and what brings it back: a single transaction, a saga with compensating actions, an outbox, or a reconciliation job. Partial failure is routine at any real call volume, so 'that shouldn't happen' is an answer with nothing behind it.

Questions & practice

Resolve effective-dated engagement versions for two million access checks

Hard · binary search · effective dating · complexity budget

The engagement table holds 500,000 effective-dated rows: (engagement_id, version, effective_from, effective_to which is NULL for the open version). A database exclusion constraint guarantees that per engagement_id the tstzrange(effective_from, effective_to) values never overlap, and the ranges are half-open. You must answer 2,000,000 access checks of the form (engagement_id, at_time) in one process with no database, returning the version open at that instant or none. Also report, per engagement, any coverage gap between adjacent versions. Give the complexity of the scan-per-check approach and say why it is not an option.

Approach

  1. State the naive cost first: scanning the table per check is 2e6 x 5e5 = 1e12 range comparisons. Even at 1e9 simple comparisons per second that is roughly fifteen minutes of pure compare work, against a budget measured in seconds. It is correct, just unaffordable, and the fix is an index built once rather than a faster comparison.
  2. Build a hash map engagement_id -> array of versions sorted by effective_from, once. O(V log V) to sort, O(V) memory. The build is amortised over 2e6 lookups, which is what makes it worth doing at all.
  3. Answer each check by binary-searching for the last effective_from <= at_time, then confirming effective_to IS NULL OR at_time < effective_to. If the confirmation fails, the instant lies in a gap and the answer is none, not the neighbouring version. O(log v) per check, O(V log V + Q log v) total.
  4. Name the precondition: predecessor search is only valid because the ranges per engagement do not overlap, which the exclusion constraint enforces with btree_gist for the equality operator. Without that guarantee you need an interval tree returning all stabbed intervals, at O(log n + k). Use the same half-open convention as the database, where at_time equal to effective_to belongs to the next version.
  5. Find gaps with one O(V) pass over each sorted array comparing prev.effective_to against next.effective_from. The exclusion constraint forbids overlaps, not holes, so a hole is entirely possible and is where an access check fails closed for a reason nobody can see in the contract.
  6. If the checks can be buffered, sort them by (engagement_id, at_time) and merge against the version arrays: O(Q log Q + V log V), usually a large constant-factor win over 2e6 random binary searches because the merge walks memory in order.

Worked solution (35 min)

  1. Index engagement E1 with v1 [2026-01-01, 2026-04-01), v2 [2026-04-01, 2026-07-01), v3 [2026-07-01, NULL), and engagement E2 with v1 [2026-01-01, 2026-03-01) and v2 [2026-04-01, NULL).
  2. Query (E1, 2025-12-31): predecessor search finds no effective_from <= at_time, so the answer is none.
  3. Query (E1, 2026-04-01T00:00:00Z): the predecessor is v2, not v1, because the boundary instant belongs to the later range under half-open semantics. Query (E1, 2026-06-30T23:59:59Z) returns v2 and (E1, 2026-09-01) returns v3.
  4. Query (E2, 2026-03-15): the predecessor is v1, but at_time is not less than v1.effective_to (2026-03-01), so the confirmation fails and the answer is none.
  5. Adjacent-pair pass on E2: v1.effective_to = 2026-03-01 is earlier than v2.effective_from = 2026-04-01, so report the gap [2026-03-01, 2026-04-01).

Expected result: E1: none, v2, v2, v3 for the four queries in order. E2: none at 2026-03-15, with a reported coverage gap of [2026-03-01, 2026-04-01). The scan-per-check approach is O(Q*V) = 1e12 comparisons; the indexed approach is O(V log V + Q log v).

Verify it yourself

  • An at_time exactly equal to a version's effective_to resolves to the later version, matching PostgreSQL's default [) range bound.
  • Every engagement has at most one version with effective_to IS NULL, which the unique partial index guarantees and the index build can assert.
  • The number of answers equals the number of checks; a 'none' is an answer, not a dropped row.
  • Brute-force scanning this five-row fixture per query reproduces every answer, including both 'none' cases.

Follow-up

  • The open version has effective_to NULL. How do you keep both the sort and the comparison total without a sentinel that overflows your timestamp type?
  • A gap exists for one engagement and a check lands in it. What should the access path return, and what should it emit so someone fixes the amendment?
  • Amendments are being written while you answer checks. What read model keeps an answer self-consistent, and what does a check mean if the snapshot is a second stale?

Collapse connector retry attempts into logical jobs and failure counts

Easy · hash aggregation · idempotency · retries

You are given one UTC day of connector_run rows: (connector_id, idempotency_key, attempt_no, status, failure_class, records_read, records_applied), unique on (connector_id, idempotency_key, attempt_no), up to 5,000,000 rows in arbitrary order. Retries of one logical job share the idempotency_key; the job's outcome is the status of its highest attempt_no. Return, per connector_id: the number of logical jobs, a count of terminally failed jobs by failure_class, and the dedupe hit total, meaning sum(records_read - records_applied) over terminal-succeeded attempts only. One pass over the input; state your memory bound.

Approach

  1. Key a hash map on (connector_id, idempotency_key) and keep only the highest attempt_no seen so far plus that row's status, failure_class and record counts. One pass, O(n) time, O(d) space where d is the number of distinct logical jobs, not O(n).
  2. Treat a repeated (key, attempt_no) as corrupt input and raise, since the table's uniqueness constraint says it cannot happen; silently overwriting hides a double-insert in the runtime.
  3. Fold the d surviving entries into per-connector counters in a second pass over the map, not over the rows. The failure_class histogram is a small fixed-width map per connector because failure_class is an enum.
  4. Compute dedupe hits only from terminal-succeeded attempts. A failed attempt's records_read is work attempted, not work deduplicated, and adding it counts the same source records once per retry.
  5. If d does not fit in memory, partition the input by hash(connector_id, idempotency_key) into p files and aggregate each partition independently. Same O(n) total work, p sequential passes, memory traded for I/O; the partition function must use the full key or a job's attempts split across files.

Follow-up

  • A job's highest attempt is 'cancelled' but an earlier attempt succeeded. What is the job's outcome, and what does that say about who writes the cancel?
  • Produce this incrementally as runs land instead of as a daily batch: what state do you keep per key, and what happens when an attempt arrives out of order?
  • records_applied is written by your runtime, not by the client. What would you reconcile it against before anyone trusts the dedupe-hit number?

Top-k longest connector runs from a stream too large to sort

Medium · heap · top-k · streaming

A day of connector_run rows sits in object storage: 200,000,000 rows, unsorted, far larger than memory, and streamable once. Each row has run_id, connector_id, started_at and finished_at, where finished_at is NULL for runs that never reached a terminal state. Return the 200 longest terminated runs by (finished_at - started_at), ties broken in favour of the lower run_id, plus the count of rows with a NULL finished_at. You may not sort the input or hold it in memory. State the time and space complexity you achieve.

Approach

  1. Keep a bounded min-heap of capacity k=200 ordered on (duration, -run_id). The root is then the weakest retained entry: smallest duration, and on a duration tie the largest run_id, which is exactly the one the stated tie-break should evict.
  2. Per row: if finished_at is NULL, increment a counter and skip; otherwise compute the duration and push when the heap holds fewer than k, else compare against the root and replace only if strictly better. O(n log k) worst case, O(k) space, and after the heap warms nearly every row costs a single comparison against the root.
  3. Reject the alternatives explicitly. A full sort is O(n log n) and at 2e8 rows means an external merge sort over hundreds of gigabytes to produce 200 rows. Quickselect is O(n) average but needs the whole array resident, which the constraint forbids.
  4. Handle the NULL as data, not as an edge case: depending on the language, subtracting from NULL either throws or yields a sentinel that lands at the top of the list. Count those rows and report them, because a spike of unterminated runs is its own signal.
  5. To parallelise, give each of p shard readers its own size-k heap and merge the p results. The union of per-shard top-k sets contains the global top-k, because an element beaten by fewer than k rows globally is beaten by fewer than k rows in its own shard, so the merge is exact at O(pk log k).

Follow-up

  • Now return the top 20 bindings by total run time instead of the top runs. What changes, and how many distinct keys must you hold to make it exact?
  • The key space is too large to hold an exact aggregate. What does a Misra-Gries summary with m counters actually guarantee about the counts it returns?
  • Rows now arrive continuously and you want a rolling one-hour top-k. Does the bounded heap still work, and if not, what breaks?

Add a non-null column to a live work-record table

Hard · online migration · lock contention · expand-contract

work_record holds roughly nine million rows and absorbs a month-end write burst. You must add currency CHAR(3) NOT NULL, backfill it from the engagement, add a foreign key to rate_card, and eventually drop the superseded hours_decimal column. Several releases are live, including client-managed installs running two-quarter-old code that still writes hours_decimal. Give the ordered plan, the lock each step takes and for roughly how long, how the backfill is chunked and restarted, and the signal that makes the drop safe.

Approach

  1. Sequence it expand, dual-write, backfill, switch reads, contract, and be explicit that only the last step is dangerous. Adding is cheap because an old release ignores a column it does not know about; removing breaks a writer you cannot upgrade and whose failure you may not see for weeks.
  2. ALTER TABLE work_record ADD COLUMN currency CHAR(3) — nullable, no default. That is a catalog-only change on every supported version and takes milliseconds. Refuse NOT NULL DEFAULT 'XXX': PostgreSQL 11 and later store a non-volatile default in the catalog and materialise it on read, so it is just as fast, but it hands all nine million rows a value that is already non-null and already wrong. That destroys the rest of the plan — the NOT NULL you add later is satisfied from the first second, so it can never find a row the backfill missed, and every consumer now has to distinguish a real currency from a sentinel forever. A NULL is the only marker that survives a crashed backfill.
  3. The statement is instant; the lock queue is not. An ACCESS EXCLUSIVE request waits behind every open transaction holding a conflicting lock on the table, and every request arriving after it queues behind that, so one long analytics SELECT turns instant DDL into a total stall. Run it with SET lock_timeout = '2s' inside a retry loop that catches 55P03 and backs off with jitter.
  4. Decide up front what an unconverged writer inserts. Two-quarter-old code omits currency, so it writes NULL, which means SET NOT NULL would start rejecting its inserts outright. Either fill the gap in the database with a BEFORE INSERT trigger that resolves currency from the engagement when NEW.currency IS NULL — one indexed lookup per insert, which you should time against the month-end burst before committing to it — or leave the column nullable and gate SET NOT NULL on the same release-convergence signal that gates the drop. What you must not do is paper over it with a default, which turns a visible NULL from an unconverged writer into an invisible 'XXX' that reconciliation will never flag.
  5. Backfill on a keyset cursor — WHERE work_record_id > $last AND currency IS NULL ORDER BY work_record_id LIMIT 5000 — one transaction per batch, advancing $last to the last work_record_id the batch wrote so filled rows are never revisited. Set statement_timeout, pause between batches so autovacuum and any replicas keep up, and persist the cursor so a crash resumes from the last committed batch. OFFSET n costs O(n) per page because the server scans and discards those rows, which makes the whole backfill quadratic in the table size.
  6. Take NOT NULL without a blocking scan, and only once the backfill reports zero rows remaining: ADD CONSTRAINT work_record_currency_nn CHECK (currency IS NOT NULL) NOT VALID takes a brief ACCESS EXCLUSIVE and scans nothing; VALIDATE CONSTRAINT runs under SHARE UPDATE EXCLUSIVE so reads and writes continue throughout, and raises check_violation, SQLSTATE 23514, naming a row if the backfill missed one — that failure is the entire reason the step exists, and it only exists because the column was added without a default; then SET NOT NULL, which on PostgreSQL 12 and later proves itself from the validated CHECK and skips its own full scan.
  7. Use the same shape for the foreign key — ADD CONSTRAINT ... REFERENCES rate_card(...) NOT VALID, then VALIDATE CONSTRAINT — and build any supporting index with CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block, scans the table twice, and on failure leaves an INVALID index that must be dropped concurrently before you retry. Note what the FK does not prove: a single-column reference is MATCH SIMPLE, so rows with NULL currency pass validation untouched, and validating it before the backfill finishes certifies only the rows that were already correct.
  8. Gate the contract step on observed convergence rather than elapsed time: every non-destroyed environment must report a reported_release_id at or past the release that stopped writing hours_decimal, and reported_release_id IS NULL counts as unknown and therefore blocks. DROP COLUMN itself is a catalog-only update, instant under ACCESS EXCLUSIVE, reclaiming no space until a rewrite — the cost is entirely in the writer you did not know was still running.

Follow-up

  • A client-managed install reports a release you no longer support and has not converged for two quarters. Do you drop, and what do you owe that client first?
  • During the dual-write window, which column does the month-end reconciliation read, and what does a mismatch between the two mean?
  • The backfill dies at batch 900 of 1,800. What exactly does the restart re-read, and how do you know it did not skip a range?

Find watermark coverage gaps with a window function

Medium · window functions · gaps and islands · event tables

Each succeeded connector_run claims a half-open watermark range [watermark_from, watermark_to). Ranges for one connector can overlap, and a re-run can be nested entirely inside an earlier range. Write the query that returns, per connector_id, every interval not covered by any succeeded run, as (gap_from, gap_to), ordered by connector then time. State the frame clause you use and why the default one is wrong, and give the index that removes the sort.

Approach

  1. Rule out lag(watermark_to) first and say why: ordered by start time, the immediately preceding row may end earlier than a row before it, so a nested or shorter range makes you report a gap that a wider earlier range already covers. The correct comparison is against the running maximum of every previous watermark_to, not the last one.
  2. Compute max(watermark_to) OVER (PARTITION BY connector_id ORDER BY watermark_from, watermark_to ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING). The ROWS frame is load-bearing: the default frame is RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, which includes the current row, so the running max always covers the current row's own end and the query reports no gaps ever.
  3. Add watermark_to as a tiebreaker in the window ORDER BY so rows sharing a start instant are ordered deterministically under a ROWS frame, then emit a gap where the running max is non-null and watermark_from is strictly greater than it. The first row per connector has an empty frame and a NULL max; that is not a gap unless you define a floor such as the binding's creation time, which you should decide explicitly.
  4. Filter status = 'succeeded' in the window's input, and argue the classification: an ambiguous run may or may not have applied its range, so counting it as covered can drop data silently. Gap detection should treat ambiguous as uncovered and leave resolution to the reconciliation path.
  5. Offer the shorter alternative with its precondition: on PostgreSQL 14 and later, range_agg(tstzrange(watermark_from, watermark_to, '[)')) per connector builds a multirange, and the gaps are its complement against the window you care about. It handles adjacency and nesting without any frame subtlety, at the cost of requiring that version.
  6. Index (connector_id, watermark_from, watermark_to) WHERE status = 'succeeded'. It supplies the window's required ordering directly, so the plan is a WindowAgg over an index scan with no Sort, and the work is O(n) over that connector's succeeded runs.

Worked solution (30 min)

  1. Load a fixture for one connector: [10:00, 11:00), [10:15, 10:30) nested inside it, and [11:30, 12:00).
  2. Write the lag version first and record what it returns, so the failure mode is concrete rather than theoretical.
  3. Replace lag with the running max under an explicit ROWS frame, then filter to rows where watermark_from exceeds it.
  4. Add a second connector and a pair of runs sharing an identical watermark_from to exercise the tiebreaker.
  5. Create the partial index and re-check the plan.

Expected result: SELECT connector_id, prev_end AS gap_from, watermark_from AS gap_to FROM (SELECT connector_id, watermark_from, watermark_to, max(watermark_to) OVER (PARTITION BY connector_id ORDER BY watermark_from, watermark_to ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING) AS prev_end FROM connector_run WHERE status = 'succeeded') s WHERE prev_end IS NOT NULL AND watermark_from > prev_end ORDER BY connector_id, gap_from; — on the fixture it returns exactly one row, 11:00 to 11:30.

Verify it yourself

  • The lag version reports 10:30 to 11:30, an hour of which is demonstrably covered by the first range; the running-max version reports only 11:00 to 11:30.
  • Remove the explicit ROWS frame and confirm the query returns no gaps at all, which is the default frame including the current row.
  • Two runs with identical watermark_from produce a stable result across repeated executions once watermark_to is in the window ORDER BY.
  • EXPLAIN shows WindowAgg over an Index Scan with no Sort once the partial index exists.

Follow-up

  • How do you distinguish a real gap from a period when the connector was deliberately paused?
  • The connector has run for two years and you only care about the last 30 days. What does adding that predicate do to the correctness of the running maximum?
  • How would you turn this query into an alert without it firing on the trailing edge, where the newest range is simply not written yet?

Does your CV accurately reflect your technical experience and project history?

Medium

Approach

  1. Say what you would check first and why it is the highest-information step.
  2. State your assumptions explicitly before working the problem.
  3. Clarify what is being asked and what a complete answer contains.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

What are you doing in your current role?

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Work from the requirement backwards to the design.
  3. Say what you would check first and why it is the highest-information step.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

What are the projects you are currently working on?

Medium

Approach

  1. Say what you would check first and why it is the highest-information step.
  2. State your assumptions explicitly before working the problem.
  3. Clarify what is being asked and what a complete answer contains.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Can you provide a brief introduction about yourself?

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Clarify what is being asked and what a complete answer contains.
  3. Work from the requirement backwards to the design.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

Design a run API where the outcome is genuinely unknown

Hard · idempotency · state-machines · ambiguity · at-least-once

connector_run holds run_id, connector_id, environment_id, idempotency_key, attempt_no, status, failure_class, watermark_from, watermark_to, records_read, records_applied, source_schema_version, started_at and finished_at; idempotency_key is derived from the connector and the watermark range, and (connector_id, idempotency_key, attempt_no) is unique. Workers claim a range, write to a client-owned target that offers no idempotency support of its own, then report back. A worker can die at any point, and a write can time out leaving the target's state unknown. Design the claim, report and settle endpoints and the status transitions a scheduler and an operator drive.

Approach

  1. Derive the run's identity from the work rather than from the caller. POST /connectors/{id}/runs carrying a watermark range computes idempotency_key deterministically, so a scheduler retrying a claim receives the same logical run instead of a second one covering the same rows. Mint a new attempt_no only on an explicit retry of an attempt that has already settled as failed, which keeps the attempt history readable and stops a crashed worker from quietly opening a parallel attempt over the same range.
  2. Give the claim a lease with an expiry rather than a lock. The worker heartbeats; an expired lease makes the run reclaimable. Without an expiry, a dead worker wedges the binding until someone notices. With a lease and no fencing, two workers can both believe they hold the range, so carry a monotonically increasing lease token and reject any settle whose token is stale — that is what stops a zombie's late report overwriting the live attempt's.
  3. Make ambiguous a first-class status with its own transitions, separate from failed. A timeout, or a reset after the request bytes went out, leaves the target's state unknown: settling as failed and retrying duplicates, settling as succeeded and advancing the watermark loses data silently. The settle endpoint therefore accepts succeeded, failed with a class, or ambiguous with evidence, and only succeeded advances the watermark.
  4. Keep the ledger write and the watermark advance in one local transaction, and have settle carry the dedupe keys actually applied. The target write and your commit cannot be made atomic without a distributed transaction no client will grant, so ambiguity is resolved by a reconciliation read: re-read the target by dedupe key over the claimed range and settle from what you find. Store the key's input fields and a derivation version alongside the hash, because the client will eventually rename or retype one of those fields and every previously computed key still has to be reproducible.
  5. Define each caller's behaviour precisely. The scheduler re-claims expired leases and retries only definite failures whose class is retryable; it never retries an ambiguous run. The operator has an explicit resolve endpoint used after reconciliation, recording who decided and on what evidence. An ambiguous run that auto-retries is the design bug — it converts an unknown into a duplicate inside a system you do not own and cannot correct.
  6. Close the loop with the counters. records_applied differs from records_read by the dedupe hit count, so a retry over a partially applied range should report a high hit count and little net change. That difference is the observable proving the dedupe key works: if a retry reports applied equal to read, the key is not matching and you are duplicating on the client's side right now.

Worked solution (40 min)

  1. Write the state machine — queued, running, succeeded, failed, ambiguous, cancelled, abandoned — with every legal transition and who triggers it.
  2. Write the claim endpoint's request and response, showing how idempotency_key is derived and what a repeated claim returns.
  3. Inject a crash between each adjacent pair of claim, fetch credential, validate, write, ledger, advance, and say what the next claim of that range does in each case.
  4. Write the settle payload for the ambiguous case, listing the evidence a later reconciliation will need.
  5. Trace a zombie worker settling ninety seconds after its lease expired, and show the token check rejecting it.
  6. Write the operator's resolve endpoint and the audit fields it records.

Expected result: A claim keyed on (connector_id, watermark range) that is safe to repeat, a leased and heartbeat-renewed claim with a fencing token checked at settle, a settle endpoint accepting succeeded, failed-with-class or ambiguous-with-evidence where only succeeded advances the watermark, a reconciliation path that resolves ambiguity by reading the target back by dedupe key, and an operator resolve step that never happens automatically.

Verify it yourself

  • Retrying a claim for the same range never produces two runs with different idempotency_keys.
  • A run left ambiguous stays ambiguous until something reads the target; nothing in the scheduler moves it.
  • Replaying a fully applied range reports records_applied of zero with a dedupe hit count equal to records_read.

Follow-up

  • The client's endpoint offers no read-back by your dedupe key. How do you settle an ambiguous run then?
  • A worker's lease expires while its write is still in flight and a second worker claims the range. What exactly prevents the double apply?
  • How long do you keep the dedupe ledger, and what happens when a record arrives after its ledger row was pruned?

One connector burns a client's entire rate budget

Easy · poison message · retry loop · rate limiting

A client reports that your integration is exhausting their API quota, and every connector binding for that client now fails with client_429. For one binding, connector_run holds 300 rows sharing one idempotency_key with attempt_no incrementing, all with status 'running', finished_at NULL and failure_class NULL, created exactly 120 seconds apart over the last ten hours. The run lease is 120 seconds. Workers in that pool show restart counts in the hundreds. Other clients are unaffected. Diagnose in order, say who is at fault, and state what stops one range doing this again.

Approach

  1. Split client fault from your fault before anyone contacts the client. Group the last hour's failures by client and by binding: every binding for one client failing while other clients are clean means a resource shared with that client is exhausted, and the only thing you share with them is their quota.
  2. Read the ledger's shape, which is the entire diagnosis. Three hundred attempts under one idempotency_key, none with a terminal status and none with a failure_class, spaced at exactly the lease interval. A handled failure writes failure_class and finished_at; writing nothing at all means the worker was killed before it could classify. The exact spacing names the killer as lease expiry, not the client endpoint.
  3. Quantify the damage from the same rows: records_applied is zero across all 300, while each attempt got far enough to pull pages. The binding consumes quota continuously and applies nothing, which is why the other bindings for that client see 429 and the faulty one does not.
  4. Reproduce offline. watermark_from and watermark_to are recorded, so replay that exact range against a captured payload with the lease disabled and find what exceeds 120 seconds: a page far larger than the others, a validator input that degrades badly, or an unbounded in-memory accumulation over the range.
  5. Fix the loop before the payload. Cap attempt_no, move the range to status 'abandoned' with an alert, and hold the watermark rather than advancing past it, because a silent gap is data loss and skipping must require a human decision. Then make the work fit the lease: stream and chunk the page, renew the lease only while progress is provable, and bound the page size at the boundary.
  6. Add the detector this evaded. Alert on attempts that never reach a terminal state, not only on failure counts, because a run that never finishes never increments any failure metric.

Follow-up

  • The client asks how much of their quota you consumed. Which rows answer that, and what would you have to add to answer it exactly?
  • The abandoned range contains records you still need. What is the safe way to get past it without breaking the at-most-once guarantee?
  • Lease renewal introduces a new failure: a worker that renews steadily but makes no progress. How do you catch that one?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

  • Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
  • Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
  • Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

  • Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
  • Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
  • Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

  • Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
  • Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
  • Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

  • For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
  • Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
  • Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
  • Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

  • Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
  • Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
  • Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
  • Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

  • Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
  • Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
  • Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

  • Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
  • Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
  • Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

The estimate and the day you knew

A slipped date is only a bad story if you sat on it. What matters is what you believed when you gave the number, the signal that told you it was wrong, how many days passed before you said so, and what you cut rather than asking for more time. Scope you defended counts as much as scope you dropped.

Estimate a first integration against an unfamiliar system

Easy · estimation · unknowns · client dependencies

You are asked for a delivery date on a first integration against a client-owned system nobody on the team has touched: unknown production schema version, unknown real rate limit, sandbox access only for now. Describe how you have estimated work of this shape before. State how you separated the unknowns you could reduce by investigation from the ones you could not, the range you gave, the assumptions you attached to it, and what you did when one of those assumptions turned out to be wrong.

Approach

  1. Split the unknowns into reducible and irreducible. The schema version, the payload shape and the real rate limit are answerable in a day of investigation; the client's change-window availability and their turnaround on an access request are not, and they usually dominate the calendar.
  2. Buy information before committing to a date: a time-boxed spike that asserts the observed schema version at the boundary and measures the throttle empirically rather than reading it from a document. Say what you would do if the spike itself were blocked on access.
  3. Give a range with the assumptions written beside it, each with an owner and a date it will be confirmed by, so that a broken assumption updates the estimate instead of silently consuming slack.
  4. Separate engineering days from elapsed days out loud. In this domain the gap between them is the estimate, because waiting on a client's approval is the schedule.
  5. Say how you communicated the first breach: early, naming the assumption, rather than absorbing it and reporting late.

Follow-up

  • The sandbox has no rate limiting and production does. What changes in the plan, not just in the number?
  • You are pushed for a single date rather than a range. What do you give, and what do you attach to it?

Own a cross-tenant read that reached a client

Hard · tenant isolation · incident response · blast radius

Prepare a five-minute account of an isolation or data-exposure incident you owned: a query that returned another tenant's rows, a cache keyed without a tenant, or a report that crossed a boundary. State the mechanism precisely, how you established blast radius (which tenants read which tenants' rows, over what window), the containment step, and the fix that made the class impossible rather than the instance. Finish with the notification decision and who made it. If you have never owned one, use the closest near-miss and say so.

Approach

  1. Open with the mechanism in one sentence rather than the symptom. The canonical version here: a session-scoped SET app.tenant_id on a connection returned to a transaction-mode pool with that value still attached, so the next checkout inherited it and row-level security then enforced the previous tenant's policy flawlessly.
  2. Separate containment from fix. Containment is what you did in the first twenty minutes (drain the pool, switch the pool to session mode, disable the endpoint); the fix is structural (SET LOCAL, which dies with the transaction, plus an assertion that the setting equals the request's tenant immediately before the first statement).
  3. Give the blast-radius method, not an adjective: reconstruct from the query log joined to request context on a correlation id, count distinct (reading tenant, row tenant) pairs and rows, and state what you could not reconstruct and why.
  4. Name the class-level fix and its cost. FORCE ROW LEVEL SECURITY so the table owner is not exempt, separate migration and application roles because a role with BYPASSRLS defeats every policy, and a test that drives two tenants' requests concurrently over one pooled connection, which is the load profile a serial integration suite never produces.
  5. Close with the notification call: it is a contractual question, not only an engineering one, so say who decided, how long the decision took, and what you would not repeat.

Follow-up

  • Your suite ran one request at a time and passed. What test would have caught this, and what does it cost to run on every change?
  • The same defect inside a dedicated deployment touches one client. Does that change your severity, your containment, or only your disclosure?
  • How do you know today that no other endpoint in the estate has the same defect?

Argue against a per-client fork of the product

Medium · maintenance cost · extension points · technical strategy

Describe a time you argued against a design that was cheaper for the request in front of you and worse for the estate: forking the product for one client, a bespoke schema, a one-off pipeline. State the alternative you proposed, the cost model you used to compare the two, what the decision actually was, and what happened afterwards. The account must include a number you computed at the time, not one you inferred later. If you lost the argument, say what the outcome taught you about the argument.

Approach

  1. State both options as costs over time rather than as good and bad. A fork is O(1) today and O(clients x changes) forever: one security patch becomes one port per fork, each with its own conflicts, its own test run and its own release window, and the forks drift further apart with every port.
  2. Name the number you put in front of the decision-maker (ports per year times hours per port, or release windows consumed) and say where each input came from.
  3. Give the alternative concretely: a named extension point with a stable interface and a stated compatibility commitment, or an explicitly time-boxed, separately priced artefact that is never expected to receive upstream fixes. 'Do it properly' is not an alternative.
  4. Say what you conceded. An answer with no concession reads as someone who has not shipped against a delivery date; the defensible concession is usually scope or a sunset date, not the interface.
  5. Report the outcome honestly, including the case where you were wrong, and name the signal that would have told you sooner.

Follow-up

  • The client is the largest on the book and the fork is a renewal condition. What changes in your recommendation, and what stays fixed?
  • You won, and the extension point shipped. A year later, how do you tell whether it is used, worked around, or quietly forked anyway?