Interview guides · Zurich Insurance
Zurich Insurance Software Engineer Interview Questions & Guide 2026
Software engineer interview prep for Zurich Insurance: original coding, system design and debugging practice, each with a worked solution.
The 60-second brief
As a Software Engineer at Zurich Insurance, you play a vital role in building and scaling digital platforms that power global insurance operations. You will develop modern applications, integrate microservices, and ensure high reliability for systems that protect millions of customers worldwide. Your day-to-day contributions directly impact how policyholders, brokers, and internal stakeholders interact with complex insurance products and digital services.
Ask how many rounds there are and what each one is before you plan your weeks, because the composition is what your hours should follow. A loop with three coding rounds and one short design conversation deserves a different split from the reverse, and whoever schedules it will usually say plainly which it is.
Focus areas
- Store money as integer minor units
- Reconcile the ledger against processor settlement files
- Name the isolation level each invariant requires
Overview
This role sits at the intersection of enterprise scale and modern software engineering. You will collaborate with cross-functional teams to modernize legacy systems, design resilient cloud architectures, and deliver secure, high-performance software. Whether you are building customer-facing applications or internal tooling, your work drives operational efficiency and enables digital transformation across one of the world's leading insurance providers.
Expect an environment that values technical excellence, collaboration, and continuous learning. You will work alongside experienced engineers, product managers, and domain experts who expect clean code, thoughtful architecture, and a strong user-first mindset. Success in this role requires balancing robust technical implementation with a clear understanding of business value in a highly regulated industry.
The Zurich Insurance interview process
Zurich Insurance candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.
Recruiter Screening
Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.
What to demonstrate
- Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
- Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
- Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled
How to prepare
- Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
- Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
- Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market
Hiring Manager Discussion
Part of what is being decided is what your manager's week looks like once you are on the team: how they find out that something you shipped is broken, and whether they hear it from you or from a customer. The material that moves this round is therefore not the launch, it is the week after. Describe how you knew the change was working, which number you watched and for how long, and what you would have reverted to. A project that ends at the word shipped leaves the manager guessing about the part they care about most.
What to demonstrate
- How a change of yours was verified in production, and whether that check existed before the deploy or was assembled afterwards once something looked wrong
- Whether the rollback story is specific, including the case where a revert is not enough: once a migration has dropped a column or rewritten data, going back is its own change with its own risk
- Whether you can describe an incident you caused, how it was found, and how much time passed between it starting and anyone noticing
- Whether bad news in your account travels early and from you, or consistently arrives via somebody else
How to prepare
- For your last significant change, write down what you watched after the deploy, for how long, and the number that would have made you revert. If nothing was watched, say that plainly instead of inventing a dashboard.
- Work out the rollback answer for a change that touched stored data, and be able to say what made it reversible or what you would have had to do instead. That distinction is a level signal on its own.
- Write the two-minute version of an incident you owned: what broke, how it surfaced, what you did in the first ten minutes, and the change that stopped it recurring. Get the timeline straight, because this is the story most likely to be interrupted with questions.
Technical Assessments
What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.
What to demonstrate
- Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
- Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
- Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly
How to prepare
- Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
- Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
- Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live
Where candidates lose points
Retrying a charge after a timeout
A timeout is not a failure; it is an unknown outcome, and the request may have been processed in full with only the response lost. Re-sending it without an idempotency key that the processor itself honours produces a duplicate charge, which is a customer-visible incident and usually a dispute. The correct handling is to treat the state as unknown, query the processor for that key or client reference, and only then decide. The mechanism also depends on the key being generated once by the caller and reused across every attempt — generating a fresh key per retry turns the whole scheme into a no-op while leaving all the code that appears to implement it in place.
Writing the state change to the database and publishing the event in the same code path
No transaction spans a relational database and a message broker, so a crash between the two leaves one done and the other not, and the failure is asymmetric in both orderings: publish-then-commit invents events for state that never existed, while commit-then-publish silently loses events for state that does. Retrying the publish after the commit is not a fix, because the process can die before the retry runs. The working shape is an outbox row written inside the same transaction plus a relay that publishes it at least once, which makes consumer-side idempotency mandatory rather than optional. Note also that 'exactly-once' in a stream processor means exactly-once processing within that system's own read-process-write transaction, and says nothing at all about an external side effect such as charging a card.
Comparing floating-point values for equality, or holding money in them
Binary floating point cannot represent 0.1 exactly, so repeated addition drifts and an equality check fails on values that are mathematically equal. Store currency as integer minor units or a decimal type, and compare floats against a tolerance you chose for a stated reason.
Sharing mutable state with no stated owner
Say which thread, request or task owns each mutable structure, and what protects it when the answer is more than one: a lock, a queue that hands ownership across, or an immutable copy per reader. A structure documented as safe for concurrent reads is usually not safe for a concurrent write alongside those reads.
Questions & practice
Detect duplicate-charge bursts in an out-of-order authorisation stream
Hard · sliding window · event time · watermarks · deduplication
Authorisations arrive as (instrument_token_id, amount_minor, currency, event_time, arrival_time) at roughly 3,000 per second, up to 60 seconds late and out of order. Flag any token with three or more authorisations of identical (amount_minor, currency) inside any 10-minute window of event time. Report each flag once, as early as correctness allows. State memory per key and in total, how late an event you will accept, and what you do with one that arrives after you have already reported — or already declined to report — that window.
Approach
- Key state by
(instrument_token_id, amount_minor, currency), not by token: the predicate is about identical amounts, so the window belongs to the triple. Each key holds its own event-time-ordered deque. - In-order, this is two pointers: on insert, pop from the front while
front <= new - 10 min, then flag if the deque reaches length 3. Amortised O(1) per event, memory proportional to that key's window occupancy. - Out-of-order arrival breaks append-only monotonicity, so insert in position instead. With lateness bounded at 60 seconds the insertion point is always near the tail, so a short sorted vector or a 600-bucket per-second ring keeps it O(w) with tiny w; a balanced tree per key is correct but over-built for a one-minute reorder.
- Drive decisions off a watermark of
max(event_time seen) - 60 s, never off wall clock. Anything older than the watermark is too late to change an answer and is counted in alate_droppedmetric. Without an explicit watermark you have still chosen a lateness policy — you just cannot state it or test it. - Report-once needs its own state: per key, a set of already-flagged 10-minute window ids. A later event inside an already-flagged window must not re-flag, and a late event that completes a window you never flagged must flag — which is why state is retired at window close plus the lateness bound, not at window close.
- Size it: state lives 660 seconds, so at 3,000 events per second there are about 1.98 million in flight, plus one flagged-window set per active key. Bound total memory explicitly and shed by key age, and say what shedding costs — a shed key can miss a flag, making the threshold a product decision rather than a tuning knob.
Follow-up
- Make the threshold and window configurable without rebuilding all in-flight state on every change. What does that constrain in the data structure?
- Two stream partitions hold events for the same token. What does that force the partitioning key to be?
- An event arrives three hours late. Does any answer change, and who is told?
Compute peak held exposure from overlapping authorisation holds
Medium · sweep line · intervals · authorisation holds
An account has up to 2 million authorisations on one business date: (auth_id, amount_minor, created_at, expires_at) with the hold live over the half-open interval [created_at, expires_at), plus capture events (auth_id, captured_minor, captured_at) that reduce the hold at captured_at, and explicit reversals that drop the remainder to zero. Timestamps are microsecond-precision timestamptz. Return the maximum total held amount across the day and the earliest instant it is reached, with complexity. Then say what changes if the deliverable is the peak per minute instead.
Approach
- Expand each authorisation into signed delta events rather than reasoning about intervals:
+amountatcreated_at,-remainingatexpires_at,-captured_minorat eachcaptured_at,-remainingatreversed_at. The problem collapses to a running sum over a sorted event list. - Sort the 2n to 4n events by
(timestamp, sign)with negative deltas ordered first on a tie. The half-open convention forces that: at exactlyexpires_atthe hold is already gone, so a-must land before a+at the same instant or you report a one-microsecond peak that never existed. O(n log n) time, O(n) space. - Sweep once, tracking
running,bestandbest_at, taking the first instant that attains the maximum. Say out loud which tie rule you are using — 'the peak' is ambiguous when the same level is reached twice, and the caller needs to know which instant they are being handed. - If timestamps are bucketed (1,440 minute buckets for the per-minute variant), drop the sort for a difference array: add the delta at the start bucket, subtract at the end bucket, prefix-sum once. O(n + B) time and O(B) space, strictly better, at the cost of answering only at bucket resolution.
- Assert the invariant during the sweep:
runningmust never go negative. A negative total means a capture exceeded its authorisation, which is an invariant violation upstream rather than a sweep bug — fail loudly instead of clamping at zero and reporting a plausible number. - Handle carry-in: a hold created before the window contributes its remaining amount as the sweep's initial value, not as a
+event inside the window. Omitting that is the off-by-a-day that makes the first minute of every day look artificially quiet.
Follow-up
- An incremental authorisation raises an existing hold after the fact. Where does that event go, and does it disturb the tie rule?
- You now need the peak for 10 million accounts inside a nightly window. What changes, and what must the partitioning key be?
- The peak sizes a funding transfer. Does the business date or the timestamp decide which day that transfer lands on?
Answer as-of balance queries over an append-only entry log
Hard · prefix sums · offline queries · append-only
Given 400 million ledger_entry rows (entry_id, account_id, direction, amount_minor, currency, business_date) and 2 million queries of (account_id, currency, as_of_date) asking for the balance at the end of that business date, produce every answer. The obvious solution — per query, sum that account's entries with business_date <= as_of_date — is correct. Say precisely why it will not finish, then give one that will, with time and space complexity. Corrections are posted as new entries carrying their own business_date.
Approach
- Cost the naive version in numbers before rejecting it. Spread uniformly over 20 million accounts, each query touches about 20 rows behind a per-account index and 2 million queries is 4e7 row touches — perfectly fine. The problem is skew: one pooled clearing or merchant settlement account holding 3e7 entries, taking 10% of the queries, is 6e12 row touches. Name the skew; 'n is large' is not the reason.
- The structural fact that buys a cheap answer: entries are append-only and never updated, so a prefix sum over an account's entries ordered by
(business_date, entry_id)is stable — nothing behind position i can change. No mutable-balance design offers that, and it is why the storage is worth paying for. - Offline sweep, when all queries are known up front: externally sort entries by
(account_id, currency, business_date, entry_id)and queries by(account_id, currency, as_of_date), then merge-walk both with a running sum, emitting each query's answer as the sweep passes its date. O((n + q) log(n + q)) dominated by the sort, O(1) beyond sort buffers, one sequential pass over each input instead of 2 million random seeks. - Online alternative: materialise end-of-day snapshots — one row per
(account_id, currency, business_date)that had activity, holding the cumulative total. A query becomes one index seek for the latest snapshot at or beforeas_of_date, O(log n) per query, over far fewer rows than n. Use snapshots when queries arrive singly and the sweep when they arrive as a batch. - Corrections are the subtlety: an entry posted today but dated back changes historical answers, so every snapshot for that account from that date forward is stale. Either keep a Fenwick tree over dates per account (O(log D) update and prefix query) or recompute that account's snapshots from the corrected date onward. Then be precise about what reproducibility means — yesterday's statement is reproducible as of a stated snapshot time, not identical forever.
- Bound the resources: int64 sums throughout, no float; 400 million rows at roughly 48 bytes of the columns you actually need is about 19 GB, so the sort is external and its fan-out is chosen from the sort buffer, not from the row count.
Worked solution (40 min)
- Compute both costs explicitly: the uniform case at about 4e7 row touches, and the skewed case at about 6e12. Showing that arithmetic is the answer to 'why'.
- Implement the offline sweep on a 10-million-row, 50,000-query fixture, merging on
(account_id, currency, business_date, entry_id). - Implement the naive version as the reference answer and assert both agree on every fixture query.
- Add a correction entry dated 30 days back, re-run, and assert that exactly the queries with
as_of_dateon or after that date move, all by the same signed amount. - Measure rows touched and wall time for each at 10 million rows, then extrapolate to 400 million and state the assumption that makes the extrapolation valid — sequential I/O, no random seeks.
Expected result: Both implementations agree on all 50,000 fixture queries. After the back-dated correction, exactly the queries at or after its business_date change, each by the identical signed amount. The sweep touches every entry row once; the naive version touches the hot account's rows once per query against it.
Verify it yourself
- The balance at the maximum
as_of_dateequals the plain full-file aggregate for that account and currency. - Summed across all accounts at any single date, the result is 0 per currency — provided every entry in a transaction shares one
business_date, which is worth asserting separately rather than assuming. - The sweep's answers do not change when entries are fed in a different physical order, because the sort key is total.
- No answer depends on
posted_at: swapping two entries'posted_atwithin one business date changes nothing.
Follow-up
- One account holds 30% of all entries. What does the external sort do with it, and what would you do for that one key instead?
- Queries now arrive online at 500 per second. Which design survives, and what does keeping the other one warm cost?
- A correction lands with a
business_date90 days back. Which snapshots are now wrong, and how does a reader find out?
Decide whether balance is derived or materialised, then hold the floor
Hard · denormalisation · isolation levels · lock contention
Authorisation needs an account's available balance inside an 80 ms budget at 3,000 requests per second; that account already has 200 million ledger_entry rows. A product rule says the balance may never fall below the account's negative overdraft limit. Decide whether the balance is summed from entries or held in a materialised account_balance(account_id, balance_minor, floor_minor, currency, version) row, and justify the choice from the read pattern. Then give the write path that holds the floor, naming the isolation level, the anomaly a weaker level permits, and the SQLSTATE you retry.
Approach
- Size the derived read before arguing about it: summing 200 million rows is not an 80 ms operation under any index, since even a covering index on (account_id, entry_id) still reads work proportional to the rows. The authorisation read pattern forces one materialised row fetched by primary key; the statement read pattern, which is low-rate and historical, stays derived from entries. That is the whole justification for the denormalisation, and its price is a write on that row per posting.
- Put the floor where it can be a single-row constraint: CHECK (balance_minor >= floor_minor) on account_balance, updated in the same transaction as the entries. As a predicate over a set of entry rows it cannot be a CHECK at all, which is the reason the materialised row earns its keep twice.
- Write the mutation as one statement: UPDATE account_balance SET balance_minor = balance_minor - $1, version = version + 1 WHERE account_id = $2 AND balance_minor - $1 >= floor_minor. Zero rows updated means refused. This is safe even under READ COMMITTED, because the UPDATE re-evaluates its predicate against the locked, post-update version of the row.
- Name the anomaly in the shape that is not safe: SELECT the balance, compute a new value in application code, then UPDATE to that constant. Every statement under READ COMMITTED takes a fresh snapshot, so two concurrent 60-unit withdrawals against 100 both read 100 and both write 40. The row then claims 40 while 120 has actually left, so the true position is -20, below a floor of 0, and the row it is checked against cannot show it. PostgreSQL REPEATABLE READ is snapshot isolation and aborts the loser with SQLSTATE 40001; SERIALIZABLE additionally closes write skew across two rows. Both require a bounded retry with backoff, and InnoDB REPEATABLE READ does not abort at all, so identical code changes behaviour on a different engine.
- For a two-account transfer, acquire the rows in a deterministic order such as ascending account_id; without it, opposing concurrent transfers deadlock and the database kills one with SQLSTATE 40P01. That is a retry, not a correctness failure, but it is a retry somebody has to write.
- Finish on the ceiling: the hot row admits one committed write per lock hold, so at a 2 ms hold it caps near 500 per second regardless of cores. Sharding into N sub-rows multiplies throughput and immediately makes the floor check cross-row again, which then needs the shard sum under SERIALIZABLE or a per-shard reserved allowance.
Follow-up
- Shard the balance into eight sub-rows. Write exactly what the floor check now does, and what it costs per authorisation.
- Someone proposes an AFTER INSERT trigger on ledger_entry to maintain the balance. What does that change about ordering, about batch posting, and about failure handling?
- How do you detect that the materialised row has drifted from the entries, how often do you run it, and on which replica?
Make a charge endpoint safe under concurrent duplicate retries
Easy · idempotency · unique constraints · read committed
idempotency_key holds id, scope, key, request_fingerprint (SHA-256 over the canonicalised body), status (in_progress, completed, failed), response_status, response_body, locked_at, completed_at, expires_at, created_at. Fifty identical create-payment requests carrying the same scope and key reach four application instances inside the same 20 ms. Give the DDL constraint and the exact statements the handler runs so that exactly one payment_intent is created and all fifty callers receive the same response body. State what you return when that key arrives with a different fingerprint, and what an arrival after expires_at means.
Approach
- Put the concurrency control in the schema: UNIQUE (scope, key). A SELECT-then-INSERT cannot work because both transactions can read nothing before either commits, so the check passes twice and the constraint then surfaces as an error on a payment that succeeded.
- Claim the key with INSERT ... ON CONFLICT (scope, key) DO NOTHING RETURNING id. A conflict returns zero rows rather than the existing row, so branch on rowcount: the winner proceeds, the loser reads the stored row.
- Keep that path on READ COMMITTED deliberately. The loser's follow-up SELECT takes a fresh statement snapshot and therefore sees the winner's committed row; under REPEATABLE READ the transaction snapshot predates that commit, the row stays invisible and the loser concludes the key does not exist.
- Split the work across two transactions because the processor call cannot sit inside one: commit the in_progress row with locked_at first so losers can see a claim, perform the effect, then write payment_intent plus status=completed with response_status and response_body in a single second transaction.
- Handle the crash window explicitly: a row stuck in_progress past its lease is an unknown outcome, not a failure, so the reaper queries the processor for that key before deciding. A loser that sees in_progress returns 409 and retries rather than repeating the effect.
- Compare request_fingerprint before replaying anything. Same key with a different body is 409, never the cached response, because replaying confirms a payment the caller did not request; and set expires_at beyond the client's and the processor's maximum retry horizon, since a replay after it is a genuinely new request.
Worked solution (20 min)
- Create idempotency_key with UNIQUE (scope, key) and write the claim statement as INSERT ... ON CONFLICT DO NOTHING RETURNING id, plus the fallback SELECT on zero rows.
- Drive it with 50 threads sending the identical body, and assert one payment_intent row and fifty byte-identical response bodies.
- Re-send the same key with amount_minor changed by one unit and assert 409 plus no new payment_intent row.
- Kill the process between the processor call and the second commit, restart, and show the reaper resolves the in_progress row by querying the processor rather than by re-charging.
Expected result: One payment_intent row, one idempotency_key row at status=completed, fifty responses with identical response_status and response_body, zero 5xx. A 51st request with the same key and a changed amount returns 409 and leaves payment_intent untouched.
Verify it yourself
- Repeat the 50-way race 20 times; COUNT(*) FROM payment_intent for that key is 1 on every run.
- Assert the fifty bodies are byte-identical, not merely all 2xx.
- Drop the UNIQUE constraint and confirm the test now fails; if it still passes, the race is not being exercised.
- Re-run the loser path under REPEATABLE READ and observe it failing to find the winner's row, which is why the level is pinned.
Follow-up
- The handler dies after the processor call and before the local commit. What does the next retry with that key observe, and how does the system converge on exactly one charge?
- Does the downstream processor honour an idempotency key of its own? Who mints it, and what breaks if a fresh one is generated per attempt?
- How do you purge rows past expires_at without the delete contending with the insert path?
Can you explain how a microservice architecture functions and what benefits it provides, and what is the monolithic alternative?
Medium · system design and architecture
Approach
- Fix the scope first: who calls this, how often, and what they do when it fails.
- State the consistency you need, and where you are willing to be stale.
- Name the failure you are designing for, then the recovery path.
Follow-up
- What would you drop to keep the system up under load?
- How does this behave when that dependency is down for an hour?
What are your past experiences in software development and deploying code repositories?
Medium · system design and architecture
Approach
- State the consistency you need, and where you are willing to be stale.
- Name the read and write paths separately; they rarely have the same bottleneck.
- Choose a partition key and say what query it makes expensive.
Follow-up
- What breaks first when traffic grows ten times?
- What would you drop to keep the system up under load?
Can you walk through virtualization questions and scenarios around how you would solve infrastructure or runtime issues?
Medium · system design and architecture
Approach
- Choose a partition key and say what query it makes expensive.
- State the consistency you need, and where you are willing to be stale.
- Name the read and write paths separately; they rarely have the same bottleneck.
Follow-up
- What would you drop to keep the system up under load?
- What breaks first when traffic grows ten times?
How would you design a case study integration of a microservice into an existing enterprise landscape?
Medium · system design and architecture
Approach
- Fix the scope first: who calls this, how often, and what they do when it fails.
- State the consistency you need, and where you are willing to be stale.
- Name the failure you are designing for, then the recovery path.
Follow-up
- How does this behave when that dependency is down for an hour?
- What would you drop to keep the system up under load?
Where do you see yourself professionally in five years?
Medium
Approach
- Clarify what is being asked and what a complete answer contains.
- Say what you would check first and why it is the highest-information step.
- State your assumptions explicitly before working the problem.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
What are your career history, professional aspirations, and reasons for wanting to change jobs?
Medium
Approach
- Work from the requirement backwards to the design.
- Say what you would check first and why it is the highest-information step.
- Clarify what is being asked and what a complete answer contains.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
Expose unknown outcomes and rate limit a retrying caller
Hard · partial failure · rate limiting · idempotency
At 3,000 authorisation requests per second, a processor slows past your callers' 2 s timeout and every merchant's client library retries three times. A caller cannot distinguish a lost response from a lost request. Design two things: the API surface that lets a caller resolve an unknown outcome without re-attempting the payment, and the rate limiting that sheds the resulting load without making outcomes unknowable. State the limiter's algorithm and scope, the status and headers returned on rejection, and which requests must never be shed.
Approach
- Order the writes so an unknown outcome always leaves evidence. Persist the intent and the idempotency key as in_progress and commit, then call the processor, passing your own client reference so the attempt is queryable by an identifier you chose. Calling first and persisting after is the one ordering that can produce a charge with no local record at all.
- Make the unknown state a first-class, readable resource: GET /payments/{id} and a lookup by idempotency key, both returning status processing with a hint. The caller's recovery then becomes a read rather than a second write. Retrying the write with the same key is also safe, but a read is cheaper, cannot be confused with a new attempt, and is what you want a panicking integration to reach for.
- Converge without the caller. A sweeper picks up in_progress rows past their lease, queries the processor by the stored reference, and resolves the row, so a caller that never returns still ends with a correct record and the reconciliation does not open a break.
- Scope the limiter to merchant plus operation and use a token bucket sized to the merchant's committed rate with a burst that covers one client's retry fan-out. A global limiter cannot tell whose retries are amplifying and will shed the innocent; a fixed window lets through twice the limit across a boundary.
- Reject with 429, Retry-After and remaining-quota headers, and make the rejection cheap and early: no idempotency row, no processor call, no database write. That is what makes a shed request unambiguously 'never happened' and therefore safe to retry. Exempt the status read from the write bucket, or give it its own generous one, because a limited caller that cannot read state can never learn the outcome of the write it already made.
- Cap amplification at the source and put it in the client contract: exponential backoff with full jitter, plus a retry budget capping retries at roughly a tenth of recent successful requests rather than a fixed count per call. Three fixed retries at 3,000 rps is 12,000 rps arriving precisely when the system is already saturated, which is how a slow dependency becomes an outage that outlives the slowdown.
Worked solution (45 min)
- Write the ordering as a sequence diagram and mark the single crash point that can produce a charge with no local record; confirm your ordering eliminates it.
- Implement the status lookup by idempotency key and assert it returns processing, not 404, for a row that is in_progress.
- Put the limiter in front of everything, including authentication, and assert with a counter that a shed request touches no table and makes no outbound call.
- Load-test at 3,000 rps with an injected 3 s processor delay and a client doing three fixed retries, record the observed arrival rate at the origin, then re-run with backoff plus jitter and a 10% retry budget and compare.
- Run the sweeper against 1,000 abandoned in_progress rows and assert every one resolves to a terminal state with exactly one charge each.
Expected result: Arrival rate at the origin approaches 4x offered load under fixed retries and stays near 1.1x under the budgeted policy; every shed request leaves no idempotency row; every abandoned in_progress row is resolved by the sweeper to a terminal state matching the processor's view.
Verify it yourself
- A 429 response carries Retry-After and leaves the idempotency key unclaimed, verified by re-sending the same key and getting a fresh attempt rather than a conflict.
- The status read still succeeds for a merchant whose write bucket is exhausted.
- After the load test, the count of charges at the fake processor equals the count of distinct idempotency keys offered, not the count of HTTP requests.
Follow-up
- The processor's own status query starts timing out too. What does the sweeper do, and how does it avoid becoming the next amplifier?
- A merchant claims your 429 cost them a sale. What in the design lets you prove the request never reached the processor?
- One merchant is 90% of your traffic. What does per-merchant bucketing do for everyone else, and what does it fail to protect?
Reconciliation breaks that close overnight and then reopen
Medium · business date · timezone · reconciliation · data model
Reconciliation opens 400 to 900 ledger_only breaks each morning, most resolve on the next run, and a subset reopens two days later. The three-way match joins on (external_reference, amount_minor, currency, business_date). ledger_entry.business_date is populated as posted_at::date, where posted_at is timestamptz. The processor closes its day at 22:00 US/Eastern. Breaks cluster on movements posted between 00:00 and 03:00 UTC. Give the ordered checklist, the cause, the fix, and what happens to the breaks already in the table.
Approach
- Test the hypothesis with data you already have rather than by reasoning about it. Bucket open breaks by hour of day on posted_at. A cutoff mismatch concentrates breaks in a fixed window every day and produces almost none outside it; a flaky job produces breaks with no time-of-day structure. One query eliminates half the candidate explanations.
- Compute the offset explicitly and note that it moves. A 22:00 US/Eastern cutoff is 03:00 UTC next day under EST and 02:00 UTC under EDT, so the processor's business date D spans UTC 03:00 on D to 03:00 on D+1, while posted_at::date assigns D to UTC 00:00 through 24:00. The disagreement window is exactly 00:00 to 03:00 UTC, narrowing to two hours during EDT, which matches the observed cluster.
- Find the second bug in the same expression. Casting a timestamptz to date in PostgreSQL applies the session TimeZone, so the identical query returns different dates for different sessions and for the same session after a SET. A business date that depends on who is asking is not a business date.
- Name the missing column rather than a better expression. business_date is a business fact determined by a cutoff rule and a business-day calendar; it is not derivable from any UTC timestamp. It must be stored and set at write time from the rule, with posted_at kept separately for ordering.
- Account for the self-resolving and reopening pattern before claiming the cause is complete. Check whether the day-spanning fuzzy fallback is resolving breaks by matching against an adjacent date, and whether two movements sharing amount, currency and merchant on adjacent dates can be crossed by it. A crossed pairing resolves today and surfaces as duplicate_match later, which is exactly the resolve-then-reopen shape and is a second reason not to widen tolerance.
- Plan the correction against the append-only rule. The trigger forbids UPDATE on a posted entry, so a business_date correction is either a deliberate, logged exception for a non-financial classification column with the trigger amended, or a separate correction table joined at read time. Pick one and get it signed off, because it decides whether prior statements change. Then re-run the matcher over corrected dates rather than bulk-resolving the open breaks; whatever survives is the genuine break population the date bug was masking.
Follow-up
- The fuzzy fallback resolved some breaks by matching an adjacent date. How do you prove it never crossed two lines?
- What does your fix do on the DST transition days themselves, when the local day is 23 or 25 hours long?
- Where does the business-day calendar live, who updates it for a newly announced holiday, and what breaks if they forget?
Suggested practice checklist
For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.
Day 1 — Numbers before diagrams
- Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
- Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
- Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.
Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.
Day 2 — One system, from requirements to schema
- Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
- Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
- Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.
Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.
Day 3 — The consistency you are actually buying
- Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
- Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
- Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.
Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.
Day 4 — Failure is the design
- For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
- Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
- Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
- Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.
Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.
Day 5 — Scaling the hot path
- Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
- Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
- Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
- Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.
Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.
Day 6 — Keep the coding hand in, at the bar that applies to you
- Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
- Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
- Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.
Deliverable: One problem hardened to review standard, with its test list and one written review comment.
Day 7 — Defend it while being interrupted
- Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
- Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
- Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.
Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.
Risky changes and the reversal path
For anything that touched live traffic, be ready to say how you would have undone it: a flag, a staged rollout, dual writes with the old path still authoritative. Once the old column is dropped or the source rows are overwritten there is no reverse, so name what you kept a copy of and for how long.
How do you typically fit into a new team and build collaborative working relationships?
Medium · behavioural and engineering judgement
Approach
- State the situation in two sentences and spend the rest on the reasoning.
- Close with what you would do differently, concretely.
- Pick a story where you made the decision, not one where you watched it.
Follow-up
- What would you do differently if you ran that again?
- What did you decide not to do, and why?
How do you handle project workflows, deadlines, and shifting priorities?
Medium · behavioural and engineering judgement
Approach
- Name the disagreement and how you resolved it with evidence.
- Give the blast radius: what could have broken, and what you measured.
- State the situation in two sentences and spend the rest on the reasoning.
Follow-up
- How did you know your change caused the improvement?
- What would you do differently if you ran that again?
Ship payouts with named, scheduled reconciliation debt
Medium · technical debt · reconciliation · scope cutting
You have six working days to ship merchant payouts. The correct reconciliation join is on (external_reference, amount_minor, currency, business_date); the version that fits the window matches on amount and business_date alone, which will mismatch same-amount, same-day lines. Describe a time you shipped with debt you knew about. State exactly what you cut, the guard that made the shortcut visible rather than silent, who agreed to carry the risk, the date it was paid down, and what it cost while it stood. Be specific about the failure you were accepting.
Approach
- The probe is whether your shortcuts are bounded and instrumented or merely undocumented. Start by stating the failure you accepted in one concrete sentence: ambiguous matches will be resolved arbitrarily, so a genuine break can be masked by pairing with the wrong line.
- Make the shortcut loud rather than silent. When more than one candidate matches, do not pick one: open a break of type duplicate_match and emit a counter on that path. A shortcut that reports its own frequency converts an unknown into a number you can bring to the paydown conversation.
- Bound the exposure before you ship: query historical settlement files for lines that collide on (amount_minor, business_date) within a merchant. If that is 0.2 percent of lines, the risk is small and quantified; if it is 8 percent, the plan does not survive contact and you have learned that before shipping rather than after.
- Name the owner and the date, and say where they are written down. 'The team agreed' is the generic answer; a ticket with an owner, a date, and the counter that forces the conversation is the strong one.
- Distinguish debt you can pay from debt that compounds. A matching shortcut that leaves breaks visible is recoverable; one that auto-resolves breaks by widening a tolerance destroys the evidence and cannot be unwound later, and a strong answer says it would have refused that version even under the deadline.
- Report the actual cost: how many ambiguous matches occurred, how much analyst time they consumed, and whether the paydown date held. If it slipped, say by how long and what finally forced it.
Follow-up
- The paydown date arrives and the team has a new deadline. What do you do differently from the first conversation?
- Which shortcut would you have refused to ship at any deadline, and why that one?
- How would you have sized the exposure if no historical settlement files were available?