cswork / libraryOpen practice workspace

Interview guides · FHLBank Chicago

FHLBank Chicago Software Engineer Interview Questions & Guide 2026

Code reading practice for FHLBank Chicago software engineer prep: work out what unfamiliar code does before judging it.

The 60-second brief

A Software Engineer at FHLBank Chicago supports the financial infrastructure that enables the bank's member institutions to provide housing finance and community investment. Beyond writing code, you build and maintain the secure, scalable applications that support FHLBank Chicago's mission-driven banking operations. The work here is characterized by a balance of modern development practices and the high-reliability standards required of a Federal Home Loan Bank.

Prepare in one language you know well enough to debug in rather than the one you think reads best. Under a clock an unfamiliar language costs you standard-library lookups and iteration mechanics, and that time comes out of your thinking budget, not your typing budget.

Focus areas

  • Make every money-moving endpoint idempotent by key
  • Name the isolation level each invariant requires
  • Reconcile the ledger against processor settlement files

Overview

You will likely contribute to critical internal systems, ranging from member engagement and CRM platforms to GRC (Governance, Risk, and Compliance) tools. Whether you are working on HR/Finance systems or leading quality engineering initiatives, your contributions directly affect how FHLBank Chicago delivers value to its members. Expect a professional environment where technical precision is valued alongside the ability to manage complex, multi-stakeholder project requirements.

The FHLBank Chicago interview process

FHLBank Chicago candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Phone Screen

Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.

What to demonstrate

  • Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
  • Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
  • Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled

How to prepare

  • Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
  • Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
  • Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market

Technical Interviews

Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.

What to demonstrate

  • Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
  • Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
  • Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
  • Whether a failing case is isolated and explained before any edit is made to the code

How to prepare

  • From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
  • Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
  • Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.

Behavioral Fit Assessment

Your first answer is not really what is scored. It buys the follow-up questions, and those decide the round. An interviewer with fifteen minutes takes one thread and pushes on it four or five times, so a story you can only tell at a single level of detail collapses under the third why. That is an argument for fewer stories known deeply rather than one prepared per prompt. Four or five pieces of work you can still explain down to the code you changed and the argument you had about it will cover nearly anything asked in this round.

What to demonstrate

  • Whether a story holds as the questioning moves from what you did to why that instead of the alternative, and then to what you would change knowing what you know now
  • Whether you can re-cut a project to answer the question actually asked rather than delivering a rehearsed block that answers an adjacent one
  • Whether your level of detail is chosen rather than habitual: going down to the schema when the question is about the data model, staying out of it when the question is about the person who disagreed with you

How to prepare

  • Pick four projects and write the chain out four levels deep for each: what you did, why that, why not the alternative, and what would have to be true for the alternative to have won. Where you cannot reach the fourth level, you have a placeholder rather than a story
  • Have someone ask why three times in a row on a single thread with nothing else added, and mark the point where you start repeating a sentence you already said. That point is where the interviewer stops learning anything
  • Build a one-page index instead of an answer bank: the common prompts in this round (disagreement, a failure that was yours, thin requirements, a deadline you missed, work you inherited) mapped to which of your four projects you would use for each, so the choosing is done now rather than while an interviewer waits

Where candidates lose points

Treating money as a decimal with two places

ISO 4217 exponents are 0 for currencies such as JPY and KRW, 2 for most, and 3 for BHD, KWD, JOD, OMR and TND, so a hard-coded multiply-by-100 is off by a factor of 100 or 10 depending on the currency, in opposite directions. Floating point is worse: IEEE 754 binary64 cannot represent 0.1 exactly, so repeated accrual accumulates drift that appears as a handful of minor units in the daily reconciliation and then gets 'fixed' by widening the match tolerance, which is how a genuine break becomes invisible. The only forms that survive a reconciliation are integer minor units with the exponent carried alongside the currency code, or a fixed-scale decimal type with exactly one documented rounding point.

Deriving the business date from the UTC timestamp

Posting date, value date and the processor's settlement date are three different dates, determined by cutoff times, business-day calendars and holidays rather than by midnight UTC. A movement recorded at 23:50 on one side of a cutoff belongs to the next business date, so computing business_date as created_at::date makes daily totals disagree with every statement and every settlement file. The signature is a reconciliation break that resolves itself the following day and then reopens, which reads like a flaky job and is actually a data model that is missing a column: business_date has to be stored explicitly and set from the cutoff rule, with the timestamptz kept separately for ordering.

Quoting amortised or average cost as if it were a worst-case guarantee

Appending to a dynamic array is amortised O(1), but the append that triggers a resize copies every element, and hash lookup is constant only while the hash spreads the actual keys. Say which guarantee you are offering when the caller cares about the latency of one call rather than the total over many.

Tests that assert on the implementation rather than the behaviour

Assert on what a caller can observe, not on the number of internal calls or the shape of a private field. A test that breaks on every refactor but still passes when the answer is wrong costs more than it protects.

Questions & practice

Match a settlement file to ledger postings under duplicate keys

Medium · hash join · reconciliation · external memory

You have one business date of ledger postings (about 12 million rows: transaction_id, source_id, amount_minor, currency, business_date) and the processor's settlement file (about 12 million lines: settlement_line_id, external_reference, amount_minor, currency, business_date), where source_id carries the external reference. Match one-to-one on (external_reference, amount_minor, currency, business_date). Duplicate keys occur legitimately — the same amount can appear twice. Emit every unmatched item classified ledger_only, file_only or duplicate_match, in linear expected time. Then say what you do when neither side fits in memory.

Approach

  1. Build the smaller side into key -> deque of row ids, never key -> row id. A duplicate key is data, not corruption; a single-row map drops one of a legitimate pair and the break report then shows a file_only that does not exist.
  2. Probe the larger side once, carrying one extra bit per bucket: whether that bucket was ever hit. Pop from the bucket on a match and set the bit. An absent key is a probe-side-only row. A present-but-empty bucket means the probe side holds more copies than the build side — surplus, so duplicate_match. After the pass, a leftover non-empty bucket that was never hit is build-side-only; one that was hit is build-side surplus, so also duplicate_match.
  3. That bit is what makes the classification a function of the per-key counts rather than of which side you happened to build. For a key with L ledger and F file copies: min(L, F) match, and the |L - F| surplus rows are duplicate_match tagged with the side that is over, degenerating to ledger_only or file_only exactly when min(L, F) is 0. Without the bit, surplus is only observable as a present-but-empty bucket, which can only ever happen on the probe side — so the same input reports different break classes depending on build order, and the smaller-side heuristic in bullet one silently decides which.
  4. A genuine amount difference does not surface as amount_mismatch here, because the amount is inside the key — it surfaces as a ledger_only and a file_only sharing a reference. Promote those in a second, separate pass keyed on reference alone, recording signed delta_minor as ledger minus file. Keep that promotion out of the exact pass.
  5. Cost: O(N+M) expected time and O(min(N,M)) memory; the hit bit packs into the bucket header and changes neither bound. The constant is the hash map, roughly 60 to 100 bytes per entry in most runtimes, so 12 million rows is order 1 GB — measure it rather than assert it.
  6. When neither side fits, use a grace hash join: partition both sides with the same hash function into P spill files so a key lands in the same partition on both sides, then join partition by partition in memory. Cost is two extra sequential passes; skew inside one partition is the failure mode, handled by re-partitioning that partition under a second hash.
  7. Sort-merge is the alternative at O(N log N + M log M) with external sort, and it wins when the file already arrives sorted by reference or the output must be ordered. It also gets the surplus classification for free, since a merge sees L and F side by side. Whichever you pick, do not widen the amount comparison to make breaks disappear: a tolerance wide enough to absorb rounding is wide enough to absorb a real loss.

Worked solution (30 min)

  1. Build the bucket map over the ledger side and assert that total bucket length equals row count — that single assertion catches the single-row-map bug immediately.
  2. Probe with the file side, popping on match and setting each touched bucket's hit bit, then drain the leftovers: hit means surplus, never hit means absent on the probe side.
  3. Fixture of 11 ledger rows and 11 file lines: 7 keys matching one-to-one, one key where the ledger has 2 copies and the file has 3, 2 ledger-only rows and 1 file-only line.
  4. Assert the counting identity 2*matched + ledger_only + file_only + duplicate_match == N + M; it holds under either build order, so it is necessary but not sufficient — it does not catch a misclassification that moves a row between the three break classes.
  5. Swap build and probe sides and re-run, asserting the four counts and the surplus side are identical, not merely mirrored. Then delete the hit bit and re-run the swap to watch the surplus file copy get reclassified as an absence.

Expected result: 9 matched pairs, 2 ledger_only, 1 file_only, 1 duplicate_match — the file's surplus third copy — satisfying 2*9 + 2 + 1 + 1 = 22 = N + M. Building on the file side and probing with the ledger returns those same four counts with the surplus still attributed to the file, because every label is derived from L and F. Without the hit bit it does not: the swapped run reports ledger_only 2, file_only 2, duplicate_match 0 — the identity still sums to 22, and the surplus file copy has been silently reclassified as an absence, which is the failure this fixture exists to catch.

Verify it yourself

  • The counting identity holds on every run, including the skewed one.
  • The full break classification — counts and per-row labels — is identical when build and probe sides are swapped. A diff there means a label is being read off the join's mechanics rather than off the data.
  • Signed delta_minor on a promoted amount_mismatch is ledger minus file, and flips sign when the sides are swapped.
  • Forcing P = 1 in the grace-hash path reproduces the in-memory join's break set exactly, proving partitioning is not changing the answer.

Follow-up

  • The file nets three fee lines into one batch total. Which pass catches that, and what is its stopping rule?
  • The processor's business date sits one cutoff behind yours for forty minutes of traffic. What does that do to the exact join, and what does it do to break ages?
  • The same break recurs on the next run. Why must it link to the existing reconciliation_break row rather than open a second one?

Derive per-account balances and catch unbalanced transactions

Easy · aggregation · double-entry · streaming

You are given ledger_entry rows streamed in entry_id order: transaction_id, account_id, direction (debit or credit), amount_minor (a positive int64), currency, business_date. Up to 500 million rows, at most 20 million distinct (account_id, currency) pairs, and the entries of one transaction are contiguous in the stream. In a single pass with no re-reads, return the closing balance per (account_id, currency) and the transaction_id of every transaction whose entries do not sum to zero within each currency. State your time and space bounds.

Approach

  1. Normalise the sign at read time from direction, not from the amount: signed = +amount_minor for debit, -amount_minor for credit (state which convention you picked). The schema constrains amount_minor > 0 precisely so the sign lives in exactly one place.
  2. Hold one hash map keyed (account_id, currency) to an int64 running total. Twenty million keys at 16 bytes of payload plus map overhead is order 1 GB in most runtimes — quote the number, and offer the fallback: partition the stream by hash(account_id) % P and run P passes for 1/P of the memory.
  3. Ride the zero-sum check on the same pass. Because a transaction's entries are contiguous, keep a tiny currency -> int64 map for the current transaction_id only, test it against zero on the boundary and at EOF, then clear it. That is O(currencies in one transaction), typically one or two.
  4. Bound the arithmetic explicitly. Int64 holds about 9.22e18, so overflowing one account across 500 million entries needs an average of 1.8e10 minor units per entry — safe here, but use a checked add so an adversarial file fails loudly rather than wrapping.
  5. Complexity: O(n) time, O(distinct account-currency pairs) space, one sequential pass, no sort. The zero-sum check adds no asymptotic cost, which is the argument for doing it here rather than in a second job.

Follow-up

  • Entries of a transaction are no longer contiguous. What does the zero-sum check cost now, and which is cheaper: buffering open transactions or an external sort on transaction_id?
  • How would you produce the same balances as of an arbitrary business_date without a second full scan?
  • The job is restarted after a crash halfway through the file. What makes the second run produce identical output?

Resolve party identity across merges without rewriting posted history

Medium · union-find · identity resolution · immutable history

Duplicate parties get merged over time: a stream of up to 5 million (loser_party_id, winner_party_id, merged_at) events, deliverable in any order, sometimes repeating a pair already merged. Ten million account rows and hundreds of millions of ledger_entry rows already reference pre-merge party ids and must not be rewritten. Build the structure that answers 'what is the canonical party for this id?' in near-constant time, and explain how a report that groups by party stays correct without touching a single historical row.

Approach

  1. The merge events form an undirected graph over party ids and the question is connected components, so use disjoint-set union with union by size and path compression: find and union amortise to O(alpha(n)), and alpha(n) is at most 4 for any n that exists. Five million merges over ten million ids is two integer arrays and a few hundred milliseconds.
  2. The domain twist: the representative cannot be whichever root union-by-size happens to pick, because the canonical party is a business decision recorded as winner_party_id. Keep union by size for the tree shape and a separate root -> canonical_party_id map for the label. Conflating them makes the canonical id flip when an unrelated merge reshapes the tree, and every report built on it moves retroactively.
  3. Never rewrite ledger_entry or account. Resolve at read through the canonical map: rewriting posted rows destroys the audit trail, locks hundreds of millions of rows, and buys nothing the map does not already give you.
  4. Materialise the map fully expanded as party_id -> canonical_party_id, one row per member rather than per edge, so a report is a single join instead of a recursive traversal. Rebuild it from the event log, which keeps it a derived artefact that can be regenerated rather than a second source of truth that drifts.
  5. Pin down determinism: a repeated merge is a no-op because both ids already share a root, but two events merging the same pair in opposite directions is a genuine conflict. Write the tiebreak down — lowest merged_at, then lowest id — or two replays of the same log produce different labels for the same data.
  6. Un-merges: disjoint-set union has no delete. If a merge can be reversed, keep the event log authoritative and rebuild the whole structure from the surviving events; at 5 million events that is seconds, and far cheaper than any incremental un-union scheme.

Follow-up

  • A merge is reversed after six months of postings. What exactly gets rebuilt, and what does the statement produced last month now say?
  • Two processes apply merge events concurrently. What guarantees they converge on an identical canonical map?
  • Resolving at read on the hot path versus materialising the map — where does that join actually live, and what does each cost?

Store multi-currency amounts and prove each transaction balances

Easy · minor units · normalisation · constraints

ledger_entry holds entry_id, transaction_id, account_id, direction (debit or credit), amount_minor (bigint, CHECK > 0), currency char(3), source_type, source_id, business_date and posted_at. The service posts in JPY, USD and KWD. Specify the column types for money and the reference table that carries each currency's ISO 4217 minor-unit exponent, and say why that exponent must not be a constant in application code. Then write the query that returns every transaction_id whose entries fail to sum to zero within each currency, ordered by the largest absolute imbalance.

Approach

  1. State the representation: amount_minor is an integer count of minor units and the scale lives with the currency, not in the column. int4 tops out at 2,147,483,647 minor units, which is about 21.5 million units of a 2-exponent currency, so bigint, and the currency code travels on every row that carries an amount.
  2. Normalise the scale into currency(code char(3) primary key, exponent smallint, name) and join it only at display and parse boundaries. The exponent is 0 for JPY and KRW, 2 for USD and EUR, 3 for KWD, BHD, JOD, OMR and TND, so a hard-coded divide by 100 is wrong by 100x for JPY and by 10x for KWD, in opposite directions.
  3. Rule out binary floating point outright: IEEE 754 binary64 cannot represent 0.1, so repeated accrual drifts by a few minor units that later appear as reconciliation breaks. If a fixed-scale decimal is used instead, pin the scale per currency and name the single place rounding happens.
  4. Write the balance check as a grouped aggregate: sign the amount with a CASE on direction, GROUP BY transaction_id, currency, and HAVING the signed sum <> 0. The sign belongs in direction, and the CASE is the one place it becomes arithmetic.
  5. Then order in a second layer, because the ordering the question asks for cannot be written on the grouped query itself. PostgreSQL lets an output alias stand alone in ORDER BY but resolves anything inside an expression against the input columns, so ORDER BY abs(imbalance_minor) DESC raises SQLSTATE 42703, column "imbalance_minor" does not exist, while the bare ORDER BY imbalance_minor DESC is accepted and sorts signed rather than absolute. Wrap the grouped query and order by abs() outside it, or repeat the aggregate as ORDER BY abs(SUM(CASE ...)) DESC, which is legal because aggregates are allowed there.
  6. Group per (transaction_id, currency) rather than per transaction, because a cross-currency movement balances only within each currency, bridged by an explicit FX position account whose rate, source and timestamp are stored on the transaction.
  7. Note that no CHECK constraint can express this, since it spans rows: enforce it at write time with a DEFERRABLE INITIALLY DEFERRED constraint trigger that fires at commit, or by routing every posting through one function, and keep this query as the independent audit.

Follow-up

  • A transaction has a USD leg and a JPY leg. Which accounts appear, and where does the rounding residual land?
  • How would you enforce the zero-sum rule at write time without serialising all postings behind one lock?
  • Someone proposes storing a display amount alongside amount_minor. What goes wrong at the first rate change or rounding rule change?

Make a charge endpoint safe under concurrent duplicate retries

Easy · idempotency · unique constraints · read committed

idempotency_key holds id, scope, key, request_fingerprint (SHA-256 over the canonicalised body), status (in_progress, completed, failed), response_status, response_body, locked_at, completed_at, expires_at, created_at. Fifty identical create-payment requests carrying the same scope and key reach four application instances inside the same 20 ms. Give the DDL constraint and the exact statements the handler runs so that exactly one payment_intent is created and all fifty callers receive the same response body. State what you return when that key arrives with a different fingerprint, and what an arrival after expires_at means.

Approach

  1. Put the concurrency control in the schema: UNIQUE (scope, key). A SELECT-then-INSERT cannot work because both transactions can read nothing before either commits, so the check passes twice and the constraint then surfaces as an error on a payment that succeeded.
  2. Claim the key with INSERT ... ON CONFLICT (scope, key) DO NOTHING RETURNING id. A conflict returns zero rows rather than the existing row, so branch on rowcount: the winner proceeds, the loser reads the stored row.
  3. Keep that path on READ COMMITTED deliberately. The loser's follow-up SELECT takes a fresh statement snapshot and therefore sees the winner's committed row; under REPEATABLE READ the transaction snapshot predates that commit, the row stays invisible and the loser concludes the key does not exist.
  4. Split the work across two transactions because the processor call cannot sit inside one: commit the in_progress row with locked_at first so losers can see a claim, perform the effect, then write payment_intent plus status=completed with response_status and response_body in a single second transaction.
  5. Handle the crash window explicitly: a row stuck in_progress past its lease is an unknown outcome, not a failure, so the reaper queries the processor for that key before deciding. A loser that sees in_progress returns 409 and retries rather than repeating the effect.
  6. Compare request_fingerprint before replaying anything. Same key with a different body is 409, never the cached response, because replaying confirms a payment the caller did not request; and set expires_at beyond the client's and the processor's maximum retry horizon, since a replay after it is a genuinely new request.

Worked solution (20 min)

  1. Create idempotency_key with UNIQUE (scope, key) and write the claim statement as INSERT ... ON CONFLICT DO NOTHING RETURNING id, plus the fallback SELECT on zero rows.
  2. Drive it with 50 threads sending the identical body, and assert one payment_intent row and fifty byte-identical response bodies.
  3. Re-send the same key with amount_minor changed by one unit and assert 409 plus no new payment_intent row.
  4. Kill the process between the processor call and the second commit, restart, and show the reaper resolves the in_progress row by querying the processor rather than by re-charging.

Expected result: One payment_intent row, one idempotency_key row at status=completed, fifty responses with identical response_status and response_body, zero 5xx. A 51st request with the same key and a changed amount returns 409 and leaves payment_intent untouched.

Verify it yourself

  • Repeat the 50-way race 20 times; COUNT(*) FROM payment_intent for that key is 1 on every run.
  • Assert the fifty bodies are byte-identical, not merely all 2xx.
  • Drop the UNIQUE constraint and confirm the test now fails; if it still passes, the race is not being exercised.
  • Re-run the loser path under REPEATABLE READ and observe it failing to find the winner's row, which is why the level is pinned.

Follow-up

  • The handler dies after the processor call and before the local commit. What does the next retry with that key observe, and how does the system converge on exactly one charge?
  • Does the downstream processor honour an idempotency key of its own? Who mints it, and what breaks if a fresh one is generated per attempt?
  • How do you purge rows past expires_at without the delete contending with the insert path?

Are you familiar with AGILE methodology?

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Say what you would check first and why it is the highest-information step.
  3. Clarify what is being asked and what a complete answer contains.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

What are the core principles of effective software testing?

Medium

Approach

  1. Clarify what is being asked and what a complete answer contains.
  2. Say what you would check first and why it is the highest-information step.
  3. Work from the requirement backwards to the design.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Specify the idempotency key contract for payment creation

Medium · idempotency · concurrency · api contract

POST /payments takes an idempotency key header plus amount_minor, currency, merchant and instrument token. The caller is a merchant server that retries on timeout and may retry concurrently from two workers. Storage is idempotency_key (id, scope text, key text with UNIQUE (scope, key), request_fingerprint bytea, status enum in_progress/completed/failed, response_status smallint, response_body jsonb, locked_at, completed_at, expires_at). Specify the scope, what the fingerprint covers, the response when the same key arrives with a different body, the response while the first request is still in flight, and what a replay after expires_at does.

Approach

  1. Scope the key to caller plus operation, for example 'merchant:1234:create_payment'. A global key space lets two merchants collide, and a key scoped only to the merchant lets a key used on one endpoint replay that endpoint's stored response for a different operation.
  2. Fingerprint the canonicalised body: sorted keys, no insignificant whitespace, amounts as integer minor units. Cover the fields that determine the effect and deliberately exclude client metadata such as a trace id, so a retry that adds one is still recognised as a retry. Same key with a different fingerprint is 409 with its own code, never the cached response and never the new effect.
  3. Win the concurrent race in the database: INSERT ... ON CONFLICT DO NOTHING RETURNING id. An empty return means this request lost, so it reads the winner's row. A SELECT-then-INSERT is not a fix; under READ COMMITTED both workers take fresh snapshots and both pass the check.
  4. Define in_progress explicitly. The loser reads status in_progress and receives 409 with a retry hint rather than a blocking wait or a second attempt. Note that Retry-After is defined for 429, 503 and 3xx, so on a 409 it is an extension you document rather than something the caller's HTTP stack already honours. A locked_at older than the lease is reclaimable only through a conditional UPDATE that matches the old locked_at, otherwise two reclaimers both proceed.
  5. Commit the key row before calling the processor, so a crash leaves evidence of an attempt that can be queried and converged. On completion store response_status and response_body and replay them verbatim, including the original status code.
  6. State the retention window (24 hours is a common choice) and its consequence: after expires_at the same key is a new request, so the caller's total retry deadline must sit inside the window or the mechanism silently stops protecting anything. The header name comes from an IETF draft rather than a ratified RFC, so document it as your convention.

Worked solution (30 min)

  1. Implement the endpoint over PostgreSQL with UNIQUE (scope, key), ON CONFLICT DO NOTHING RETURNING, an in_progress row written and committed before the processor call, and a stored response replayed on completion.
  2. Fire 50 concurrent identical requests with one key and assert one payment row, one processor call, and 50 byte-identical response bodies and status codes.
  3. Re-send the same key with amount_minor changed and assert 409 with the fingerprint-mismatch code and no new payment row.
  4. Kill the process between the processor call and the local commit, restart, and let the sweeper query the processor by the stored reference; assert the key converges to completed with exactly one charge.
  5. Advance the clock past expires_at, replay the key, and assert the documented behaviour (a new payment) rather than an undefined one.

Expected result: One payment row and one processor call for the 50 concurrent requests, identical replayed responses including status code, 409 with a distinct code for the changed body, and convergence to a single charge after the injected crash.

Verify it yourself

  • An instrumented processor client records exactly one outbound call across the 50 concurrent requests.
  • The losing inserts never execute the effect; they only read the winner's stored response.
  • The post-expiry replay behaviour is asserted by a test rather than left to whatever the code happens to do.

Follow-up

  • The processor call succeeds but your commit fails. What does the next request with the same key see, and what converges the row?
  • A merchant reuses one key for two genuinely different payments a month apart. Which of your rules catches it, and which lets it through?
  • Where does the key live end to end, and what changes if the downstream processor honours its own key?

One merchant stops receiving webhooks while the rest deliver

Easy · poison message · head-of-line blocking · outbox

The outbound relay delivers outbox_event rows to merchant endpoints preserving per-destination ordering. One merchant has received nothing since 09:14; every other destination is current. The relay process is healthy and CPU is flat. For that merchant, unpublished rows are accumulating, the oldest has attempts = 412 with backoff capped at 30 seconds, and last_error is the same string on every attempt. Diagnose in order, and state what you change so one undeliverable row can never stall a destination again.

Approach

  1. Separate destination down from one row undeliverable before anything else, because they are identical from the merchant's side. The discriminator is the error class: a transport error (connection reset, 503, TLS failure) varies and implicates the endpoint, while an identical deterministic error on every one of 412 attempts implicates the row.
  2. Read the blocked set in version order: SELECT event_id, aggregate_id, aggregate_version, attempts, last_error FROM outbox_event WHERE published_at IS NULL AND aggregate_id = $1 ORDER BY aggregate_version LIMIT 5. If everything is queued contiguously behind one row, the ordering guarantee is working exactly as designed and the defect is that it has no exit.
  3. Confirm outside the relay: replay that single payload against the destination by hand. Reproducing the identical error proves the row, not the transport, and stops the escalation to the merchant before it is sent.
  4. Decide what skipping means under a per-destination ordering contract. You cannot simply publish the next row; you either park the poison row in a dead-letter table and accept that the consumer sees an aggregate_version gap, or you deliver a degraded payload. Name which, and say what the consumer does with the gap.
  5. Fix the real defect, which is unbounded retry, not the malformed payload. Bound retries by attempt count or row age, park on exhaustion, alert on the age of the oldest unpublished row per destination, and validate the payload at INSERT time so it cannot be written inside the state-change transaction at all.

Follow-up

  • The consumer orders on aggregate_version. What does it do with the gap your dead-letter created, and how does it learn the gap is intentional?
  • How do you distinguish this from a destination returning 4xx on every request during its own deploy?
  • Does the alert go on attempts, on the age of the oldest unpublished row, or both, and what are the thresholds?

Suggested practice checklist

For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.

Day 1 — Measure before reasoning

  • Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
  • Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
  • Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.

Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.

Day 2 — References, copies, and the bugs they produce

  • Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
  • Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
  • Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.

Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.

Day 3 — Types, once, in a language that checks them

  • Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
  • Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
  • Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.

Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.

Day 4 — Concurrency, starting with what actually runs at the same time

  • Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
  • Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
  • Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.

Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.

Day 5 — Debugging as a procedure rather than an instinct

  • Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
  • Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
  • Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.

Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.

Day 6 — Tests that catch the bug you are about to write

  • Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
  • Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
  • Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.

Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.

Day 7 — Debug something broken, out loud

  • Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
  • Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
  • Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.

Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.

The bug you decided not to fix

Engineers over-index on what they repaired. A stronger answer covers something you knowingly left broken: the alert you tuned down, the data inconsistency you documented instead of chasing, the cleanup you deferred past two quarters. Give the reasoning and the condition that would have reopened it, so it reads as a decision and not as neglect.

Describe a time you had to work with a difficult stakeholder.

Medium · behavioural and engineering judgement

Approach

  1. Close with what you would do differently, concretely.
  2. Pick a story where you made the decision, not one where you watched it.
  3. Name the disagreement and how you resolved it with evidence.

Follow-up

  • What did you decide not to do, and why?
  • What would you do differently if you ran that again?

Walk me through your resume.

Medium · behavioural and engineering judgement

Approach

  1. Pick a story where you made the decision, not one where you watched it.
  2. State the situation in two sentences and spend the rest on the reasoning.
  3. Give the blast radius: what could have broken, and what you measured.

Follow-up

  • How did you know your change caused the improvement?
  • What would you do differently if you ran that again?

Describe your experience with [specific software package].

Medium · behavioural and engineering judgement

Approach

  1. Close with what you would do differently, concretely.
  2. Pick a story where you made the decision, not one where you watched it.
  3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

  • What would you do differently if you ran that again?
  • What did you decide not to do, and why?

How do you manage competing deliverables?

Medium · behavioural and engineering judgement

Approach

  1. Pick a story where you made the decision, not one where you watched it.
  2. Give the blast radius: what could have broken, and what you measured.
  3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

  • What did you decide not to do, and why?
  • How did you know your change caused the improvement?