cswork / libraryOpen practice workspace

Interview guides · Investec

Investec Software Engineer Interview Questions & Guide 2026

Debugging practice for Investec software engineer candidates: reproduce it, narrow it, then change one thing per run.

The 60-second brief

A Software Engineer at Investec plays a pivotal role in maintaining the technical excellence that supports a global specialist bank and wealth manager. You are not just writing code; you are building robust, scalable solutions that underpin complex financial services and client-facing digital products. Your contributions directly impact the efficiency of internal operations and the quality of the experience delivered to Investec clients.

Scope in the operational half of the job when the seat carries on-call. Rollout, rollback and what you would put on a dashboard belong inside a design answer rather than after it, and an answer that never reaches them reads as someone who has built systems but not run them.

Focus areas

  • Reconcile the ledger against processor settlement files
  • Consume webhooks duplicated, delayed and out of order
  • Make every money-moving endpoint idempotent by key

Overview

The environment is fast-paced and highly professional, often requiring a blend of deep technical expertise and strong business acumen. You will likely work within multidisciplinary teams, collaborating closely with product owners, architects, and stakeholders to translate business requirements into high-quality software. Whether you are working on core banking systems, cloud-native applications, or internal infrastructure, you are expected to take ownership of your work, maintain high standards of code quality, and demonstrate a commitment to the firm’s long-term growth.

This role is ideal for engineers who thrive on complexity and value a culture that emphasizes personal accountability, integrity, and intellectual curiosity. Success here requires the ability to navigate ambiguity, communicate clearly with non-technical stakeholders, and deliver solutions that are not only technically sound but also strategically aligned with 's goals.

The Investec interview process

Investec candidates report 6 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

  • Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
  • Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
  • Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

  • Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
  • Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
  • Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Video Interviews

An unlabelled round is first an information problem, and the cheapest information is free. Whoever schedules it can usually tell you how long it runs, who will be in the room and what they work on, whether you will be writing code and in what environment, and whether anything is being sent beforehand. Ask in writing so the answer is on record, then prepare for the two or three formats those answers still leave open instead of betting on one. What separates a strong candidate is not guessing right; it is having an opening that works whichever one it turns out to be.

What to demonstrate

  • Whether you can start work from an ambiguous brief, since tolerating a vague scope without stalling is the same thing the job asks for
  • Whether the questions you asked beforehand were ones that change your preparation, such as duration, medium and who is joining, rather than ones whose answers you could not have acted on
  • Whether you adapt when the round turns out to be something other than what you were told, instead of spending the first ten minutes visibly recalibrating

How to prepare

  • Send one short scheduling message asking four things: how long, who is joining and what they work on, whether you will be writing code and where, and whether to prepare anything in advance. Treat a vague reply as real information, since it means the round is loosely structured and you will be shaping it yourself.
  • Write one opening that works in any of the formats still open: restate in your own words what you have been asked to do, then ask which of two directions is more useful to them. Say it aloud until it stops sounding recited.
  • Set up for the two most likely formats before the call starts, with a blank editor in the language you would choose and a shared document you can type into, so a format surprise costs you nothing in the first minutes

Live Coding Sessions

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

  • Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
  • Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
  • Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
  • Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

  • For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
  • For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
  • Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Panel Discussions

A day like this is several different games in a row, and the expensive mistake is carrying the previous one into the next room. Coding rewards narrow precision and finishing inside a timer. Design rewards breadth, stated assumptions and naming what you are deliberately not building. Behavioural rewards specificity about people and decisions. Candidates who over-engineer a coding problem they were supposed to finish, or who start sketching class hierarchies before anyone has agreed what the system has to do, are usually still playing the last round. Between rooms, name out loud which game the next one is.

What to demonstrate

  • Whether the coding round ends with something that runs and has been traced against a degenerate input, rather than an extensible design that was never finished
  • Whether a design discussion opens by agreeing on traffic shape, read-to-write ratio and what is allowed to be stale, instead of proceeding from an architecture you arrived with
  • Whether a behavioural answer names a person, a disagreement and what you did about it, rather than describing the system the story happened inside
  • Whether the opening habits still appear late in the day: restating the problem, asking for constraints, saying the plan before typing

How to prepare

  • Book three mocks of different types back to back on one afternoon and ask each interviewer afterwards which round you answered in the wrong mode
  • Write a three-line opening script per round type — coding: restate, name the approach and its cost, then type; design: ask for scale, read-write mix and what must not break; behavioural: name the person, the stakes and the decision — and run it off a card so the switch is mechanical rather than remembered
  • Practise coding with a timer you do not extend, stopping when it stops, so the trained reflex is to finish a correct solution rather than to keep improving one

Cultural Assessment

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

  • In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
  • In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
  • Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

  • Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
  • Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
  • For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Final Decision

Nobody in the room with you decides this. Interviewers typically write their rounds up separately, often before seeing anyone else's, and the outcome is settled later from those write-ups. A split panel gets resolved by whichever note carries specific evidence, so what you want out of each room is one concrete thing that person could write down: a bug you caught yourself, a trade-off you named, a decision you owned. The rest is arithmetic. The project you describe in a behavioural conversation is often the same system you sketched an hour earlier, and the two accounts have to agree.

What to demonstrate

  • Whether the scale, team size and timeline you attach to a project hold steady when that project resurfaces in a different round
  • Whether each interviewer leaves with a specific thing to cite rather than a general impression of competence
  • Whether a trade-off you defended in one round survives a challenge in another, instead of being quietly swapped for the answer the new interviewer seemed to want
  • Whether a question you have already answered earlier in the day gets the same answer at the same depth, without visible impatience

How to prepare

  • Write a one-page sheet per project fixing the figures you will quote — request volume, data size, team size, elapsed time, what broke — and say them aloud from the sheet until they come out identical every time
  • For each round on the schedule, decide in advance the one sentence you want in that person's notes, then check in a mock that you said it outright instead of leaving it to be inferred
  • Have someone ask you the same project question twice, an hour apart, and diff the two answers for numbers that moved or a trade-off that reversed

Where candidates lose points

Assuming the default isolation level enforces the invariant you wrote down

PostgreSQL defaults to READ COMMITTED, where every statement takes a fresh snapshot, so a read-modify-write on a balance loses updates under concurrency. Its REPEATABLE READ is snapshot isolation, which blocks that particular anomaly by aborting the loser with SQLSTATE 40001 but still permits write skew across two different rows; only SERIALIZABLE closes that, and both levels therefore require a bounded retry loop on 40001 that many implementations simply never write. MySQL's InnoDB REPEATABLE READ behaves differently again — it does not abort on a conflicting write, so the identical application code silently changes behaviour when the engine changes. Two-sided transfers add a second failure mode on top: without a deterministic lock ordering, such as always locking account ids in ascending order, concurrent opposing transfers deadlock (SQLSTATE 40P01).

Treating money as a decimal with two places

ISO 4217 exponents are 0 for currencies such as JPY and KRW, 2 for most, and 3 for BHD, KWD, JOD, OMR and TND, so a hard-coded multiply-by-100 is off by a factor of 100 or 10 depending on the currency, in opposite directions. Floating point is worse: IEEE 754 binary64 cannot represent 0.1 exactly, so repeated accrual accumulates drift that appears as a handful of minor units in the daily reconciliation and then gets 'fixed' by widening the match tolerance, which is how a genuine break becomes invisible. The only forms that survive a reconciliation are integer minor units with the exponent carried alongside the currency code, or a fixed-scale decimal type with exactly one documented rounding point.

Designing for a scale nobody asked for

Ask for request rate, data size, read-to-write ratio and expected growth, then size the simplest option first; one relational instance on current hardware covers a large share of real workloads. Reaching for shards, queues and a cache tier before any number has been quoted reads as pattern-matching rather than judgement.

Abandoning working code to chase the optimal solution

Get the straightforward version correct, state its complexity, and only then optimise, keeping the working version until the faster one passes the same cases. A correct quadratic solution with a stated path to linear beats a half-written optimal one that never ran.

Questions & practice

How would you read, normalize, and group data from a JSON API file?

Medium · data structures and algorithms

Approach

  1. Restate the input: its shape, its size, and what is guaranteed about it.
  2. Name the brute-force solution and its complexity before improving on it.
  3. Walk one small example through your approach before writing the whole thing.

Follow-up

  • Which test case would catch an off-by-one here?
  • How does this change if the input no longer fits in memory?

Describe a project you worked on that you found particularly interesting.

Medium · data structures and algorithms

Approach

  1. Restate the input: its shape, its size, and what is guaranteed about it.
  2. Walk one small example through your approach before writing the whole thing.
  3. Name the brute-force solution and its complexity before improving on it.

Follow-up

  • How does this change if the input no longer fits in memory?
  • What is the worst case, and how likely is it on real data?

Answer as-of balance queries over an append-only entry log

Hard · prefix sums · offline queries · append-only

Given 400 million ledger_entry rows (entry_id, account_id, direction, amount_minor, currency, business_date) and 2 million queries of (account_id, currency, as_of_date) asking for the balance at the end of that business date, produce every answer. The obvious solution — per query, sum that account's entries with business_date <= as_of_date — is correct. Say precisely why it will not finish, then give one that will, with time and space complexity. Corrections are posted as new entries carrying their own business_date.

Approach

  1. Cost the naive version in numbers before rejecting it. Spread uniformly over 20 million accounts, each query touches about 20 rows behind a per-account index and 2 million queries is 4e7 row touches — perfectly fine. The problem is skew: one pooled clearing or merchant settlement account holding 3e7 entries, taking 10% of the queries, is 6e12 row touches. Name the skew; 'n is large' is not the reason.
  2. The structural fact that buys a cheap answer: entries are append-only and never updated, so a prefix sum over an account's entries ordered by (business_date, entry_id) is stable — nothing behind position i can change. No mutable-balance design offers that, and it is why the storage is worth paying for.
  3. Offline sweep, when all queries are known up front: externally sort entries by (account_id, currency, business_date, entry_id) and queries by (account_id, currency, as_of_date), then merge-walk both with a running sum, emitting each query's answer as the sweep passes its date. O((n + q) log(n + q)) dominated by the sort, O(1) beyond sort buffers, one sequential pass over each input instead of 2 million random seeks.
  4. Online alternative: materialise end-of-day snapshots — one row per (account_id, currency, business_date) that had activity, holding the cumulative total. A query becomes one index seek for the latest snapshot at or before as_of_date, O(log n) per query, over far fewer rows than n. Use snapshots when queries arrive singly and the sweep when they arrive as a batch.
  5. Corrections are the subtlety: an entry posted today but dated back changes historical answers, so every snapshot for that account from that date forward is stale. Either keep a Fenwick tree over dates per account (O(log D) update and prefix query) or recompute that account's snapshots from the corrected date onward. Then be precise about what reproducibility means — yesterday's statement is reproducible as of a stated snapshot time, not identical forever.
  6. Bound the resources: int64 sums throughout, no float; 400 million rows at roughly 48 bytes of the columns you actually need is about 19 GB, so the sort is external and its fan-out is chosen from the sort buffer, not from the row count.

Worked solution (40 min)

  1. Compute both costs explicitly: the uniform case at about 4e7 row touches, and the skewed case at about 6e12. Showing that arithmetic is the answer to 'why'.
  2. Implement the offline sweep on a 10-million-row, 50,000-query fixture, merging on (account_id, currency, business_date, entry_id).
  3. Implement the naive version as the reference answer and assert both agree on every fixture query.
  4. Add a correction entry dated 30 days back, re-run, and assert that exactly the queries with as_of_date on or after that date move, all by the same signed amount.
  5. Measure rows touched and wall time for each at 10 million rows, then extrapolate to 400 million and state the assumption that makes the extrapolation valid — sequential I/O, no random seeks.

Expected result: Both implementations agree on all 50,000 fixture queries. After the back-dated correction, exactly the queries at or after its business_date change, each by the identical signed amount. The sweep touches every entry row once; the naive version touches the hot account's rows once per query against it.

Verify it yourself

  • The balance at the maximum as_of_date equals the plain full-file aggregate for that account and currency.
  • Summed across all accounts at any single date, the result is 0 per currency — provided every entry in a transaction shares one business_date, which is worth asserting separately rather than assuming.
  • The sweep's answers do not change when entries are fed in a different physical order, because the sort key is total.
  • No answer depends on posted_at: swapping two entries' posted_at within one business date changes nothing.

Follow-up

  • One account holds 30% of all entries. What does the external sort do with it, and what would you do for that one key instead?
  • Queries now arrive online at 500 per second. Which design survives, and what does keeping the other one warm cost?
  • A correction lands with a business_date 90 days back. Which snapshots are now wrong, and how does a reader find out?

Decide whether balance is derived or materialised, then hold the floor

Hard · denormalisation · isolation levels · lock contention

Authorisation needs an account's available balance inside an 80 ms budget at 3,000 requests per second; that account already has 200 million ledger_entry rows. A product rule says the balance may never fall below the account's negative overdraft limit. Decide whether the balance is summed from entries or held in a materialised account_balance(account_id, balance_minor, floor_minor, currency, version) row, and justify the choice from the read pattern. Then give the write path that holds the floor, naming the isolation level, the anomaly a weaker level permits, and the SQLSTATE you retry.

Approach

  1. Size the derived read before arguing about it: summing 200 million rows is not an 80 ms operation under any index, since even a covering index on (account_id, entry_id) still reads work proportional to the rows. The authorisation read pattern forces one materialised row fetched by primary key; the statement read pattern, which is low-rate and historical, stays derived from entries. That is the whole justification for the denormalisation, and its price is a write on that row per posting.
  2. Put the floor where it can be a single-row constraint: CHECK (balance_minor >= floor_minor) on account_balance, updated in the same transaction as the entries. As a predicate over a set of entry rows it cannot be a CHECK at all, which is the reason the materialised row earns its keep twice.
  3. Write the mutation as one statement: UPDATE account_balance SET balance_minor = balance_minor - $1, version = version + 1 WHERE account_id = $2 AND balance_minor - $1 >= floor_minor. Zero rows updated means refused. This is safe even under READ COMMITTED, because the UPDATE re-evaluates its predicate against the locked, post-update version of the row.
  4. Name the anomaly in the shape that is not safe: SELECT the balance, compute a new value in application code, then UPDATE to that constant. Every statement under READ COMMITTED takes a fresh snapshot, so two concurrent 60-unit withdrawals against 100 both read 100 and both write 40. The row then claims 40 while 120 has actually left, so the true position is -20, below a floor of 0, and the row it is checked against cannot show it. PostgreSQL REPEATABLE READ is snapshot isolation and aborts the loser with SQLSTATE 40001; SERIALIZABLE additionally closes write skew across two rows. Both require a bounded retry with backoff, and InnoDB REPEATABLE READ does not abort at all, so identical code changes behaviour on a different engine.
  5. For a two-account transfer, acquire the rows in a deterministic order such as ascending account_id; without it, opposing concurrent transfers deadlock and the database kills one with SQLSTATE 40P01. That is a retry, not a correctness failure, but it is a retry somebody has to write.
  6. Finish on the ceiling: the hot row admits one committed write per lock hold, so at a 2 ms hold it caps near 500 per second regardless of cores. Sharding into N sub-rows multiplies throughput and immediately makes the floor check cross-row again, which then needs the shard sum under SERIALIZABLE or a per-shard reserved allowance.

Follow-up

  • Shard the balance into eight sub-rows. Write exactly what the floor check now does, and what it costs per authorisation.
  • Someone proposes an AFTER INSERT trigger on ledger_entry to maintain the balance. What does that change about ordering, about batch posting, and about failure handling?
  • How do you detect that the materialised row has drifted from the entries, how often do you run it, and on which replica?

Add and backfill business_date on a live ledger table

Hard · online migration · lock queue · backfill

ledger_entry holds 4 billion rows, is append-only, takes 10,000 inserts per second, and every reporting query currently derives the business date as posted_at::date. You must add business_date date NOT NULL, populated from the cutoff rule (17:00 in the account's own timezone), backfilled across all history, indexed, and cut over, with no write downtime and no long-held lock. Give the ordered migration steps with the lock each one takes, how you make the backfill restartable and throttled, and how you retire the old expression safely.

Approach

  1. Add the column nullable and with no default. ALTER TABLE ... ADD COLUMN takes ACCESS EXCLUSIVE but is a catalogue-only change held for microseconds. The hazard is the lock queue, not the statement: a blocked ALTER waits behind one long reader holding ACCESS SHARE, and every query arriving afterwards queues behind the ALTER's pending ACCESS EXCLUSIVE, so set lock_timeout to a couple of seconds and retry rather than letting a metadata change take the table down.
  2. Deploy the write path before the backfill, so new inserts populate business_date from the cutoff rule while reads stay on the old expression. The backfill then chases a closed set with a fixed upper bound instead of a moving target.
  3. Backfill in bounded batches keyed by entry_id range, on the order of 50,000 rows per statement, committing between batches and recording the high-water mark in its own table so a killed run resumes instead of restarting. WHERE business_date IS NULL makes each batch idempotent, and the pacing is set by replica lag and dead-tuple growth rather than CPU, since each UPDATE writes a new row version and the WAL volume is proportional to the rows touched.
  4. Install the constraint without a blocking scan: ALTER TABLE ... ADD CONSTRAINT ck_business_date CHECK (business_date IS NOT NULL) NOT VALID takes a brief ACCESS EXCLUSIVE and scans nothing, then VALIDATE CONSTRAINT takes SHARE UPDATE EXCLUSIVE and runs alongside reads and writes. On PostgreSQL 12 and later, SET NOT NULL can then use the validated CHECK and skip its own full scan; on 11 and earlier it always scans, so the CHECK is the migration on those versions.
  5. Build the index with CREATE INDEX CONCURRENTLY, which avoids ACCESS EXCLUSIVE at the cost of two table passes, cannot run inside a transaction block, and on failure leaves an INVALID index that must be dropped and rebuilt rather than reused.
  6. Cut over behind a flag: run the new and old expressions side by side for one reporting cycle and compare totals per day, since the cutoff rule will legitimately move entries near 17:00 across the boundary. Only once they reconcile do you retire the posted_at::date expression index, and you keep posted_at as the ordering key rather than repurposing it.

Worked solution (40 min)

  1. Write the migration as numbered SQL statements, annotating each with its lock mode and expected duration, and set lock_timeout plus a retry around every ALTER.
  2. Rehearse on a 10-million-row copy under a concurrent insert load, measuring per-batch duration, WAL generated and replica lag.
  3. Kill the backfill at a random point, restart it, and confirm it resumes from the high-water mark.
  4. Add the CHECK as NOT VALID, VALIDATE it while inserts continue, then SET NOT NULL and build the index CONCURRENTLY.
  5. Run old and new date expressions side by side for one cycle and diff the daily totals before dropping the old expression index.

Expected result: A migration in which no statement holds ACCESS EXCLUSIVE for more than a few milliseconds, every batch is bounded and resumable, VALIDATE and CREATE INDEX CONCURRENTLY run while writes continue, and the only rows whose daily bucket changes are those between 17:00 local and midnight UTC on their posting day.

Verify it yourself

  • SELECT count(*) FROM ledger_entry WHERE business_date IS NULL reaches exactly zero before the constraint is validated.
  • During the backfill, an INSERT from another session returns in single-digit milliseconds and no session waits on a relation lock longer than lock_timeout.
  • After a kill and restart, the count of rows updated equals the row count exactly once, with no batch re-applied and none skipped.
  • pg_index.indisvalid is true for the new index; an INVALID index means the CONCURRENTLY build failed and must be dropped, not left in place.

Follow-up

  • Reporting now wants ledger_entry partitioned by business_date. Why can this not be another ALTER, and what is the migration instead?
  • Nightly totals move for the days around the cutoff change. How do you tell a correct restatement from a backfill bug?
  • The backfill is halfway done when a replica falls 20 minutes behind. What do you throttle, and what do you refuse to throttle?

Describe the challenges of supporting a large-scale enterprise environment.

Medium

Approach

  1. Say what you would check first and why it is the highest-information step.
  2. State your assumptions explicitly before working the problem.
  3. Clarify what is being asked and what a complete answer contains.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Explain the difference between Authentication and Authorization.

Medium

Approach

  1. Work from the requirement backwards to the design.
  2. Say what you would check first and why it is the highest-information step.
  3. State your assumptions explicitly before working the problem.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

Specify the idempotency key contract for payment creation

Medium · idempotency · concurrency · api contract

POST /payments takes an idempotency key header plus amount_minor, currency, merchant and instrument token. The caller is a merchant server that retries on timeout and may retry concurrently from two workers. Storage is idempotency_key (id, scope text, key text with UNIQUE (scope, key), request_fingerprint bytea, status enum in_progress/completed/failed, response_status smallint, response_body jsonb, locked_at, completed_at, expires_at). Specify the scope, what the fingerprint covers, the response when the same key arrives with a different body, the response while the first request is still in flight, and what a replay after expires_at does.

Approach

  1. Scope the key to caller plus operation, for example 'merchant:1234:create_payment'. A global key space lets two merchants collide, and a key scoped only to the merchant lets a key used on one endpoint replay that endpoint's stored response for a different operation.
  2. Fingerprint the canonicalised body: sorted keys, no insignificant whitespace, amounts as integer minor units. Cover the fields that determine the effect and deliberately exclude client metadata such as a trace id, so a retry that adds one is still recognised as a retry. Same key with a different fingerprint is 409 with its own code, never the cached response and never the new effect.
  3. Win the concurrent race in the database: INSERT ... ON CONFLICT DO NOTHING RETURNING id. An empty return means this request lost, so it reads the winner's row. A SELECT-then-INSERT is not a fix; under READ COMMITTED both workers take fresh snapshots and both pass the check.
  4. Define in_progress explicitly. The loser reads status in_progress and receives 409 with a retry hint rather than a blocking wait or a second attempt. Note that Retry-After is defined for 429, 503 and 3xx, so on a 409 it is an extension you document rather than something the caller's HTTP stack already honours. A locked_at older than the lease is reclaimable only through a conditional UPDATE that matches the old locked_at, otherwise two reclaimers both proceed.
  5. Commit the key row before calling the processor, so a crash leaves evidence of an attempt that can be queried and converged. On completion store response_status and response_body and replay them verbatim, including the original status code.
  6. State the retention window (24 hours is a common choice) and its consequence: after expires_at the same key is a new request, so the caller's total retry deadline must sit inside the window or the mechanism silently stops protecting anything. The header name comes from an IETF draft rather than a ratified RFC, so document it as your convention.

Worked solution (30 min)

  1. Implement the endpoint over PostgreSQL with UNIQUE (scope, key), ON CONFLICT DO NOTHING RETURNING, an in_progress row written and committed before the processor call, and a stored response replayed on completion.
  2. Fire 50 concurrent identical requests with one key and assert one payment row, one processor call, and 50 byte-identical response bodies and status codes.
  3. Re-send the same key with amount_minor changed and assert 409 with the fingerprint-mismatch code and no new payment row.
  4. Kill the process between the processor call and the local commit, restart, and let the sweeper query the processor by the stored reference; assert the key converges to completed with exactly one charge.
  5. Advance the clock past expires_at, replay the key, and assert the documented behaviour (a new payment) rather than an undefined one.

Expected result: One payment row and one processor call for the 50 concurrent requests, identical replayed responses including status code, 409 with a distinct code for the changed body, and convergence to a single charge after the injected crash.

Verify it yourself

  • An instrumented processor client records exactly one outbound call across the 50 concurrent requests.
  • The losing inserts never execute the effect; they only read the winner's stored response.
  • The post-expiry replay behaviour is asserted by a test rather than left to whatever the code happens to do.

Follow-up

  • The processor call succeeds but your commit fails. What does the next request with the same key see, and what converges the row?
  • A merchant reuses one key for two genuinely different payments a month apart. Which of your rules catches it, and which lets it through?
  • Where does the key live end to end, and what changes if the downstream processor honours its own key?

Approval rate collapses when one feature-store shard fails

Medium · cache stampede · partial outage · bulkhead · retry amplification

The risk decision service answers about 3,000 authorisations per second within an 80 ms p99, reading features from an eight-shard store behind a local cache with a fixed 60-second TTL, a 2-second client timeout and two retries on timeout. One shard becomes unreachable at 14:02. Approval rate falls from 96% to 71% within 90 seconds, and the seven healthy shards show p99 rising to 900 ms with normal error rates. Diagnose in order, name the cause of the approval drop, and state the degraded behaviour this service should have had.

Approach

  1. Establish what declined actually means here before touching the cache. Split the rate into approve, decline, refer and timeout. If decline counts are flat and timeouts account for the whole 25-point drop, the model is not declining anything; it is failing to answer, and something downstream is reading silence as a decline. That reading is the first finding and the most expensive one.
  2. Break the store's metrics out per shard. One shard at 100% errors and seven at elevated latency with baseline error rates is the signature of queueing, not of seven shards failing. Healthy dependencies with healthy error rates and bad latency are almost always victims of a shared resource.
  3. Follow the resource. A single connection or thread pool shared across shards means requests to the dead shard occupy slots for the full 2-second client timeout, which is 25 times the whole p99 budget, and with two retries each such request can hold a slot for 6 seconds. Confirm by comparing pool acquisition wait against per-shard service time: if wait dominates, the pool is the bottleneck.
  4. Quantify the amplification. Two retries on timeout offer 3x load, and an unjittered 60-second TTL expires a warm set together, so misses arrive as a synchronised wave rather than spread across the window. Retrying a timeout is only correct when the failure is likely transient; against a hard-down shard it is pure amplification and needs a circuit breaker to gate it.
  5. Fix the three separate defects separately: per-shard pools so one shard cannot starve seven, a breaker that stops retrying a shard that is failing, and per-key TTL jitter with single-flight refresh so N concurrent misses for one key produce one upstream read.
  6. Write down the degraded decision as a product decision, not a timeout handler's accident. With features unavailable the service must return a stated answer, such as deterministic rules alone below an amount threshold and refer above it, and must record that the decision was degraded so it is auditable and reproducible from stored inputs.

Follow-up

  • Approving on deterministic rules alone during a feature outage carries a fraud cost. How do you size the amount threshold, and who signs it off?
  • What should the orchestrator do with a risk timeout, given it cannot distinguish one from a decline?
  • Does the circuit breaker fix the healthy shards' latency, or is the bulkhead doing that work? Which would you ship first if you could only ship one?

Suggested practice checklist

For someone who has spent the last few years shipping features and reading other people's code, and who has not solved a timed problem from a blank file in a long time. Five days rebuild the primitives and the patterns that sit on them, working from invariants rather than remembered solutions, and the last two attach that back to the rest of the loop.

Day 1 — Rebuild the primitives by implementing them

  • Implement a dynamic array with doubling growth and an operation counter, then change the growth rule to add a fixed sixteen slots instead, and time both for n of ten thousand, a hundred thousand and a million. The fixed-increment version resizes n/16 times at O(n) each, so its total work is quadratic; doubling is what makes append amortised constant.
  • Implement a hash map with separate chaining and a load-factor resize, then insert ten thousand keys engineered to land in one bucket and record what happens to lookup time, so that average-case O(1) becomes a claim with a stated precondition rather than a reflex.
  • For dynamic-array append and hash-map insert, write down which cost is amortised rather than worst-case, which single operation pays the whole bill, and what a system with a hard per-operation deadline would have to do instead.

Deliverable: Two working implementations plus a timing table showing the input at which each structure's advertised complexity stops holding.

Day 2 — Arrays under an invariant: two pointers, sliding window, binary search

  • Solve longest-subarray-with-sum-at-most-K using a sliding window, then run it on an input containing negative numbers and watch it return the wrong answer: extending the window only moves the sum monotonically when every element is non-negative, and that precondition is the whole reason the technique works.
  • Write the binary search that finds the first index satisfying a predicate rather than an exact value, put the loop invariant above the loop in a comment, and verify termination on the two inputs that break careless versions: the empty range, and a range where every element satisfies the predicate.
  • Compute the midpoint as lo + (hi - lo) / 2 and write one line on why the obvious (lo + hi) / 2 is a genuine defect in a fixed-width integer type and a non-issue in a language with arbitrary-precision integers.

Deliverable: Three solved problems, each with its invariant written above the loop, plus one recorded input on which the sliding window is provably wrong.

Day 3 — Sorting, heaps, and the greedy argument that has to be proved

  • Solve one top-k problem three ways, by full sort, by a size-k heap, and by quickselect, then write the values of n and k at which each becomes the right choice, along with quickselect's quadratic worst case and why a randomised pivot makes that unlikely rather than impossible.
  • Implement bottom-up heapify and count sift-down steps to confirm it does linear work rather than n log n, because most nodes sit near the bottom of the tree and therefore move only a short distance.
  • Take interval scheduling by earliest finishing time and write the exchange argument out in full: given any optimal schedule, swapping in the earliest-finishing interval keeps it feasible and no smaller. Then construct the weighted variant where that same greedy fails and name what has to replace it.

Deliverable: A three-way top-k comparison with measured crossover points, one written exchange argument, and one counterexample to a greedy rule that looks almost identical.

Day 4 — Recursion, memoisation, and the step to a table

  • Take one problem with overlapping subproblems, such as edit distance or coin change, instrument the plain recursion with a call counter to show the blow-up, then add memoisation and re-count.
  • Convert the memoised version to a bottom-up table and state the two properties you relied on: each subproblem's result depends only on its arguments, and the dependencies form a DAG you can enumerate in order.
  • Rewrite one deep recursion with an explicit stack, then find the input length at which the original hits the interpreter's frame limit, which defaults to about a thousand frames in CPython, so you know when the rewrite is required rather than decorative.

Deliverable: One problem in three forms, naive, memoised and tabulated, with call counts for each and the input length at which recursion depth becomes the binding constraint.

Day 5 — Graphs, where most of the work is choosing the traversal

  • Implement BFS and DFS over one adjacency list, then answer for each which finds a shortest path in an unweighted graph and which you would use to detect a cycle in a directed graph, including why the in-progress versus finished distinction matters for the second.
  • Implement topological sort by in-degree, feed it a graph containing a cycle, and confirm the failure signature is that fewer than V nodes come out rather than an exception, then note that the order it produces is one of several valid ones.
  • Run a shortest-path search on a graph with a single negative edge weight and show the wrong answer, then write the precondition Dijkstra actually needs, non-negative weights, because it finalises a node's distance the first time that node is popped, and name the algorithm you would switch to and its own limit.

Deliverable: A small graph library with BFS, DFS and topological sort, plus two inputs that produce documented wrong answers under the wrong algorithm choice.

Day 6 — One day for everything that is not an algorithm

  • Sketch one system only to the depth a coding-heavy loop tends to reach: the endpoints, what the service stores, and the single query pattern that decides the schema. Stop at twenty-five minutes.
  • Prepare the project answer for an interviewer who codes, which means rehearsing the two levels they push to: the specific thing you built, and why you chose that approach over the alternative they will name. Open with a number and be ready to say what it excludes.
  • Prepare the answer to what you would do differently, choosing a real technical mistake with a specific fix rather than a complaint about process or staffing.

Deliverable: One design sketch at endpoint-and-schema depth, plus a project answer rehearsed to two levels of follow-up.

Day 7 — Solve out loud, under time

  • Do three timed problems at twenty-five minutes each in a plain editor with no autocomplete and no execution until the end, then tally separately the failures that were syntax and the ones that were approach, because those two numbers call for different fixes.
  • Narrate one solution from the first sentence, stating the approach and its complexity before writing any code, and rehearse the sentence you will use when you realise mid-solution that the approach is wrong.
  • Re-solve from blank the two problems you were slowest on this week and compare the times against the day they first appeared.

Deliverable: A recording of one fully narrated solution and a tally that separates syntax failures from approach failures.

Outage answers are graded on instruments

Nobody is scoring your stamina at three in the morning. What carries weight is which signal told you something was wrong, what you measured before touching anything, what you rolled back versus what you fixed forward, and why you picked one. 'We restarted it and it went away' is a story about not knowing.

How do you resolve conflicts within a team setting?

Medium · behavioural and engineering judgement

Approach

  1. State the situation in two sentences and spend the rest on the reasoning.
  2. Give the blast radius: what could have broken, and what you measured.
  3. Close with what you would do differently, concretely.

Follow-up

  • What would you do differently if you ran that again?
  • What did you decide not to do, and why?

What makes you better suited for this role than the other candidates?

Medium · behavioural and engineering judgement

Approach

  1. Pick a story where you made the decision, not one where you watched it.
  2. Name the disagreement and how you resolved it with evidence.
  3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

  • What did you decide not to do, and why?
  • What would you do differently if you ran that again?

Tell us a little bit more about yourself that gives us a sense of your story.

Medium · behavioural and engineering judgement

Approach

  1. State the situation in two sentences and spend the rest on the reasoning.
  2. Give the blast radius: what could have broken, and what you measured.
  3. Close with what you would do differently, concretely.

Follow-up

  • What did you decide not to do, and why?
  • What would you do differently if you ran that again?