Interview guides · ATB Financial

ATB Financial Software Engineer Interview Questions & Guide 2026

API design practice for ATB Financial software engineer candidates: compatibility, versioning, and changes you cannot take back.

The 60-second brief

A Software Engineer at ATB Financial plays a pivotal role in evolving the digital landscape of one of Canada’s most prominent financial institutions. Your work directly impacts how customers interact with their finances, ranging from core banking systems to the experimental projects housed within innovation-focused divisions like ATB Ventures. You are expected to be more than just a coder; you are a problem solver who balances the stability required by a financial institution with the agility needed to deliver modern, user-centric software.

For a design discussion, a catalogue of architectures is worth less than the ability to turn a vague requirement into a data model and an API contract. A box diagram with no schema under it collapses at the first follow-up question.

Focus areas

Reconcile the ledger against processor settlement files
Model money movement as balanced double-entry postings
Store money as integer minor units

Overview

The role involves navigating complex technical environments, including cloud infrastructure, distributed systems, and legacy integration. Whether you are optimizing asynchronous transactions or designing scalable architecture using Docker and Kubernetes, your contributions are critical to maintaining trust and driving financial technology forward. You will often collaborate across multidisciplinary teams, bridging the gap between high-level business strategy and low-level technical execution.

The ATB Financial interview process

ATB Financial candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.

What to demonstrate

Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled

How to prepare

Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market

Technical Deep Dives

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Behavioral Evaluations

Your first answer is not really what is scored. It buys the follow-up questions, and those decide the round. An interviewer with fifteen minutes takes one thread and pushes on it four or five times, so a story you can only tell at a single level of detail collapses under the third why. That is an argument for fewer stories known deeply rather than one prepared per prompt. Four or five pieces of work you can still explain down to the code you changed and the argument you had about it will cover nearly anything asked in this round.

What to demonstrate

Whether a story holds as the questioning moves from what you did to why that instead of the alternative, and then to what you would change knowing what you know now
Whether you can re-cut a project to answer the question actually asked rather than delivering a rehearsed block that answers an adjacent one
Whether your level of detail is chosen rather than habitual: going down to the schema when the question is about the data model, staying out of it when the question is about the person who disagreed with you

How to prepare

Pick four projects and write the chain out four levels deep for each: what you did, why that, why not the alternative, and what would have to be true for the alternative to have won. Where you cannot reach the fourth level, you have a placeholder rather than a story
Have someone ask why three times in a row on a single thread with nothing else added, and mark the point where you start repeating a sentence you already said. That point is where the interviewer stops learning anything
Build a one-page index instead of an answer bank: the common prompts in this round (disagreement, a failure that was yours, thin requirements, a deadline you missed, work you inherited) mapped to which of your four projects you would use for each, so the choosing is done now rather than while an interviewer waits

Where candidates lose points

Assuming the default isolation level enforces the invariant you wrote down

PostgreSQL defaults to READ COMMITTED, where every statement takes a fresh snapshot, so a read-modify-write on a balance loses updates under concurrency. Its REPEATABLE READ is snapshot isolation, which blocks that particular anomaly by aborting the loser with SQLSTATE 40001 but still permits write skew across two different rows; only SERIALIZABLE closes that, and both levels therefore require a bounded retry loop on 40001 that many implementations simply never write. MySQL's InnoDB REPEATABLE READ behaves differently again — it does not abort on a conflicting write, so the identical application code silently changes behaviour when the engine changes. Two-sided transfers add a second failure mode on top: without a deterministic lock ordering, such as always locking account ids in ascending order, concurrent opposing transfers deadlock (SQLSTATE 40P01).

Deriving the business date from the UTC timestamp

Posting date, value date and the processor's settlement date are three different dates, determined by cutoff times, business-day calendars and holidays rather than by midnight UTC. A movement recorded at 23:50 on one side of a cutoff belongs to the next business date, so computing business_date as created_at::date makes daily totals disagree with every statement and every settlement file. The signature is a reconciliation break that resolves itself the following day and then reopens, which reads like a flaky job and is actually a data model that is missing a column: business_date has to be stored explicitly and set from the cutoff rule, with the timestamptz kept separately for ordering.

Choosing a schema before the access patterns are known

Write the queries first, with their filters, sort orders, cardinalities and which ones sit on the latency-critical path, then design tables and indexes to serve them. An index nothing queries still costs write throughput and storage, and a hot query with no supporting index becomes a full scan that only hurts once the table is big.

Issuing one query per row of a result set

Fetch related rows in a single batched query keyed by the ids you already hold, or join them into the original query. A per-row round trip multiplies network latency by the row count, and it looks perfectly fine against the ten rows in your development database.

Questions & practice

Rank merchants by unmatched break value in one streamed pass

Medium · top-k · heap · streaming · space-bounded

A reconciliation run streams up to 50 million break rows of (merchant_id, currency, delta_minor signed, break_type). Return the 50 merchants with the largest total absolute delta_minor in a single currency, from one pass, with memory that does not grow with merchant count — there are up to 8 million distinct merchants and room for roughly 200,000 counters. Give both an exact and an approximate design, state the error bound the approximate one actually guarantees, and say which you would put in the nightly job.

Approach

Start by testing whether the constraint is real. Exact needs one hash map merchant_id -> int64 plus a size-50 min-heap: O(n) to aggregate, O(m log 50) to rank, O(m) space. At 8 million merchants and 16 bytes of payload that is a few hundred megabytes — quote the number before reaching for a sketch.
If it genuinely does not fit, exact still costs only two passes or an external group-by: partition by hash(merchant_id) % P to disk, aggregate each partition independently, then merge with a K-way heap. This is the answer whenever exactness is non-negotiable, which for money it usually is.
One-pass approximate: weighted Space-Saving with C counters. An item either hits an existing counter or evicts the minimum one and inherits its value as an over-estimate. With total weight W and C counters, every reported total over-estimates by at most W/C, and any merchant whose true total exceeds W/C is guaranteed to be in the table. Quote W/200,000 as a fraction of the day's break value.
State what the bound does not give you: the ordering within the top 50 is not guaranteed, and a merchant just below the threshold can be missed entirely. The honest claim to an operations reader is 'contains everyone above 0.0005% of today's break value', not 'the top 50'.
Decide the metric before either design, because sum(abs(delta)) and abs(sum(delta)) are different questions: a merchant with offsetting +1,000,000 and -1,000,000 breaks is top-ranked under the first and invisible under the second. Break work wants the first.
Recommend exact (two-pass or external group-by) for the nightly job — it runs once, has hours of budget, and an analyst acts on its output. Keep the sketch for a live dashboard, where a cheap bounded approximation beats an exact number that is four hours stale.

Worked solution (25 min)

Fix the metric in writing as sum(abs(delta_minor)) per (merchant_id, currency), because that is what an analyst works from.
Implement exact: hash-map aggregate, then a size-50 min-heap — push while the heap is under 50, then push-pop only when an item beats the root.
Implement weighted Space-Saving with 200,000 counters over the same stream.
Generate 50 million rows with a Zipfian merchant distribution near s = 1.1 so a few merchants dominate, then generate a uniform variant for contrast.
Compare: how many of the exact top 50 the sketch recovers, and the largest observed over-estimate, against the W/C bound.

Expected result: On the Zipfian stream the sketch recovers all 50 and every reported total over-estimates by at most W/200,000. On the uniform stream it recovers materially fewer — which is the result that tells you the bound, not the benchmark, is what you may promise a reader.

Verify it yourself

Exact top-50 via the heap equals exact top-50 via a full sort on a 1-million-row fixture.
Every sketch total is greater than or equal to the true total and never less; an under-estimate means the counter-inheritance step is wrong.
The largest observed over-estimate is at or below W/C on every run.
Heap memory stays at 50 entries regardless of input size.

Follow-up

Totals are now needed per currency as well as overall. What does that do to the key and to the counter budget?
Prove the Space-Saving over-estimate bound to me in two sentences.
The nightly job restarts halfway. Is the aggregate idempotent, and what does the heap do with a partially consumed stream?

Match a settlement file to ledger postings under duplicate keys

Medium · hash join · reconciliation · external memory

You have one business date of ledger postings (about 12 million rows: transaction_id, source_id, amount_minor, currency, business_date) and the processor's settlement file (about 12 million lines: settlement_line_id, external_reference, amount_minor, currency, business_date), where source_id carries the external reference. Match one-to-one on (external_reference, amount_minor, currency, business_date). Duplicate keys occur legitimately — the same amount can appear twice. Emit every unmatched item classified ledger_only, file_only or duplicate_match, in linear expected time. Then say what you do when neither side fits in memory.

Approach

Build the smaller side into key -> deque of row ids, never key -> row id. A duplicate key is data, not corruption; a single-row map drops one of a legitimate pair and the break report then shows a file_only that does not exist.
Probe the larger side once, carrying one extra bit per bucket: whether that bucket was ever hit. Pop from the bucket on a match and set the bit. An absent key is a probe-side-only row. A present-but-empty bucket means the probe side holds more copies than the build side — surplus, so duplicate_match. After the pass, a leftover non-empty bucket that was never hit is build-side-only; one that was hit is build-side surplus, so also duplicate_match.
That bit is what makes the classification a function of the per-key counts rather than of which side you happened to build. For a key with L ledger and F file copies: min(L, F) match, and the |L - F| surplus rows are duplicate_match tagged with the side that is over, degenerating to ledger_only or file_only exactly when min(L, F) is 0. Without the bit, surplus is only observable as a present-but-empty bucket, which can only ever happen on the probe side — so the same input reports different break classes depending on build order, and the smaller-side heuristic in bullet one silently decides which.
A genuine amount difference does not surface as amount_mismatch here, because the amount is inside the key — it surfaces as a ledger_only and a file_only sharing a reference. Promote those in a second, separate pass keyed on reference alone, recording signed delta_minor as ledger minus file. Keep that promotion out of the exact pass.
Cost: O(N+M) expected time and O(min(N,M)) memory; the hit bit packs into the bucket header and changes neither bound. The constant is the hash map, roughly 60 to 100 bytes per entry in most runtimes, so 12 million rows is order 1 GB — measure it rather than assert it.
When neither side fits, use a grace hash join: partition both sides with the same hash function into P spill files so a key lands in the same partition on both sides, then join partition by partition in memory. Cost is two extra sequential passes; skew inside one partition is the failure mode, handled by re-partitioning that partition under a second hash.
Sort-merge is the alternative at O(N log N + M log M) with external sort, and it wins when the file already arrives sorted by reference or the output must be ordered. It also gets the surplus classification for free, since a merge sees L and F side by side. Whichever you pick, do not widen the amount comparison to make breaks disappear: a tolerance wide enough to absorb rounding is wide enough to absorb a real loss.

Follow-up

The file nets three fee lines into one batch total. Which pass catches that, and what is its stopping rule?
The processor's business date sits one cutoff behind yours for forty minutes of traffic. What does that do to the exact join, and what does it do to break ages?
The same break recurs on the next run. Why must it link to the existing reconciliation_break row rather than open a second one?

Answer as-of balance queries over an append-only entry log

Hard · prefix sums · offline queries · append-only

Given 400 million ledger_entry rows (entry_id, account_id, direction, amount_minor, currency, business_date) and 2 million queries of (account_id, currency, as_of_date) asking for the balance at the end of that business date, produce every answer. The obvious solution — per query, sum that account's entries with business_date <= as_of_date — is correct. Say precisely why it will not finish, then give one that will, with time and space complexity. Corrections are posted as new entries carrying their own business_date.

Approach

Cost the naive version in numbers before rejecting it. Spread uniformly over 20 million accounts, each query touches about 20 rows behind a per-account index and 2 million queries is 4e7 row touches — perfectly fine. The problem is skew: one pooled clearing or merchant settlement account holding 3e7 entries, taking 10% of the queries, is 6e12 row touches. Name the skew; 'n is large' is not the reason.
The structural fact that buys a cheap answer: entries are append-only and never updated, so a prefix sum over an account's entries ordered by (business_date, entry_id) is stable — nothing behind position i can change. No mutable-balance design offers that, and it is why the storage is worth paying for.
Offline sweep, when all queries are known up front: externally sort entries by (account_id, currency, business_date, entry_id) and queries by (account_id, currency, as_of_date), then merge-walk both with a running sum, emitting each query's answer as the sweep passes its date. O((n + q) log(n + q)) dominated by the sort, O(1) beyond sort buffers, one sequential pass over each input instead of 2 million random seeks.
Online alternative: materialise end-of-day snapshots — one row per (account_id, currency, business_date) that had activity, holding the cumulative total. A query becomes one index seek for the latest snapshot at or before as_of_date, O(log n) per query, over far fewer rows than n. Use snapshots when queries arrive singly and the sweep when they arrive as a batch.
Corrections are the subtlety: an entry posted today but dated back changes historical answers, so every snapshot for that account from that date forward is stale. Either keep a Fenwick tree over dates per account (O(log D) update and prefix query) or recompute that account's snapshots from the corrected date onward. Then be precise about what reproducibility means — yesterday's statement is reproducible as of a stated snapshot time, not identical forever.
Bound the resources: int64 sums throughout, no float; 400 million rows at roughly 48 bytes of the columns you actually need is about 19 GB, so the sort is external and its fan-out is chosen from the sort buffer, not from the row count.

Follow-up

One account holds 30% of all entries. What does the external sort do with it, and what would you do for that one key instead?
Queries now arrive online at 500 per second. Which design survives, and what does keeping the other one warm cost?
A correction lands with a business_date 90 days back. Which snapshots are now wrong, and how does a reader find out?

Decide whether balance is derived or materialised, then hold the floor

Hard · denormalisation · isolation levels · lock contention

Authorisation needs an account's available balance inside an 80 ms budget at 3,000 requests per second; that account already has 200 million ledger_entry rows. A product rule says the balance may never fall below the account's negative overdraft limit. Decide whether the balance is summed from entries or held in a materialised account_balance(account_id, balance_minor, floor_minor, currency, version) row, and justify the choice from the read pattern. Then give the write path that holds the floor, naming the isolation level, the anomaly a weaker level permits, and the SQLSTATE you retry.

Approach

Size the derived read before arguing about it: summing 200 million rows is not an 80 ms operation under any index, since even a covering index on (account_id, entry_id) still reads work proportional to the rows. The authorisation read pattern forces one materialised row fetched by primary key; the statement read pattern, which is low-rate and historical, stays derived from entries. That is the whole justification for the denormalisation, and its price is a write on that row per posting.
Put the floor where it can be a single-row constraint: CHECK (balance_minor >= floor_minor) on account_balance, updated in the same transaction as the entries. As a predicate over a set of entry rows it cannot be a CHECK at all, which is the reason the materialised row earns its keep twice.
Write the mutation as one statement: UPDATE account_balance SET balance_minor = balance_minor - $1, version = version + 1 WHERE account_id = $2 AND balance_minor - $1 >= floor_minor. Zero rows updated means refused. This is safe even under READ COMMITTED, because the UPDATE re-evaluates its predicate against the locked, post-update version of the row.
Name the anomaly in the shape that is not safe: SELECT the balance, compute a new value in application code, then UPDATE to that constant. Every statement under READ COMMITTED takes a fresh snapshot, so two concurrent 60-unit withdrawals against 100 both read 100 and both write 40. The row then claims 40 while 120 has actually left, so the true position is -20, below a floor of 0, and the row it is checked against cannot show it. PostgreSQL REPEATABLE READ is snapshot isolation and aborts the loser with SQLSTATE 40001; SERIALIZABLE additionally closes write skew across two rows. Both require a bounded retry with backoff, and InnoDB REPEATABLE READ does not abort at all, so identical code changes behaviour on a different engine.
For a two-account transfer, acquire the rows in a deterministic order such as ascending account_id; without it, opposing concurrent transfers deadlock and the database kills one with SQLSTATE 40P01. That is a retry, not a correctness failure, but it is a retry somebody has to write.
Finish on the ceiling: the hot row admits one committed write per lock hold, so at a 2 ms hold it caps near 500 per second regardless of cores. Sharding into N sub-rows multiplies throughput and immediately makes the floor check cross-row again, which then needs the shard sum under SERIALIZABLE or a per-shard reserved allowance.

Follow-up

Shard the balance into eight sub-rows. Write exactly what the floor check now does, and what it costs per authorisation.
Someone proposes an AFTER INSERT trigger on ledger_entry to maintain the balance. What does that change about ordering, about batch posting, and about failure handling?
How do you detect that the materialised row has drifted from the entries, how often do you run it, and on which replica?

Rebuild a statement with a running balance from entries alone

Medium · window functions · frames · event tables

From ledger_entry(entry_id bigint and monotonic, account_id, direction, amount_minor, currency, business_date, posted_at) produce one month's statement for a single account: every entry in order, its signed amount, and the running balance after it, starting from a supplied opening balance. The account is a customer deposit, so a credit increases it. Also return the first business_date in the month on which the running balance went negative, or null if it never did. Write the query using window functions, state which frame you depend on, and say what makes the ordering deterministic.

Approach

Sign the amount first: CASE WHEN direction = 'credit' THEN amount_minor ELSE -amount_minor END, because the sign lives in direction and never in the column. State the convention out loud, since a customer deposit is a liability of the institution and a credit increases it, while for an asset account the same expression inverts.
Compute the running balance as SUM(signed) OVER (PARTITION BY account_id ORDER BY business_date, entry_id ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW), then add the opening balance as a scalar. The opening figure is a constant for the statement, not a second window.
Be explicit about the frame, because it is the whole exercise: with an ORDER BY and no frame clause the default is RANGE UNBOUNDED PRECEDING AND CURRENT ROW, which includes every peer row tied on the ordering key. Ordering on business_date alone therefore reports the same end-of-day figure on every line of that day, and the statement still looks plausible.
Make the order deterministic with a column that has no ties: business_date has ties by construction, entry_id is monotonic and unique, and posted_at is neither guaranteed unique nor correct as an ordering key when a backdated entry posts late.
Get the first breach from the same computation rather than a second pass: wrap the running balance in a subquery and take MIN(business_date) FILTER (WHERE running_balance_minor < 0). LAG is the wrong tool here, because the question is about a level rather than a change.
Support it with an index on (account_id, business_date, entry_id) so the partition and the order both come from the index; then confirm in EXPLAIN that there is no Sort node above the index scan.

Worked solution (25 min)

Seed one account with 40 entries across a month, including three on the same business_date with mixed directions and one backdated entry posted late.
Write the signed-amount expression and the SUM window with an explicit ROWS frame ordered by (business_date, entry_id).
Wrap it and add MIN(business_date) FILTER (WHERE running_balance_minor < 0) as first_negative_date.
Re-run with the frame clause removed and with entry_id dropped from the ORDER BY, and record exactly which rows change.

Expected result: One row per entry in date then entry_id order. The final row's running balance equals the opening balance plus the sum of all signed amounts in the month, to the minor unit. first_negative_date is the earliest business_date at which the ordered running balance dipped below zero, and null when it never does.

Verify it yourself

Sum the signed column independently with a plain aggregate and compare it against the last running balance; they must be equal.
The three entries sharing a business_date show three different running balances, which proves the frame and tiebreak are right.
Removing the ROWS frame reproduces the end-of-day artefact on exactly those three rows and nowhere else.
EXPLAIN shows no Sort node above the index scan on (account_id, business_date, entry_id).

Follow-up

An entry for 3 March posts on 7 March, after the statement for that week was sent. Where does it appear, and what does the running balance do?
The same statement has to be reproducible a year from now. What stops it changing, and what would silently change it?
At what account size does deriving this per request stop being viable, and what would you materialise first, a daily closing balance or a monthly one?

What are the primary differences and use cases for REST versus GraphQL?

Medium

Approach

Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer contains.
State your assumptions explicitly before working the problem.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Can you walk me through a complex troubleshooting scenario in an IT infrastructure environment?

Medium

Approach

Work from the requirement backwards to the design.
State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Capture endpoint that survives concurrent duplicate retries

Medium · idempotency · concurrency · unique constraints · retries

POST /payments/{intent_id}/capture carries an Idempotency-Key header and an amount. Callers retry on timeout, and two retries can land concurrently on different instances. You have idempotency_key(id, scope, key, UNIQUE(scope,key), request_fingerprint bytea, status in_progress|completed|failed, response_status, response_body jsonb, locked_at, completed_at, expires_at). The processor capture takes 200 to 2,000 ms. Design the path so exactly one capture reaches the processor, every duplicate receives the identical response, and a crash between the processor call and your commit converges. Name the single statement that is the concurrency control.

Approach

The concurrency control is INSERT INTO idempotency_key (...) VALUES (...) ON CONFLICT (scope, key) DO NOTHING RETURNING id, and it must commit before the processor call. A returned row means you own the effect; zero rows means you lost and must read the winner's result. A SELECT-then-INSERT check cannot substitute: two requests both read 'absent' and both proceed, and the window is exactly the concurrency you are defending against.
Commit the in_progress row in its own short transaction. A concurrent duplicate insert blocks on an in-flight conflicting insert until that transaction ends, so wrapping the 2 s processor call in the same transaction turns every duplicate into a 2 s lock wait and a retry storm into pool exhaustion.
Define loser behaviour per status rather than uniformly: completed replays response_status and response_body unchanged; in_progress returns 409 with Retry-After and performs nothing; failed splits by cause, since a terminal processor decline should replay but a transport failure should let the key be retried. Getting this wrong in the safe direction (replay a decline) is better than returning an error for a capture that succeeded.
Fingerprint the canonicalised body with SHA-256 and compare on every hit. Same key with a different amount is a client bug and must return 409 or 422, never the cached response, because returning the cached body silently captures the old amount and looks successful.
Pass the same key to the processor so deduplication holds end to end, and generate it once at the originating caller. A key regenerated per attempt leaves every line of idempotency code in place while disabling the mechanism entirely.
Converge after a crash by querying the authoritative side rather than guessing: a reaper picks up rows in_progress past locked_at plus a bound, asks the processor for that key or the intent's processor_reference, and completes the row from the answer. Set expires_at longer than the caller's full retry schedule and document that a replay after expiry is a new request.

Worked solution (30 min)

Implement the endpoint with the ON CONFLICT DO NOTHING insert committed before the processor call, and a processor stub that counts calls and sleeps 1,500 ms.
Drive 50 concurrent identical requests through two application instances.
Repeat with the same key and the amount changed by one minor unit.
Kill the instance between the stub's response and the local commit, restart, and run the reaper.

Expected result: Exactly one processor call across the 50 requests. One request returns the capture; the other 49 either replay the stored body unchanged or return 409 in_progress, and none returns a 500 or a second capture. The changed-amount request returns 409 or 422 and never the cached body. After the crash, the reaper completes the row from the processor's own answer, leaving exactly one capture.

Verify it yourself

Time the 49 losers: if they each take about 1,500 ms, the winner is holding the key row inside the transaction wrapping the processor call.
The stub's call counter reads 1, not 1 plus a retry.
Swapping the constraint for a check-then-insert reproduces two captures under the same load; keep that as the regression test rather than trusting the fix.

Follow-up

The processor does not honour idempotency keys. What is the end-to-end design now, and what can you no longer promise?
Two merchants send the same key value. What makes that safe?
You keep keys for 24 hours at 3,000 requests/s. Size the table and the index, and say what expires them.

Approval rate collapses when one feature-store shard fails

Medium · cache stampede · partial outage · bulkhead · retry amplification

The risk decision service answers about 3,000 authorisations per second within an 80 ms p99, reading features from an eight-shard store behind a local cache with a fixed 60-second TTL, a 2-second client timeout and two retries on timeout. One shard becomes unreachable at 14:02. Approval rate falls from 96% to 71% within 90 seconds, and the seven healthy shards show p99 rising to 900 ms with normal error rates. Diagnose in order, name the cause of the approval drop, and state the degraded behaviour this service should have had.

Approach

Establish what declined actually means here before touching the cache. Split the rate into approve, decline, refer and timeout. If decline counts are flat and timeouts account for the whole 25-point drop, the model is not declining anything; it is failing to answer, and something downstream is reading silence as a decline. That reading is the first finding and the most expensive one.
Break the store's metrics out per shard. One shard at 100% errors and seven at elevated latency with baseline error rates is the signature of queueing, not of seven shards failing. Healthy dependencies with healthy error rates and bad latency are almost always victims of a shared resource.
Follow the resource. A single connection or thread pool shared across shards means requests to the dead shard occupy slots for the full 2-second client timeout, which is 25 times the whole p99 budget, and with two retries each such request can hold a slot for 6 seconds. Confirm by comparing pool acquisition wait against per-shard service time: if wait dominates, the pool is the bottleneck.
Quantify the amplification. Two retries on timeout offer 3x load, and an unjittered 60-second TTL expires a warm set together, so misses arrive as a synchronised wave rather than spread across the window. Retrying a timeout is only correct when the failure is likely transient; against a hard-down shard it is pure amplification and needs a circuit breaker to gate it.
Fix the three separate defects separately: per-shard pools so one shard cannot starve seven, a breaker that stops retrying a shard that is failing, and per-key TTL jitter with single-flight refresh so N concurrent misses for one key produce one upstream read.
Write down the degraded decision as a product decision, not a timeout handler's accident. With features unavailable the service must return a stated answer, such as deterministic rules alone below an amount threshold and refer above it, and must record that the decision was degraded so it is auditable and reproducible from stored inputs.

Follow-up

Approving on deterministic rules alone during a feature outage carries a fraud cost. How do you size the amount threshold, and who signs it off?
What should the orchestrator do with a risk timeout, given it cannot distinguish one from a decline?
Does the circuit breaker fix the healthy shards' latency, or is the bulkhead doing that work? Which would you ship first if you could only ship one?

Suggested practice checklist

Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.

Day 1 — Coding, one pass at shallow depth

Solve one problem from each of six families, an array with two pointers, hash counting, binary search, a tree traversal, a graph traversal and one dynamic program, under a hard twenty-minute cap with no extensions, marking each finished, late, or stalled.
For every stall, write the exact move you could not make rather than the subject, so the note reads could not turn the recurrence into a loop rather than bad at dynamic programming.
Fix nothing today. The value of the pass is the unfixed record.

Deliverable: Six timed attempts marked finished, late or stalled, each stall carrying a named blocking move.

Day 2 — Design, one pass at shallow depth

Spend twenty minutes each on three different shapes, a read-heavy feed, a write-heavy ingest path, and something needing a transaction across two entities, stopping each at requirements, interface and data model.
After each, write the first question you could not answer, which is usually a number you could not estimate or a failure mode you had no vocabulary for.
Mark which of the three you would be most relieved not to be asked, and treat that as data rather than as a preference.

Deliverable: Three shallow designs, each with the first unanswerable question written at the bottom.

Day 3 — Fundamentals and the practical rounds

Answer eight short questions in writing at four minutes each, covering the material that fills the gaps between the big rounds: what happens between a URL and a rendered page, what an index costs on write, when a process is preferable to a thread, and what conditions a deadlock requires.
Do one thirty-minute practical task of the kind a take-home compresses: read an unfamiliar two-hundred-line file and write what it does, what you would change, and the one thing you remain unsure of.
Score every answer fluent, correct but slow, or absent, and keep the absent ones visible.

Deliverable: Eight scored short answers and one written reading of unfamiliar code.

Day 4 — The rounds that are about you, and the map

Deliver three behavioural answers aloud against a timer, a conflict, a failure you owned, and a decision made without enough information, marking any that ran past three minutes or contained no number.
Assemble the map: every marked item from days one to three on a single page, sorted by how likely it is to appear in your loop rather than by how uncomfortable it felt.
Choose exactly two areas for the remaining three days and write down what you are deliberately abandoning.

Deliverable: A one-page scored map of the whole surface area with two areas chosen and the rest explicitly abandoned.

Day 5 — First chosen area, to the depth you skipped

Work the higher-ranked area in four focused blocks, choosing items one level above where you stalled rather than repeating what already works.
After each block write the rule you extracted in one sentence with its precondition attached, since a rule carrying no precondition is exactly what fails under a variation.
Re-attempt the day-one or day-two item that exposed this area and compare against the original timing.

Deliverable: Four worked blocks, a timed re-attempt against the original, and three one-sentence rules with preconditions.

Day 6 — Second chosen area, where the gap is coverage rather than speed

Treat the second area differently from the first. Day five drilled something you could already half-do; this one is usually a topic you had simply never met, so build one worked reference example end to end and keep it, rather than attempting six problems badly.
Write down the vocabulary you were missing on day two or three, five terms at most, each with the one sentence that makes it usable in an answer rather than the textbook definition.
Redo the shallow attempt that exposed this area and note whether you now fail later in the problem, because moving the failure point is the realistic gain from a single day and is worth more than a score that did not change.

Deliverable: One worked reference example for the newly covered area, a five-term vocabulary list, and a note on where the failure point moved.

Day 7 — Reassemble the loop

Sit two rounds back to back with no gap, ordering them so the area you chose second comes last, because the map was built from rested, isolated attempts and the loop will reach your weaker area when you are already spent.
Write where the second round suffered from the first, which is normally the point at which structure collapses into narration.
Reduce the week to one page holding only the rules you can state without reading them.

Deliverable: Mock notes on cross-round carryover plus a one-page card of rules you can recite from memory.

The bug you decided not to fix

Engineers over-index on what they repaired. A stronger answer covers something you knowingly left broken: the alert you tuned down, the data inconsistency you documented instead of chasing, the cleanup you deferred past two quarters. Give the reasoning and the condition that would have reopened it, so it reads as a decision and not as neglect.

How do you handle asynchronous transactions in Apex?

Medium · behavioural and engineering judgement

Approach

Give the blast radius: what could have broken, and what you measured.
State the situation in two sentences and spend the rest on the reasoning.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?

How do you manage your work under pressure or during high-stress periods?

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
Pick a story where you made the decision, not one where you watched it.
Give the blast radius: what could have broken, and what you measured.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?

Can you tell me a little about yourself and your professional journey?

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
Give the blast radius: what could have broken, and what you measured.
Name the disagreement and how you resolved it with evidence.

Follow-up

What did you decide not to do, and why?
How did you know your change caused the improvement?

What is the last book you read?

Medium · behavioural and engineering judgement

Approach

State the situation in two sentences and spend the rest on the reasoning.
Pick a story where you made the decision, not one where you watched it.
Give the blast radius: what could have broken, and what you measured.

Follow-up

What did you decide not to do, and why?
What would you do differently if you ran that again?