Interview guides · Atlantic Bay Mortgage Group

Atlantic Bay Mortgage Group Software Engineer Interview Questions & Guide 2026

Testing practice for Atlantic Bay Mortgage Group software engineer candidates: the boundaries worth covering, and why flakiness is a real defect.

The 60-second brief

As a Software Engineer at Atlantic Bay Mortgage Group, you play a vital role in building and maintaining the technological infrastructure that powers a fast-paced mortgage lending environment. Your work directly impacts how internal teams operate and how clients experience the home-buying process. You are expected to be a problem solver who can translate business needs into reliable, efficient software solutions.

For a design discussion, a catalogue of architectures is worth less than the ability to turn a vague requirement into a data model and an API contract. A box diagram with no schema under it collapses at the first follow-up question.

Focus areas

Make every money-moving endpoint idempotent by key
Store money as integer minor units
Consume webhooks duplicated, delayed and out of order

Overview

This position is both challenging and rewarding because it requires a balance of technical rigor and creative adaptability. You will not only be responsible for core engineering tasks, such as managing servers and virtualization, but also for contributing to a culture of innovation. Whether you are troubleshooting daily incidents or optimizing system architectures, your contributions are central to the operational success of the company.

The Atlantic Bay Mortgage Group interview process

Atlantic Bay Mortgage Group candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.

What to demonstrate

Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled

How to prepare

Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market

In-Person Meeting

An unlabelled round is first an information problem, and the cheapest information is free. Whoever schedules it can usually tell you how long it runs, who will be in the room and what they work on, whether you will be writing code and in what environment, and whether anything is being sent beforehand. Ask in writing so the answer is on record, then prepare for the two or three formats those answers still leave open instead of betting on one. What separates a strong candidate is not guessing right; it is having an opening that works whichever one it turns out to be.

What to demonstrate

Whether you can start work from an ambiguous brief, since tolerating a vague scope without stalling is the same thing the job asks for
Whether the questions you asked beforehand were ones that change your preparation, such as duration, medium and who is joining, rather than ones whose answers you could not have acted on
Whether you adapt when the round turns out to be something other than what you were told, instead of spending the first ten minutes visibly recalibrating

How to prepare

Send one short scheduling message asking four things: how long, who is joining and what they work on, whether you will be writing code and where, and whether to prepare anything in advance. Treat a vague reply as real information, since it means the round is loosely structured and you will be shaping it yourself.
Write one opening that works in any of the formats still open: restate in your own words what you have been asked to do, then ask which of two directions is more useful to them. Say it aloud until it stops sounding recited.
Set up for the two most likely formats before the call starts, with a blank editor in the language you would choose and a shared document you can type into, so a format surprise costs you nothing in the first minutes

Where candidates lose points

Treating money as a decimal with two places

ISO 4217 exponents are 0 for currencies such as JPY and KRW, 2 for most, and 3 for BHD, KWD, JOD, OMR and TND, so a hard-coded multiply-by-100 is off by a factor of 100 or 10 depending on the currency, in opposite directions. Floating point is worse: IEEE 754 binary64 cannot represent 0.1 exactly, so repeated accrual accumulates drift that appears as a handful of minor units in the daily reconciliation and then gets 'fixed' by widening the match tolerance, which is how a genuine break becomes invisible. The only forms that survive a reconciliation are integer minor units with the exponent carried alongside the currency code, or a fixed-scale decimal type with exactly one documented rounding point.

Deriving the business date from the UTC timestamp

Posting date, value date and the processor's settlement date are three different dates, determined by cutoff times, business-day calendars and holidays rather than by midnight UTC. A movement recorded at 23:50 on one side of a cutoff belongs to the next business date, so computing business_date as created_at::date makes daily totals disagree with every statement and every settlement file. The signature is a reconciliation break that resolves itself the following day and then reopens, which reads like a flaky job and is actually a data model that is missing a column: business_date has to be stored explicitly and set from the cutoff rule, with the timestamptz kept separately for ordering.

Answering a debugging question with a guess instead of a bisection

Give a procedure that halves the search space at each step: confirm the symptom reproduces, establish the last known-good version, input or timestamp, then bisect over commits, over the data, or over the layers of the request path. A plausible cause with no way to confirm it is the same move whether it happens to be right or wrong, which is why it scores nothing.

Retrying a write that is not safe to repeat

A timeout tells you nothing about whether the server applied the write, so a blind retry of a create or a charge can duplicate it. Either make the operation idempotent, with a caller-supplied key the server deduplicates on or a conditional update, or do not retry it; and use exponential backoff with jitter so the retries of many clients do not synchronise into a second outage.

Questions & practice

Detect duplicate-charge bursts in an out-of-order authorisation stream

Hard · sliding window · event time · watermarks · deduplication

Authorisations arrive as (instrument_token_id, amount_minor, currency, event_time, arrival_time) at roughly 3,000 per second, up to 60 seconds late and out of order. Flag any token with three or more authorisations of identical (amount_minor, currency) inside any 10-minute window of event time. Report each flag once, as early as correctness allows. State memory per key and in total, how late an event you will accept, and what you do with one that arrives after you have already reported — or already declined to report — that window.

Approach

Key state by (instrument_token_id, amount_minor, currency), not by token: the predicate is about identical amounts, so the window belongs to the triple. Each key holds its own event-time-ordered deque.
In-order, this is two pointers: on insert, pop from the front while front <= new - 10 min, then flag if the deque reaches length 3. Amortised O(1) per event, memory proportional to that key's window occupancy.
Out-of-order arrival breaks append-only monotonicity, so insert in position instead. With lateness bounded at 60 seconds the insertion point is always near the tail, so a short sorted vector or a 600-bucket per-second ring keeps it O(w) with tiny w; a balanced tree per key is correct but over-built for a one-minute reorder.
Drive decisions off a watermark of max(event_time seen) - 60 s, never off wall clock. Anything older than the watermark is too late to change an answer and is counted in a late_dropped metric. Without an explicit watermark you have still chosen a lateness policy — you just cannot state it or test it.
Report-once needs its own state: per key, a set of already-flagged 10-minute window ids. A later event inside an already-flagged window must not re-flag, and a late event that completes a window you never flagged must flag — which is why state is retired at window close plus the lateness bound, not at window close.
Size it: state lives 660 seconds, so at 3,000 events per second there are about 1.98 million in flight, plus one flagged-window set per active key. Bound total memory explicitly and shed by key age, and say what shedding costs — a shed key can miss a flag, making the threshold a product decision rather than a tuning knob.

Follow-up

Make the threshold and window configurable without rebuilding all in-flight state on every change. What does that constrain in the data structure?
Two stream partitions hold events for the same token. What does that force the partitioning key to be?
An event arrives three hours late. Does any answer change, and who is told?

Derive per-account balances and catch unbalanced transactions

Easy · aggregation · double-entry · streaming

You are given ledger_entry rows streamed in entry_id order: transaction_id, account_id, direction (debit or credit), amount_minor (a positive int64), currency, business_date. Up to 500 million rows, at most 20 million distinct (account_id, currency) pairs, and the entries of one transaction are contiguous in the stream. In a single pass with no re-reads, return the closing balance per (account_id, currency) and the transaction_id of every transaction whose entries do not sum to zero within each currency. State your time and space bounds.

Approach

Normalise the sign at read time from direction, not from the amount: signed = +amount_minor for debit, -amount_minor for credit (state which convention you picked). The schema constrains amount_minor > 0 precisely so the sign lives in exactly one place.
Hold one hash map keyed (account_id, currency) to an int64 running total. Twenty million keys at 16 bytes of payload plus map overhead is order 1 GB in most runtimes — quote the number, and offer the fallback: partition the stream by hash(account_id) % P and run P passes for 1/P of the memory.
Ride the zero-sum check on the same pass. Because a transaction's entries are contiguous, keep a tiny currency -> int64 map for the current transaction_id only, test it against zero on the boundary and at EOF, then clear it. That is O(currencies in one transaction), typically one or two.
Bound the arithmetic explicitly. Int64 holds about 9.22e18, so overflowing one account across 500 million entries needs an average of 1.8e10 minor units per entry — safe here, but use a checked add so an adversarial file fails loudly rather than wrapping.
Complexity: O(n) time, O(distinct account-currency pairs) space, one sequential pass, no sort. The zero-sum check adds no asymptotic cost, which is the argument for doing it here rather than in a second job.

Worked solution (20 min)

Write the sign rule down in one sentence before any code, naming which side debit is positive on, and apply it at read.
Implement with two maps — balances: (account_id, currency) -> int64 and txn: currency -> int64 — plus the current transaction_id.
On a change of transaction_id, assert every currency in txn sums to zero, record the id if not, then clear.
Feed a fixture: one 2-entry transaction that balances; one 4-entry transaction with USD and JPY legs that balances within each currency; one 3-entry transaction off by a single minor unit.
Re-run with the entries shuffled inside each transaction to prove the result is order-independent within a transaction.

Expected result: The balances map is identical under any within-transaction ordering; exactly the third transaction is reported; the mixed-currency transaction is accepted, because zero-sum is required per currency, not across the transaction.

Verify it yourself

Across the whole file, the sum of every account balance is exactly 0 per currency whenever no transaction is flagged.
Two runs over the same file produce byte-identical output.
Swapping the int64 accumulator for binary64 makes a fixture with amounts above 2^53 minor units disagree — that is a failing test, not a review comment.

Follow-up

Entries of a transaction are no longer contiguous. What does the zero-sum check cost now, and which is cheaper: buffering open transactions or an external sort on transaction_id?
How would you produce the same balances as of an arbitrary business_date without a second full scan?
The job is restarted after a crash halfway through the file. What makes the second run produce identical output?

Rank merchants by unmatched break value in one streamed pass

Medium · top-k · heap · streaming · space-bounded

A reconciliation run streams up to 50 million break rows of (merchant_id, currency, delta_minor signed, break_type). Return the 50 merchants with the largest total absolute delta_minor in a single currency, from one pass, with memory that does not grow with merchant count — there are up to 8 million distinct merchants and room for roughly 200,000 counters. Give both an exact and an approximate design, state the error bound the approximate one actually guarantees, and say which you would put in the nightly job.

Approach

Start by testing whether the constraint is real. Exact needs one hash map merchant_id -> int64 plus a size-50 min-heap: O(n) to aggregate, O(m log 50) to rank, O(m) space. At 8 million merchants and 16 bytes of payload that is a few hundred megabytes — quote the number before reaching for a sketch.
If it genuinely does not fit, exact still costs only two passes or an external group-by: partition by hash(merchant_id) % P to disk, aggregate each partition independently, then merge with a K-way heap. This is the answer whenever exactness is non-negotiable, which for money it usually is.
One-pass approximate: weighted Space-Saving with C counters. An item either hits an existing counter or evicts the minimum one and inherits its value as an over-estimate. With total weight W and C counters, every reported total over-estimates by at most W/C, and any merchant whose true total exceeds W/C is guaranteed to be in the table. Quote W/200,000 as a fraction of the day's break value.
State what the bound does not give you: the ordering within the top 50 is not guaranteed, and a merchant just below the threshold can be missed entirely. The honest claim to an operations reader is 'contains everyone above 0.0005% of today's break value', not 'the top 50'.
Decide the metric before either design, because sum(abs(delta)) and abs(sum(delta)) are different questions: a merchant with offsetting +1,000,000 and -1,000,000 breaks is top-ranked under the first and invisible under the second. Break work wants the first.
Recommend exact (two-pass or external group-by) for the nightly job — it runs once, has hours of budget, and an analyst acts on its output. Keep the sketch for a live dashboard, where a cheap bounded approximation beats an exact number that is four hours stale.

Follow-up

Totals are now needed per currency as well as overall. What does that do to the key and to the counter budget?
Prove the Space-Saving over-estimate bound to me in two sentences.
The nightly job restarts halfway. Is the aggregate idempotent, and what does the heap do with a partially consumed stream?

Stop the capture-and-refund join from double-counting merchant money

Medium · join fan-out · conditional aggregation · grain

Report captured_minor, refunded_minor and net_minor per merchant per business_date. payment_intent holds intent_id, merchant_id, amount_minor, captured_minor, refunded_minor. ledger_entry holds transaction_id, account_id, direction, amount_minor, source_type in (capture, refund, fee, chargeback), source_id = the intent_id as text, and business_date; every capture posts one debit and one credit. A draft joins payment_intent to ledger_entry twice, once filtered to captures and once to refunds, and sums both. On intents with two partial captures and one refund the totals are wrong. Name both multiplications precisely and write the correct query.

Approach

Name the first multiplication by cardinality: intent to capture entries is one-to-N and intent to refund entries is one-to-M, so joining both yields N times M rows per intent. Every capture row is then counted M times and every refund row N times, and the two columns are inflated by different factors, which is why the totals look almost right rather than obviously broken.
Name the second, which is easier to miss: double-entry means each capture posts two legs, so summing every entry of a capture counts the amount twice regardless of joins. Select the single leg by ACCOUNT - the merchant's settlement payable, which carries exactly one leg of every capture and of every refund - and not by direction. Direction is not a substitute for that filter: a capture credits the payable and a refund debits it, so adding direction = 'credit' keeps the captures and silently drops every refund leg, leaving refunded_minor at exactly zero on a query that otherwise reads as correct.
Reject the reflexive fixes: SELECT DISTINCT and SUM(DISTINCT ...) de-duplicate values, not rows, so two genuine captures of equal amount collapse into one and the total is wrong in the other direction while looking tidier.
Prefer a single pass with conditional aggregation over ledger_entry, restricted to that one account: SUM(amount_minor) FILTER (WHERE source_type = 'capture') and the same for refunds, grouped by merchant and business_date, joining payment_intent only to reach merchant_id. One scan, no intent-level fan-out, and fees or chargebacks are added as another FILTER rather than another join. amount_minor is a positive magnitude, so net_minor is captured minus refunded; the signed form over the same rows, SUM(CASE WHEN direction = 'credit' THEN amount_minor ELSE -amount_minor END), returns the same net in one column and is the check on the sign convention.
Keep aggregate-then-join in reserve for when per-intent detail is required: aggregate each side to intent grain in its own CTE, then join the two aggregates one-to-one. LATERAL works too and reads better when the right side needs the left's key.
Get the account predicate, the join column and the date right. Each merchant has its own settlement payable, so the predicate is a join to the account dimension (a.account_type = 'merchant_payable' AND a.merchant_id = pi.merchant_id); a single $settlement_account literal is only correct for a one-merchant report. source_id is text and intent_id is uuid, and PostgreSQL has no implicit cast between them, so one side must be cast and the side decides which index stays usable - pi.intent_id = e.source_id::uuid probes payment_intent's primary key per entry row, e.source_id = pi.intent_id::text probes an index on ledger_entry(source_id), and if neither fits the driving table the planner hashes both sides, which is fine for a day's report and not for a single-intent lookup. Group on ledger_entry.business_date rather than posted_at::date, because they differ across the cutoff, and decide explicitly whether a later-dated refund reduces its own day or the original capture's.

Follow-up

A refund lands on a later business_date than its capture. Which day does net_minor move, and what does that do to a daily merchant payout?
Add scheme fees, which arrive netted into a batch total rather than per transaction. How does the query change, and what can it no longer claim?
How would you reconcile this output against payment_intent.captured_minor, and which one is authoritative when they disagree?

Store multi-currency amounts and prove each transaction balances

Easy · minor units · normalisation · constraints

ledger_entry holds entry_id, transaction_id, account_id, direction (debit or credit), amount_minor (bigint, CHECK > 0), currency char(3), source_type, source_id, business_date and posted_at. The service posts in JPY, USD and KWD. Specify the column types for money and the reference table that carries each currency's ISO 4217 minor-unit exponent, and say why that exponent must not be a constant in application code. Then write the query that returns every transaction_id whose entries fail to sum to zero within each currency, ordered by the largest absolute imbalance.

Approach

State the representation: amount_minor is an integer count of minor units and the scale lives with the currency, not in the column. int4 tops out at 2,147,483,647 minor units, which is about 21.5 million units of a 2-exponent currency, so bigint, and the currency code travels on every row that carries an amount.
Normalise the scale into currency(code char(3) primary key, exponent smallint, name) and join it only at display and parse boundaries. The exponent is 0 for JPY and KRW, 2 for USD and EUR, 3 for KWD, BHD, JOD, OMR and TND, so a hard-coded divide by 100 is wrong by 100x for JPY and by 10x for KWD, in opposite directions.
Rule out binary floating point outright: IEEE 754 binary64 cannot represent 0.1, so repeated accrual drifts by a few minor units that later appear as reconciliation breaks. If a fixed-scale decimal is used instead, pin the scale per currency and name the single place rounding happens.
Write the balance check as a grouped aggregate: sign the amount with a CASE on direction, GROUP BY transaction_id, currency, and HAVING the signed sum <> 0. The sign belongs in direction, and the CASE is the one place it becomes arithmetic.
Then order in a second layer, because the ordering the question asks for cannot be written on the grouped query itself. PostgreSQL lets an output alias stand alone in ORDER BY but resolves anything inside an expression against the input columns, so ORDER BY abs(imbalance_minor) DESC raises SQLSTATE 42703, column "imbalance_minor" does not exist, while the bare ORDER BY imbalance_minor DESC is accepted and sorts signed rather than absolute. Wrap the grouped query and order by abs() outside it, or repeat the aggregate as ORDER BY abs(SUM(CASE ...)) DESC, which is legal because aggregates are allowed there.
Group per (transaction_id, currency) rather than per transaction, because a cross-currency movement balances only within each currency, bridged by an explicit FX position account whose rate, source and timestamp are stored on the transaction.
Note that no CHECK constraint can express this, since it spans rows: enforce it at write time with a DEFERRABLE INITIALLY DEFERRED constraint trigger that fires at commit, or by routing every posting through one function, and keep this query as the independent audit.

Worked solution (15 min)

Define currency(code, exponent, name) and seed JPY 0, USD 2, KWD 3; declare amount_minor bigint NOT NULL CHECK (amount_minor > 0) and currency char(3) REFERENCES currency(code).
Write it in two layers so it runs: SELECT transaction_id, currency, imbalance_minor FROM (SELECT transaction_id, currency, SUM(CASE WHEN direction = 'credit' THEN amount_minor ELSE -amount_minor END) AS imbalance_minor FROM ledger_entry GROUP BY transaction_id, currency HAVING SUM(CASE WHEN direction = 'credit' THEN amount_minor ELSE -amount_minor END) <> 0) b ORDER BY abs(imbalance_minor) DESC.
Post one clean capture, one deliberately unbalanced transaction, and one cross-currency transfer with an FX position account, then run the query.
Convert 1000 JPY and 1000 KWD through the exponent table in both directions and confirm the round trip is exact.

Expected result: Exactly one row: the deliberately unbalanced transaction, with a signed imbalance_minor equal to the amount omitted. The clean capture and both single-currency sides of the FX transfer return nothing.

Verify it yourself

The cross-currency transaction produces two groups, one per currency, and neither appears in the output.
Flipping one entry's direction makes imbalance_minor exactly twice that entry's amount, which confirms the sign convention.
Inserting an entry with amount_minor = 0 or a negative value is rejected by the CHECK rather than by application code.
Execute the statement rather than reading it: ordering the grouped query directly by abs(imbalance_minor) fails with 42703, which is why the abs() sits outside the subquery.

Follow-up

A transaction has a USD leg and a JPY leg. Which accounts appear, and where does the rounding residual land?
How would you enforce the zero-sum rule at write time without serialising all postings behind one lock?
Someone proposes storing a display amount alongside amount_minor. What goes wrong at the first rate change or rounding rule change?

Name as many uses as you can for a coffee cup in 60 seconds.

Medium

Approach

Clarify what is being asked and what a complete answer contains.
Say what you would check first and why it is the highest-information step.
Work from the requirement backwards to the design.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Create something on the fly using provided materials (e.g., Play-Doh) within 30 seconds.

Medium

Approach

State your assumptions explicitly before working the problem.
Work from the requirement backwards to the design.
Clarify what is being asked and what a complete answer contains.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Ingest daily settlement files idempotently and restartably

Easy · idempotency · batch ingestion · partitioning · restartability

A processor delivers one settlement file per business date containing 10 to 50 million lines, each with an external reference, a signed amount in minor units, a currency, a fee and the processor's settlement date. Files arrive late, are sometimes re-sent byte-identical, and are sometimes re-sent corrected under the same filename. Design ingestion: how a line becomes a settlement_line row, how a re-send avoids duplicating 50 million rows, how a correction supersedes the earlier version without deleting it, and how a run that dies at line 30 million resumes. State your throughput target.

Approach

Treat the file as a versioned object rather than a stream of events. Key a file record on (processor_id, business_date, content_sha256): an identical re-send matches the digest and becomes a no-op in O(1), while a corrected re-send has a different digest and becomes file_version 2, which supersedes version 1 by stamping superseded_at on its lines rather than deleting them. Deleting them destroys the evidence for any break already raised against version 1.
Give lines two levels of identity. UNIQUE (file_version_id, line_no) makes a resumed chunk unable to double-insert. Deduplication on business content must not collapse (external_reference, amount_minor, currency) into one row, because two identical genuine charges in one day are ordinary; if you need a business-level key, it has to include the line ordinal.
Make restart cheap by committing in chunks of about 10,000 lines and recording a high-water line_no per file_version, so a crash costs at most one chunk. Smaller chunks mean more commit and WAL overhead; larger chunks mean more re-work after a failure. Pick from measured commit cost, not from a round number.
Set the throughput target from the loading mechanism: a bulk COPY of narrow rows runs at order 10^5 rows/s per connection where row-by-row INSERT runs at order 10^3 to 10^4, so 50 million lines is minutes rather than hours. Parallelise by splitting the file into byte ranges aligned to line boundaries, one worker and one checkpoint per range.
Partition settlement_line by business_date. A full re-ingest then rebuilds one partition instead of issuing a 50-million-row DELETE, and retention becomes a partition DETACH rather than a long-running vacuum problem.
Validate before publishing the version: compare the file trailer's control totals against SUM(amount_minor) over the loaded rows, and fail the run on mismatch. A truncated file that loads cleanly otherwise looks exactly like a day when the processor sent less money.

Worked solution (30 min)

Generate a 10,000,000 line file, compute its SHA-256, and load it with COPY into a staging table partitioned by business_date, committing every 10,000 lines and recording the high-water line number.
Ingest the identical file again and assert zero new rows, with no work done beyond the digest comparison.
Alter 100 lines, re-ingest under the same filename, and assert a new file_version supersedes the first while version 1 rows remain fully readable.
Kill the worker at line 4,000,000, restart it, and count the final rows.

Expected result: COPY sustains order 10^5 narrow rows per second per connection, so 10 million lines land in roughly one to two minutes on a single connection and scale close to linearly with byte-range parallelism. The identical re-send costs one digest comparison. The crash-restart run ends with exactly 10,000,000 rows: no duplicate chunk, no gap at the restart boundary.

Verify it yourself

UNIQUE (file_version_id, line_no) exists and is what makes the resumed chunk safe, rather than the worker's own bookkeeping.
Two lines identical in reference, amount and currency both survive ingestion.
Control totals from the trailer equal SUM(amount_minor) for the version, and a deliberate truncation fails the run instead of opening breaks.

Follow-up

The same file starts ingesting twice concurrently from two workers. What stops the second one?
The corrected file has fewer lines than the original. Which lines disappeared, and how does a break already raised against one of them get resolved?
How do you know the file is complete rather than still being written by the transfer?

Settlement postings plateau at 310 per second and deadlock

Medium · lock contention · deadlock · throughput ceiling · sharding

Ledger postings against one pooled merchant settlement account plateau at about 310 committed transactions per second. Adding workers beyond 24 raises latency linearly and leaves throughput flat. Separately, about 0.4% of two-account transfers abort with SQLSTATE 40P01. Each posting takes SELECT ... FOR UPDATE on the materialised balance row, validates a floor, inserts the entries, then updates the balance. Explain both numbers, give the fix for each, and state precisely what sharding the hot balance would cost the floor check.

Approach

Compute the ceiling instead of guessing. A row-level exclusive lock serialises every transaction touching that row, so the maximum committed writes per second is 1 divided by the lock hold time, where hold runs from FOR UPDATE to COMMIT and includes the entry inserts, the WAL flush and anything else inside the transaction. 310 per second implies about 3.2 ms held. Measure it via pg_locks joined to pg_stat_activity rather than inferring it.
Recognise what the flat-throughput, rising-latency curve proves. Past the ceiling, extra workers only lengthen the wait queue; that is serialisation, not saturation, and no amount of CPU, replicas or pool size changes it. Establishing this rules out the three most common wrong fixes before proposing anything.
Shorten the critical section before sharding anything. Take the lock last, never hold it across an application round trip or a processor call, and replace SELECT-then-UPDATE with one conditional statement: UPDATE balance SET amount_minor = amount_minor - $1 WHERE account_id = $2 AND amount_minor - $1 >= floor_minor. It needs no prior read and holds the row only for that statement, so halving hold time doubles the ceiling for free.
Treat the 40P01 as a separate defect with a separate fix. Two transfers moving money in opposite directions between accounts A and B acquire the two row locks in opposite orders and wait on each other until the detector aborts one. Acquiring locks in a deterministic order, such as ascending account_id, makes the cycle impossible rather than merely rarer; a bounded retry on 40P01 remains prudent but is no longer the mechanism.
Only then shard, and price it honestly. N sub-rows multiply the ceiling by roughly N, but the floor becomes a predicate over N rows that a single-row conditional UPDATE cannot express. Either each sub-row carries its own floor, which over-restricts by refusing a payment while funds sit on another shard, or you move to SERIALIZABLE with a bounded retry on 40001 across the sub-rows. Establish first whether this pooled clearing account has a floor at all, because if it does not, sharding is nearly free and the whole trade-off disappears.

Follow-up

At what N does rebalancing funds between sub-rows cost more than the throughput it buys?
How would you measure lock hold time in production without attaching a profiler?
Does moving to SERIALIZABLE eliminate the 40P01 aborts?

Suggested practice checklist

Roughly ninety minutes on weeknights with one longer weekend block. The plan cuts scope rather than compressing everything, on the assumption that one thing finished per night beats four half-started.

Day 1 — Fix the scope and take a cold baseline

Read the role description and write the three things the loop will almost certainly test, then write an explicit not-doing list and keep it visible all week.
Take one twenty-five-minute coding problem and one fifteen-minute design prompt cold, and write the single sentence naming what blocked each, because those two sentences decide where the remaining evenings go.
Set the week's rule: one thing finished every night, including the night you only have forty minutes.

Deliverable: A one-page scope with a not-doing list and two cold attempts, each carrying one sentence on what blocked it.

Day 2 — One pattern, written three times from blank

Choose the single pattern most likely to appear in your loop and write it three times from an empty file rather than editing the previous attempt.
On the third pass, write the invariant as a comment before the loop body and the complexity before the first line of code.
Stop at ninety minutes even if the third version is imperfect, and write the one thing you would fix given another hour.

Deliverable: Three independent implementations of the same pattern plus a note on what changed between them.

Day 3 — One design, only to the depth you can defend

Take one system shape and go only as far as requirements, interface and data model, refusing to draw a box you could not survive a follow-up about.
Attach one number to each non-functional requirement, deriving it rather than asserting it, and write the assumption the number rests on.
Write the one tradeoff you are choosing against and the observation that would make you reverse it.

Deliverable: One design at interface-and-schema depth with derived numbers and one written reversible tradeoff.

Day 4 — Only the fundamentals you will have to defend

Write, in under two hundred words each, the answers to the two questions that follow almost any implementation: why this structure and not the obvious alternative, and what happens to this code at a hundred times the input.
Write what an index actually costs: faster lookups on the indexed columns against a write that now maintains a second structure, plus the cases where the planner declines to use it anyway, low selectivity, or a predicate wrapping the column in a function.
Delete any answer you cannot deliver aloud in under a minute, since an answer that needs reading is not an answer you have.

Deliverable: Three written answers, each under two hundred words and each timed aloud.

Day 5 — Your own work, timed

Write a ninety-second and a four-minute version of your main project and time both aloud rather than reading them.
Prepare the two follow-ups that always come: what you would do differently, and how you knew it worked.
Put one number in the first sentence and be ready to say exactly where it came from and what it excludes.

Deliverable: Two timed narratives with one defensible number in the opening line.

Day 6 — The one full rehearsal, in the weekend block

Run a sixty-minute mock covering a coding round and a design round in one sitting with no break, because sustained attention is the thing evenings have not trained.
Immediately afterwards, and before hearing any feedback, write the three moments you lost the thread.
Spend the rest of the block only on those three moments, and on nothing you merely feel shaky about.

Deliverable: Mock notes naming three failure moments with a specific fix written under each.

Day 7 — Taper

Write the twenty-minute warm-up you will actually do on the morning: one problem you can already solve from a blank file, one design you can narrate, and nothing you have never seen.
Re-read only your own notes from this week and open no new material.
Write the logistics down: the editor or shared document you will be working in, whether execution and lookups are permitted, and the sentence you will use when you do not know something.

Deliverable: A one-page card holding the design structure, the project numbers, and the logistics.

Disagreement needs an outcome you did not choose

Conflict answers where you were right and everyone came round are the weakest ones. Stronger: the evidence you went and collected, what would have changed your mind, and what you did in the weeks after the call went against you. Implementing a design you argued against, properly, is a specific and checkable behaviour.

What is your experience with server management and virtualization?

Medium · behavioural and engineering judgement

Approach

State the situation in two sentences and spend the rest on the reasoning.
Close with what you would do differently, concretely.
Give the blast radius: what could have broken, and what you measured.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?

Describe a time you failed and how did you handle it?

Medium · behavioural and engineering judgement

Approach

Pick a story where you made the decision, not one where you watched it.
Close with what you would do differently, concretely.
Name the disagreement and how you resolved it with evidence.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?

What are your hobbies and interests?

Medium · behavioural and engineering judgement

Approach

Pick a story where you made the decision, not one where you watched it.
Name the disagreement and how you resolved it with evidence.
State the situation in two sentences and spend the rest on the reasoning.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?

How do you approach team collaboration in a fast-paced environment?

Medium · behavioural and engineering judgement

Approach

Pick a story where you made the decision, not one where you watched it.
Name the disagreement and how you resolved it with evidence.
State the situation in two sentences and spend the rest on the reasoning.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?