Interview guides · Bank of America

Bank of America Software Engineer Interview Questions & Guide 2026

Bank of America software engineer prep on partial failure: a timed-out request has an unknown outcome, not a failed one.

The 60-second brief

A Software Engineer at Bank of America plays a vital role in building, maintaining, and modernizing the tech stack behind one of the world's largest financial institutions. Engineers at Bank of America design enterprise applications that process billions of dollars in transactions daily, power consumer-facing platforms like online and mobile banking, and support high-frequency trading engines, risk analytics, and global market infrastructures. In this environment, software development is directly tied to platform availability, regulatory compliance, performance, and data security.

The title spans product, platform and infrastructure work, and which of those the seat actually is decides whether design or algorithms carries more weight in your preparation. The posting rarely settles it; what the team is on call for usually does.

Focus areas

Consume webhooks duplicated, delayed and out of order
Model money movement as balanced double-entry postings
Store money as integer minor units

Overview

In this role, you will work across diverse engineering ecosystems ranging from core Java microservices, cloud-native deployments, and event-driven architectures with Apache Kafka to modern frontend frameworks like React and high-throughput data processing pipelines using Python and SQL. Engineers collaborate closely with cross-functional partners including quantitative analysts, infrastructure specialists, cybersecurity leads, and product managers to deliver resilient solutions across lines of business such as Global Markets, Consumer Banking, and Enterprise Risk.

What sets software engineering at Bank of America apart is the sheer scale and responsibility. You are not simply writing code; you are architecting systems that must guarantee strict consistency, fault tolerance, and low latency. The engineering culture values clear logic, deep understanding of computer science fundamentals, robust system architecture, and clear communication under technical scrutiny.

The Bank of America interview process

Bank of America candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screening Call

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Technical Interviews

Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.

What to demonstrate

Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
Whether a failing case is isolated and explained before any edit is made to the code

How to prepare

From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.

Coding Assessments

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Behavioral Interviews

Your first answer is not really what is scored. It buys the follow-up questions, and those decide the round. An interviewer with fifteen minutes takes one thread and pushes on it four or five times, so a story you can only tell at a single level of detail collapses under the third why. That is an argument for fewer stories known deeply rather than one prepared per prompt. Four or five pieces of work you can still explain down to the code you changed and the argument you had about it will cover nearly anything asked in this round.

What to demonstrate

Whether a story holds as the questioning moves from what you did to why that instead of the alternative, and then to what you would change knowing what you know now
Whether you can re-cut a project to answer the question actually asked rather than delivering a rehearsed block that answers an adjacent one
Whether your level of detail is chosen rather than habitual: going down to the schema when the question is about the data model, staying out of it when the question is about the person who disagreed with you

How to prepare

Pick four projects and write the chain out four levels deep for each: what you did, why that, why not the alternative, and what would have to be true for the alternative to have won. Where you cannot reach the fourth level, you have a placeholder rather than a story
Have someone ask why three times in a row on a single thread with nothing else added, and mark the point where you start repeating a sentence you already said. That point is where the interviewer stops learning anything
Build a one-page index instead of an answer bank: the common prompts in this round (disagreement, a failure that was yours, thin requirements, a deadline you missed, work you inherited) mapped to which of your four projects you would use for each, so the choosing is done now rather than while an interviewer waits

Where candidates lose points

Deriving the business date from the UTC timestamp

Posting date, value date and the processor's settlement date are three different dates, determined by cutoff times, business-day calendars and holidays rather than by midnight UTC. A movement recorded at 23:50 on one side of a cutoff belongs to the next business date, so computing business_date as created_at::date makes daily totals disagree with every statement and every settlement file. The signature is a reconciliation break that resolves itself the following day and then reopens, which reads like a flaky job and is actually a data model that is missing a column: business_date has to be stored explicitly and set from the cutoff rule, with the timestamptz kept separately for ordering.

Retrying a charge after a timeout

A timeout is not a failure; it is an unknown outcome, and the request may have been processed in full with only the response lost. Re-sending it without an idempotency key that the processor itself honours produces a duplicate charge, which is a customer-visible incident and usually a dispute. The correct handling is to treat the state as unknown, query the processor for that key or client reference, and only then decide. The mechanism also depends on the key being generated once by the caller and reused across every attempt — generating a fresh key per retry turns the whole scheme into a no-op while leaving all the code that appears to implement it in place.

Writing code before the input contract is pinned down

Before the first line, state the types, the size bounds, whether duplicates, negatives or an empty input are possible, whether the input is sorted, whether you may mutate it, and what the function returns when nothing matches. Every one of those answers changes the code, and discovering one at minute twenty costs a rewrite you no longer have time for.

Arguing past a hint

When the interviewer asks what happens for a particular input or floats a different data structure, stop and take it seriously; it is almost always a correction rather than idle curiosity. Talking over it converts a recoverable wrong turn into a data point about how you handle review.

Questions & practice

Given a string, find the length of the longest substring without repeating characters.

Medium · data structures and algorithms

Approach

State the target complexity and say which constraint rules the naive version out.
Choose the data structure from the access pattern, not from familiarity.
Name the brute-force solution and its complexity before improving on it.

Follow-up

What is the worst case, and how likely is it on real data?
How does this change if the input no longer fits in memory?

Write a program to reverse the words in a sentence while retaining character order within specific constraints.

Medium · data structures and algorithms

Approach

Restate the input: its shape, its size, and what is guaranteed about it.
Walk one small example through your approach before writing the whole thing.
Name the brute-force solution and its complexity before improving on it.

Follow-up

What is the worst case, and how likely is it on real data?
Which test case would catch an off-by-one here?

How do you implement a simple in-memory caching mechanism with expiration policies?

Medium · data structures and algorithms

Approach

Walk one small example through your approach before writing the whole thing.
Name the brute-force solution and its complexity before improving on it.
State the target complexity and say which constraint rules the naive version out.

Follow-up

What is the worst case, and how likely is it on real data?
How does this change if the input no longer fits in memory?

Derive per-account balances and catch unbalanced transactions

Easy · aggregation · double-entry · streaming

You are given ledger_entry rows streamed in entry_id order: transaction_id, account_id, direction (debit or credit), amount_minor (a positive int64), currency, business_date. Up to 500 million rows, at most 20 million distinct (account_id, currency) pairs, and the entries of one transaction are contiguous in the stream. In a single pass with no re-reads, return the closing balance per (account_id, currency) and the transaction_id of every transaction whose entries do not sum to zero within each currency. State your time and space bounds.

Approach

Normalise the sign at read time from direction, not from the amount: signed = +amount_minor for debit, -amount_minor for credit (state which convention you picked). The schema constrains amount_minor > 0 precisely so the sign lives in exactly one place.
Hold one hash map keyed (account_id, currency) to an int64 running total. Twenty million keys at 16 bytes of payload plus map overhead is order 1 GB in most runtimes — quote the number, and offer the fallback: partition the stream by hash(account_id) % P and run P passes for 1/P of the memory.
Ride the zero-sum check on the same pass. Because a transaction's entries are contiguous, keep a tiny currency -> int64 map for the current transaction_id only, test it against zero on the boundary and at EOF, then clear it. That is O(currencies in one transaction), typically one or two.
Bound the arithmetic explicitly. Int64 holds about 9.22e18, so overflowing one account across 500 million entries needs an average of 1.8e10 minor units per entry — safe here, but use a checked add so an adversarial file fails loudly rather than wrapping.
Complexity: O(n) time, O(distinct account-currency pairs) space, one sequential pass, no sort. The zero-sum check adds no asymptotic cost, which is the argument for doing it here rather than in a second job.

Worked solution (20 min)

Write the sign rule down in one sentence before any code, naming which side debit is positive on, and apply it at read.
Implement with two maps — balances: (account_id, currency) -> int64 and txn: currency -> int64 — plus the current transaction_id.
On a change of transaction_id, assert every currency in txn sums to zero, record the id if not, then clear.
Feed a fixture: one 2-entry transaction that balances; one 4-entry transaction with USD and JPY legs that balances within each currency; one 3-entry transaction off by a single minor unit.
Re-run with the entries shuffled inside each transaction to prove the result is order-independent within a transaction.

Expected result: The balances map is identical under any within-transaction ordering; exactly the third transaction is reported; the mixed-currency transaction is accepted, because zero-sum is required per currency, not across the transaction.

Verify it yourself

Across the whole file, the sum of every account balance is exactly 0 per currency whenever no transaction is flagged.
Two runs over the same file produce byte-identical output.
Swapping the int64 accumulator for binary64 makes a fixture with amounts above 2^53 minor units disagree — that is a failing test, not a review comment.

Follow-up

Entries of a transaction are no longer contiguous. What does the zero-sum check cost now, and which is cheaper: buffering open transactions or an external sort on transaction_id?
How would you produce the same balances as of an arbitrary business_date without a second full scan?
The job is restarted after a crash halfway through the file. What makes the second run produce identical output?

Decide whether balance is derived or materialised, then hold the floor

Hard · denormalisation · isolation levels · lock contention

Authorisation needs an account's available balance inside an 80 ms budget at 3,000 requests per second; that account already has 200 million ledger_entry rows. A product rule says the balance may never fall below the account's negative overdraft limit. Decide whether the balance is summed from entries or held in a materialised account_balance(account_id, balance_minor, floor_minor, currency, version) row, and justify the choice from the read pattern. Then give the write path that holds the floor, naming the isolation level, the anomaly a weaker level permits, and the SQLSTATE you retry.

Approach

Size the derived read before arguing about it: summing 200 million rows is not an 80 ms operation under any index, since even a covering index on (account_id, entry_id) still reads work proportional to the rows. The authorisation read pattern forces one materialised row fetched by primary key; the statement read pattern, which is low-rate and historical, stays derived from entries. That is the whole justification for the denormalisation, and its price is a write on that row per posting.
Put the floor where it can be a single-row constraint: CHECK (balance_minor >= floor_minor) on account_balance, updated in the same transaction as the entries. As a predicate over a set of entry rows it cannot be a CHECK at all, which is the reason the materialised row earns its keep twice.
Write the mutation as one statement: UPDATE account_balance SET balance_minor = balance_minor - $1, version = version + 1 WHERE account_id = $2 AND balance_minor - $1 >= floor_minor. Zero rows updated means refused. This is safe even under READ COMMITTED, because the UPDATE re-evaluates its predicate against the locked, post-update version of the row.
Name the anomaly in the shape that is not safe: SELECT the balance, compute a new value in application code, then UPDATE to that constant. Every statement under READ COMMITTED takes a fresh snapshot, so two concurrent 60-unit withdrawals against 100 both read 100 and both write 40. The row then claims 40 while 120 has actually left, so the true position is -20, below a floor of 0, and the row it is checked against cannot show it. PostgreSQL REPEATABLE READ is snapshot isolation and aborts the loser with SQLSTATE 40001; SERIALIZABLE additionally closes write skew across two rows. Both require a bounded retry with backoff, and InnoDB REPEATABLE READ does not abort at all, so identical code changes behaviour on a different engine.
For a two-account transfer, acquire the rows in a deterministic order such as ascending account_id; without it, opposing concurrent transfers deadlock and the database kills one with SQLSTATE 40P01. That is a retry, not a correctness failure, but it is a retry somebody has to write.
Finish on the ceiling: the hot row admits one committed write per lock hold, so at a 2 ms hold it caps near 500 per second regardless of cores. Sharding into N sub-rows multiplies throughput and immediately makes the floor check cross-row again, which then needs the shard sum under SERIALIZABLE or a per-shard reserved allowance.

Follow-up

Shard the balance into eight sub-rows. Write exactly what the floor check now does, and what it costs per authorisation.
Someone proposes an AFTER INSERT trigger on ledger_entry to maintain the balance. What does that change about ordering, about batch posting, and about failure handling?
How do you detect that the materialised row has drifted from the entries, how often do you run it, and on which replica?

Stop the capture-and-refund join from double-counting merchant money

Medium · join fan-out · conditional aggregation · grain

Report captured_minor, refunded_minor and net_minor per merchant per business_date. payment_intent holds intent_id, merchant_id, amount_minor, captured_minor, refunded_minor. ledger_entry holds transaction_id, account_id, direction, amount_minor, source_type in (capture, refund, fee, chargeback), source_id = the intent_id as text, and business_date; every capture posts one debit and one credit. A draft joins payment_intent to ledger_entry twice, once filtered to captures and once to refunds, and sums both. On intents with two partial captures and one refund the totals are wrong. Name both multiplications precisely and write the correct query.

Approach

Name the first multiplication by cardinality: intent to capture entries is one-to-N and intent to refund entries is one-to-M, so joining both yields N times M rows per intent. Every capture row is then counted M times and every refund row N times, and the two columns are inflated by different factors, which is why the totals look almost right rather than obviously broken.
Name the second, which is easier to miss: double-entry means each capture posts two legs, so summing every entry of a capture counts the amount twice regardless of joins. Select the single leg by ACCOUNT - the merchant's settlement payable, which carries exactly one leg of every capture and of every refund - and not by direction. Direction is not a substitute for that filter: a capture credits the payable and a refund debits it, so adding direction = 'credit' keeps the captures and silently drops every refund leg, leaving refunded_minor at exactly zero on a query that otherwise reads as correct.
Reject the reflexive fixes: SELECT DISTINCT and SUM(DISTINCT ...) de-duplicate values, not rows, so two genuine captures of equal amount collapse into one and the total is wrong in the other direction while looking tidier.
Prefer a single pass with conditional aggregation over ledger_entry, restricted to that one account: SUM(amount_minor) FILTER (WHERE source_type = 'capture') and the same for refunds, grouped by merchant and business_date, joining payment_intent only to reach merchant_id. One scan, no intent-level fan-out, and fees or chargebacks are added as another FILTER rather than another join. amount_minor is a positive magnitude, so net_minor is captured minus refunded; the signed form over the same rows, SUM(CASE WHEN direction = 'credit' THEN amount_minor ELSE -amount_minor END), returns the same net in one column and is the check on the sign convention.
Keep aggregate-then-join in reserve for when per-intent detail is required: aggregate each side to intent grain in its own CTE, then join the two aggregates one-to-one. LATERAL works too and reads better when the right side needs the left's key.
Get the account predicate, the join column and the date right. Each merchant has its own settlement payable, so the predicate is a join to the account dimension (a.account_type = 'merchant_payable' AND a.merchant_id = pi.merchant_id); a single $settlement_account literal is only correct for a one-merchant report. source_id is text and intent_id is uuid, and PostgreSQL has no implicit cast between them, so one side must be cast and the side decides which index stays usable - pi.intent_id = e.source_id::uuid probes payment_intent's primary key per entry row, e.source_id = pi.intent_id::text probes an index on ledger_entry(source_id), and if neither fits the driving table the planner hashes both sides, which is fine for a day's report and not for a single-intent lookup. Group on ledger_entry.business_date rather than posted_at::date, because they differ across the cutoff, and decide explicitly whether a later-dated refund reduces its own day or the original capture's.

Worked solution (30 min)

Build a fixture with one intent that has two partial captures and one refund, each posting a debit and a credit, and compute the correct totals by hand first.
Run the double-join draft against it and express each column's error as a product of the two mechanisms, rather than guessing: four capture rows and two refund rows, so the capture sum is doubled by its legs and multiplied by the two refund rows, and the refund sum is doubled by its legs and multiplied by the four capture rows.
Write the single-pass version: SELECT pi.merchant_id, e.business_date, COALESCE(SUM(e.amount_minor) FILTER (WHERE e.source_type = 'capture'), 0) AS captured_minor, COALESCE(SUM(e.amount_minor) FILTER (WHERE e.source_type = 'refund'), 0) AS refunded_minor FROM ledger_entry e JOIN payment_intent pi ON pi.intent_id = e.source_id::uuid JOIN account a ON a.account_id = e.account_id AND a.account_type = 'merchant_payable' AND a.merchant_id = pi.merchant_id GROUP BY 1, 2, with net_minor as the difference. No direction predicate: the account join already picks one leg out of each pair, and a direction filter would drop the refunds.
Compare both queries against the hand-computed fixture and against SUM(payment_intent.captured_minor) for the same merchant and day.

Expected result: On that fixture the draft reports captured_minor at four times the true figure and refunded_minor at eight times it, the two factors being 2 legs times 2 refund rows and 2 legs times 4 capture rows. The single-pass query matches the hand-computed totals exactly, refunded_minor is the refund's own magnitude rather than zero, and net_minor equals captured_minor minus refunded_minor to the minor unit.

Verify it yourself

Add a second refund and confirm the correct query's captured_minor does not move at all, while the draft's rises again.
Drop the merchant-payable predicate and watch every total double exactly, which isolates the double-entry legs from the join fan-out. That test only works because there is no direction filter left to mask one leg.
Assert refunded_minor is non-zero on the fixture, then re-add direction = 'credit' and watch it fall to exactly zero while captured_minor does not move - the silent failure this account predicate avoids.
Cross-check the day's captured_minor against SUM over payment_intent.captured_minor and account for any difference by business_date rather than by tolerance.

Follow-up

A refund lands on a later business_date than its capture. Which day does net_minor move, and what does that do to a daily merchant payout?
Add scheme fees, which arrive netted into a batch total rather than per transaction. How does the query change, and what can it no longer claim?
How would you reconcile this output against payment_intent.captured_minor, and which one is authoritative when they disagree?

How do you externalize configuration settings securely across different environment deployments?

Medium

Approach

State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer contains.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Explain the core differences between abstraction and encapsulation in Java.

Medium

Approach

Say what you would check first and why it is the highest-information step.
Work from the requirement backwards to the design.
Clarify what is being asked and what a complete answer contains.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Verify, deduplicate and order inbound processor webhooks

Medium · webhooks · signature verification · out-of-order events

A processor posts JSON events to your endpoint: at least once, with no ordering guarantee, retried for 24 hours on any non-2xx, carrying a signature header with a timestamp and an HMAC. Events mutate payment_intent (intent_id, status, captured_minor, version integer incremented on every transition). You are the receiver, and a succeeded event can arrive before the processing event that logically precedes it. Specify the verification steps in order, the deduplication key and where it is stored, the exact write that applies an event, and what you return when you cannot process one.

Approach

Verify before parsing, over the exact raw request bytes, with a constant-time comparison. Re-serialising a parsed body changes key order and whitespace and makes valid signatures fail, which historically gets 'fixed' by weakening verification. A byte-by-byte comparison that short-circuits leaks the matching prefix length through timing.
Reject outside a timestamp tolerance (five minutes is a common value) to bound replay of a captured request. This only works because the timestamp is part of the signed payload; if the signature covers the body alone, an attacker edits the timestamp freely and the tolerance is decorative.
Deduplicate on the provider's event id with a UNIQUE constraint on a received_event table, inserted in the same transaction as the effect. A cache lookup is a check-then-act and admits both copies when two retries arrive at once.
Order by version, not arrival: UPDATE payment_intent SET status = $s, captured_minor = $c, version = $incoming WHERE intent_id = $1 AND version < $incoming. Zero rows updated means the event is stale, which is a successful no-op rather than an error. Arrival order and event timestamps both encode retry and network behaviour rather than causality, so neither can order these events.
Answer 2xx once the event is durably recorded, then do the expensive work from that record. Returning 2xx before durability converts a crash into permanent loss, because the sender will never retry a 2xx. Doing all the work inline risks exceeding the sender's timeout and earning a duplicate that your dedup has to absorb anyway.
Treat an unrecognised event type as success and store the raw payload. Returning an error makes the provider retry for 24 hours over something you will never handle, and buries the events you do care about in the retry backlog.

Worked solution (30 min)

Write the receiver so the raw body is captured before any JSON parsing, and verify HMAC(timestamp + '.' + raw_body) with a constant-time compare.
Add the tolerance check and prove it: replay a captured, validly signed request six minutes later and assert rejection.
Add received_event with UNIQUE (provider, event_id) and insert it in the same transaction as the intent update; fire the same event twice concurrently and assert one effect.
Send succeeded (version 7) before processing (version 5) and assert the final row is version 7 with status succeeded, and that the late event returns 2xx with zero rows updated.
Send an unknown event_type and assert 2xx plus a stored raw payload, and confirm no retry arrives.

Expected result: One state change per event id however many copies arrive, a final intent at the highest version received regardless of arrival order, rejection of a replay outside the tolerance window, and 2xx for both stale and unknown events.

Verify it yourself

Flipping one byte of the raw body causes verification to fail, and the failure path never parses the body.
The out-of-order test leaves status succeeded, not the value from the later-arriving, lower-version event.
Every path that returns 2xx has already committed either the effect or the stored raw event.

Follow-up

The provider rotates its signing secret. How does the endpoint accept both secrets without a window where either is trusted blindly?
You receive an event for an intent_id you have never seen. Is that a bug, a race, or an attack, and how do you tell?
Your handler throws on a malformed payload that is correctly signed. What status do you return, and how do you stop the 24-hour retry loop?

Authorisation p99 tripled overnight with no deploy

Medium · index bloat · autovacuum · long-running transaction · latency

Payment orchestration serves about 3,000 authorisations per second against a 150 ms p99 budget. Since 02:00, p99 is 460 ms and rising about 8 ms per hour, while p50 is unchanged at 11 ms. There was no deploy, no traffic change and no processor degradation. The hot path does one INSERT into idempotency_key, which has UNIQUE (scope, key), then two UPDATEs on that row: locked_at, then status and response_body. A nightly reconciliation job started at 01:50 and is still running. Give the ordered diagnostic checklist and the cause.

Approach

Read the shape first. p50 flat with p99 rising and no deploy is a resource or data-volume effect, not a code path, because a code change moves the median too. A tail that climbs monotonically at fixed workload means something monotonically grows.
Ask what started at 01:50. In PostgreSQL an open transaction holds back the xmin horizon cluster-wide, so autovacuum can reclaim no dead tuple newer than that snapshot. Confirm with pg_stat_activity (state, now() - xact_start, backend_xmin) and with pg_stat_all_tables (n_dead_tup, last_autovacuum) for idempotency_key.
Connect it to the write pattern. Three writes per key produce up to two dead tuples each, so at 3,000 rps the table sheds roughly 6,000 dead tuples per second. Heap-only tuple updates would keep those out of the index, but only when no indexed column changes and the page has room, and appending response_body grows the tuple enough to force a new page. So the unique index on (scope, key) grows too.
Explain why only the tail suffers. A larger index means more pages per lookup and a rising fraction of them missing shared_buffers; the median request still hits cache while the tail pays physical I/O. This is exactly the p50-flat, p99-rising signature, and it is worth stating before acting.
Verify before fixing rather than after. n_dead_tup in the millions and rising, last_autovacuum stale since about 01:50, and pg_relation_size on the unique index measured twice fifteen minutes apart showing growth at constant workload. Index bloat cannot be inferred from row count alone; use the size series or pgstattuple.
Fix in two moves and prevent separately. End or chunk the long transaction so the reconciliation job commits per batch instead of holding one snapshot over 50M lines, then let autovacuum catch up or run REINDEX CONCURRENTLY. Add a transaction-age alert and a statement timeout on the reporting role. Collapsing the two UPDATEs into one and lowering fillfactor halves dead-tuple production, but that is an optimisation, not the cause.

Follow-up

The reconciliation job legitimately needs a consistent view of 50M lines. How do you give it one without pinning the xmin horizon?
Why did p50 not move at all?
You also run an expires_at cleanup job that DELETEs old idempotency keys. During this incident, does running it help or hurt, and what design avoids the question entirely?

Suggested practice checklist

Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.

Day 1 — Diagnostic, scored before you study anything

Sit a 110-minute diagnostic in four blocks: forty-five minutes on two coding problems, twenty-five on one design prompt taken to interface and data model, twenty of short-answer fundamentals, and twenty delivering two behavioural answers aloud.
Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct and 0 is stuck, grading the artifact rather than how the attempt felt.
Allocate days two to five in proportion to 3 minus each block's score, write the allocation down, and commit to leaving it alone.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

Split the weakest area into named sub-skills and rate each separately. For coding those are restating the problem, choosing the structure, stating the invariant, turning the invariant into loop bounds, handling empty and single-element input, and accounting for complexity out loud.
Attempt three items positioned just above where the rating drops off, and for each write the first move you failed to make.
Re-attempt one of them from blank four hours later with nothing open.

Deliverable: A sub-skill map with the two blocking sub-skills circled.

Day 3 — Drill the blocking sub-skill by repeating the shape

Do eight short repetitions of the same shape rather than eight different problems, so what gets practised is the pattern and not the puzzle.
State the rule you now hold in one sentence, then test it against a case built to break it, a sliding window over an array containing negative values, or a cache-aside read path whose invalidation message is dropped.
Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on the strongest area

Run the same sub-skill decomposition on the second-largest gap in half the time.
Spend twenty-five timed minutes on the block you scored highest, choosing the hardest item you can still finish rather than a warm-up.
Write whether each area fails you on recall, on setup, or on execution, and set the fix accordingly: repetition for recall, a written checklist for setup, timed work for execution.

Deliverable: A second sub-skill map plus a one-line failure diagnosis for each area.

Day 5 — The gap that is not a skill

Record one technical and one behavioural answer, then count two things in the playback: seconds before your first clarifying question, and sentences you began without knowing where they would end.
Practise saying that you do not know, followed by how you would find out, without letting it soften into a guess, and practise stating a complexity or an estimate before being asked for it.
Redeliver one answer under a hard ninety-second cap, which forces structure ahead of detail.

Deliverable: Two recordings with a counted reduction in time-to-first-question.

Day 6 — Retest under day-one conditions

Sit the same 110-minute structure with new prompts of comparable difficulty and score it on the identical rubric.
For any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
Write down which single block you would still lose the offer on.

Deliverable: A second scored rubric placed beside the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

Run a sixty-minute mock over the two blocks that moved least, with an interviewer briefed to interrupt and change direction mid-answer.
Write the recovery script for going blank: restate the question, state your assumption, name the first thing you would check.
Say every rule from the week aloud without reading it, and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive an interruption.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

The failure that was yours

Keep one story where the bad call was yours rather than a dependency's or a manager's. Name the check that would have caught it, whether you added that check afterwards, and whether it has fired since. Answers that route blame outward end the conversation early; answers that end in a guardrail someone still relies on tend to open it up.

Tell me about a accomplishment in your recent engineering projects that you are particularly proud of.

Medium · behavioural and engineering judgement

Approach

Pick a story where you made the decision, not one where you watched it.
State the situation in two sentences and spend the rest on the reasoning.
Close with what you would do differently, concretely.

Follow-up

What did you decide not to do, and why?
How did you know your change caused the improvement?

How do you handle differences of opinion with senior team members when choosing a technical architecture?

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
State the situation in two sentences and spend the rest on the reasoning.
Give the blast radius: what could have broken, and what you measured.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?

Describe a situation where you had to lead a initiative or take on extra responsibility beyond your assigned scope.

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
Give the blast radius: what could have broken, and what you measured.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?