Amex Software Engineer Interview Questions & Guide 2026

The 60-second brief

As a Software Engineer at American Express, you will build and support mission-critical technology platforms that power global financial transactions for millions of cardmembers, merchants, and enterprise partners. Engineering at Amex blends high-scale distributed systems, stringent security requirements, and modern digital customer experiences. Whether you are modernizing core backend payment processors using Java and Go, developing responsive frontend web applications with React and Next.js, or building event-driven microservices on top of Kafka, your code directly influences global commerce.

State every complexity claim with the assumption sitting under it. Hash lookup is O(1) on average and only for a hash that spreads your actual keys; comparison-based sorting cannot beat n log n, though counting or radix sort can when the keys are bounded integers.

Focus areas

• Make every write idempotent under client retries
• Scope every query and cache key by tenant
• Build at-least-once pipelines with explicit deduplication horizons

Overview

The scale of Amex requires engineering solutions that maintain four-nines availability, handle millions of peak requests per second, and process sensitive data with uncompromising precision. As a Software Engineer, you won't just write functional code; you will design resilient software architectures, enforce clean coding practices, and collaborate closely with product managers, security engineers, and enterprise data teams.

Joining Amex offers the opportunity to tackle complex technical challenges at enterprise scale within a supportive, highly collaborative engineering culture. The company prioritizes continuous technical evolution, transitioning legacy workloads into modern cloud infrastructure, adopting containerized microservices, and refining developer platforms to accelerate delivery while upholding rigorous standards.

The Amex interview process

Amex candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Application Review

Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.

What to demonstrate

• Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
• Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
• Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled

How to prepare

• Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
• Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
• Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market

Online Assessment

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

• Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
• Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
• Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

• Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
• Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
• Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Recruiter Screen

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

• Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
• Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
• Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

• Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
• Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
• Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Live Coding Session

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

• Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
• Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
• Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
• Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

• For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
• For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
• Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Final Assessment Center

Nobody in the room with you decides this. Interviewers typically write their rounds up separately, often before seeing anyone else's, and the outcome is settled later from those write-ups. A split panel gets resolved by whichever note carries specific evidence, so what you want out of each room is one concrete thing that person could write down: a bug you caught yourself, a trade-off you named, a decision you owned. The rest is arithmetic. The project you describe in a behavioural conversation is often the same system you sketched an hour earlier, and the two accounts have to agree.

What to demonstrate

• Whether the scale, team size and timeline you attach to a project hold steady when that project resurfaces in a different round
• Whether each interviewer leaves with a specific thing to cite rather than a general impression of competence
• Whether a trade-off you defended in one round survives a challenge in another, instead of being quietly swapped for the answer the new interviewer seemed to want
• Whether a question you have already answered earlier in the day gets the same answer at the same depth, without visible impatience

How to prepare

• Write a one-page sheet per project fixing the figures you will quote — request volume, data size, team size, elapsed time, what broke — and say them aloud from the sheet until they come out identical every time
• For each round on the schedule, decide in advance the one sentence you want in that person's notes, then check in a mock that you said it outright instead of leaving it to be inferred
• Have someone ask you the same project question twice, an hour apart, and diff the two answers for numbers that moved or a trade-off that reversed

Where candidates lose points

One shared connection pool for every tenant and every query class

A single tenant with a large table and a missing index can occupy every connection with slow queries, and every other tenant then waits in connection acquisition -- a queue invisible in database metrics, because the database itself looks healthy while the application starves. Containment is bulkheads: separate pools or per-tenant concurrency caps for interactive requests, background jobs and exports, a statement timeout low enough that a pathological query dies before it accumulates, and an idle-in-transaction timeout so a stuck client cannot pin a connection and its locks indefinitely. One caveat worth knowing in advance: if a transaction-pooling proxy sits in front of the database, session-scoped behaviour changes, so session-level advisory locks and settings applied outside a transaction do not survive the way they do on a direct connection.

Serialising a tenant's writes through select ... for update on a single counter row

It is the first change that makes a counter correct, and it caps that tenant's write throughput at roughly one divided by the lock hold time. A transaction that takes the lock, makes a network call and then commits holds it for the entire round trip: at 2 ms that is about 500 writes per second for the whole tenant, and the largest tenants are exactly the ones that exceed it. The damage then spreads, because every waiter holds a database connection while it queues, so one hot tenant drains the shared pool and the symptom presents as a site-wide latency incident rather than as a lock problem. The repairs are to shrink the critical section to a single statement, to shard the counter into per-(tenant, hour) or per-(tenant, bucket) rows and sum on read, or to batch in memory and flush periodically while accepting the bounded loss that batching implies.

Starting work without saying what you are about to spend time on

State the plan before executing it: the approach, roughly how long it will take, and what you intend to leave hand-waved. That gives the interviewer a chance to redirect you in ten seconds rather than watching you spend fifteen minutes on the wrong sub-problem.

Writing code before the input contract is pinned down

Before the first line, state the types, the size bounds, whether duplicates, negatives or an empty input are possible, whether the input is sorted, whether you may mutate it, and what the function returns when nothing matches. Every one of those answers changes the code, and discovering one at minute twenty costs a rewrite you no longer have time for.

Questions & practice

"How do you detect a cycle in a singly linked list, and what is the optimal space complexity?"

Medium · data structures and algorithms

Approach

1. Choose the data structure from the access pattern, not from familiarity.
2. State the target complexity and say which constraint rules the naive version out.
3. Walk one small example through your approach before writing the whole thing.

Follow-up

• Which test case would catch an off-by-one here?
• What is the worst case, and how likely is it on real data?

"Explain the brute-force approach and the optimal approach for finding the longest substring without repeating characters."

Medium · data structures and algorithms

Approach

1. State the target complexity and say which constraint rules the naive version out.
2. Walk one small example through your approach before writing the whole thing.
3. Choose the data structure from the access pattern, not from familiarity.

Follow-up

• Which test case would catch an off-by-one here?
• How does this change if the input no longer fits in memory?

"Given a tree structure, calculate how long it takes for an infection to spread to all nodes starting from a specific infected node."

Medium · data structures and algorithms

Approach

1. State the target complexity and say which constraint rules the naive version out.
2. Walk one small example through your approach before writing the whole thing.
3. Restate the input: its shape, its size, and what is guaranteed about it.

Follow-up

• How does this change if the input no longer fits in memory?
• What is the worst case, and how likely is it on real data?

"Given an array of integers, return all unique triplets that sum to zero."

Medium · data structures and algorithms

Approach

1. Choose the data structure from the access pattern, not from familiarity.
2. Name the brute-force solution and its complexity before improving on it.
3. Restate the input: its shape, its size, and what is guaranteed about it.

Follow-up

• What is the worst case, and how likely is it on real data?
• How does this change if the input no longer fits in memory?

"Explain the concurrency models in Go, specifically how goroutines, channels, and mutexes work under the hood."

Medium · languages, concurrency and fundamentals

Approach

1. Identify the window where an invariant is briefly untrue.
2. Say what the runtime actually does before reasoning about the code.
3. Distinguish a value from a reference to it, and say which one you handed out.

Follow-up

• What happens if two callers reach this at the same time?
• How would you prove the race exists rather than suspect it?

"How would you design a REST API for a high-concurrency booking or payment processing service?"

Medium · languages, concurrency and fundamentals

Approach

1. Say what the runtime actually does before reasoning about the code.
2. Reach for the cheapest primitive that closes the race, not the broadest lock.
3. Name what is shared across threads and what owns each piece of state.

Follow-up

• What happens if two callers reach this at the same time?
• Where could this allocate more than you expect?

Seal an hour under late data with bounded memory

Hard · watermarks · late-data · quantile-sketch · conditional-write

Metering ingest reads 256 partitions at 10,000 to 40,000 events/second. Events carry occurred_at and ingested_at, and during a producer replay the gap between them is hours. Seal each UTC hour once no more than 50 parts per million of that hour's eventual quantity can still arrive, using memory that does not grow with the size of the replay. Define the watermark, the lateness parameter and how you measure it, the structure holding open hours, and the write that performs the seal. State what an idle partition does to your watermark.

Approach

1. Two clocks, two jobs. Bucket by occurred_at, because that is the hour the customer is billed for, and advance the watermark on ingested_at, because that is what the fold has consumed and what source_max_ingested_at records. Conflating them is what makes late data invisible.
2. The global watermark is the min over partitions of each partition's committed ingested_at, not the max: the fold is trustworthy only as far as the slowest partition. The consequence is that one idle partition pins the watermark forever and nothing seals, so an idle partition must promote its watermark to wall clock after a stated idle timeout, and that timeout becomes a correctness parameter, because a partition that is slow rather than idle gets sealed past.
3. Choose the lateness L from the measured distribution of ingested_at - occurred_at, weighted by quantity rather than by event count. The target is 50 ppm of the hour's quantity, and a replay is rare in events while carrying disproportionate mass, so an event-weighted quantile picks an L that is comfortably wrong at exactly the moment it matters.
4. Measure that quantile in bounded memory. A Greenwald-Khanna summary gives epsilon-approximate quantiles in O((1/epsilon) log(epsilon n)) space; a t-digest costs more per merge but has relative error that tightens at the tails, which is the half of the distribution you are reading at p99.99. Keep a separate summary per tenant class, because one tenant's batch importer is not the population.
5. Hold open hours in a min-heap keyed by hour_start. When the watermark advances, pop every hour with hour_end + L < W and seal it: O(log H_open) per advance and O(1) amortised per event to touch its bucket. Memory is open hours multiplied by distinct (tenant, workspace, sku) keys, so cap the number of simultaneously open hours and spill the oldest into usage_rollup_hourly as status='open' with a revision bump. While an hour is open the row is upsertable, so the store is your overflow.
6. The seal itself is a conditional write: update ... set status='sealed', sealed_at=now() where status='open' returning .... Two sealers race on every restart, and the loser must see zero rows and stop rather than write a second value. After the seal, an event for that hour is not an upsert but an adjustment, and source_max_ingested_at is what proves it arrived afterwards.

Worked solution (40 min)

1. Replay a day of events with a synthetic lateness distribution: 99.9% under two minutes, plus a 0.05% tail at four to six hours that carries 3% of total quantity.
2. Compute the p99.99 lateness two ways, event-weighted and quantity-weighted, and put the two numbers side by side.
3. Implement the min-heap of open hours with the watermark as the min over 256 partitions, then stall one partition for 20 minutes and observe what seals.
4. Set the idle-partition timeout to 60 seconds, repeat the stall, and measure how much quantity arrives after the seal.
5. Attempt the seal from two workers at once and confirm the conditional update lets exactly one through.

Expected result: The quantity-weighted p99.99 is hours larger than the event-weighted one. Choosing L from the event-weighted number lets roughly the tail's 3% of quantity land after the seal, 600 times the 50 ppm target. With the min watermark and no idle timeout, the stalled partition blocks all sealing; with a 60-second timeout, the stall is sealed past and its events arrive late.

Verify it yourself

• Post-seal quantity as a fraction of the hour's total is at or below 50 ppm at the chosen L, measured rather than assumed.
• The second sealer's update reports zero rows affected and writes nothing.
• Heap size equals the number of open hours and does not grow with the replay's event count.

Follow-up

• A replay starts during the sealing window for a period you are about to close. What do you do, and what is the customer-visible consequence of each option?
• Your measured quantity-weighted p99.99 lateness is six hours and the invoice must be issued at 02:00 UTC on the first. How do you reconcile those two numbers?
• How would you detect that L has drifted before it costs you an hour's quantity?

Decide which facts an invoice line copies instead of joining

Medium · denormalisation · immutable records · foreign keys · rounding

invoice_line_item already denormalises tenant_id, which is reachable through invoice_id, and stores amount_minor even though quantity times unit_price_micros would recompute it. A reviewer asks you to normalise both away, and separately asks whether the tenant's legal name and billing address should be copied onto the invoice header. Decide each case. For every field you keep denormalised, name the read pattern or the invariant that justifies it, the anomaly the copy can develop, and the mechanism that prevents that anomaly here.

Approach

1. Split the question into two kinds of copy, because they fail differently. A copy of a currently mutable fact is a cache: it drifts and needs invalidation. A copy of a fact frozen at write time is not a cache at all, it is the record of what happened, and normalising it away destroys information the source no longer holds.
2. Keep tenant_id on the line. It costs 8 bytes, it leads every index on the table so no read is ever accidentally cross-tenant, and it turns a wrong join into an empty result rather than another tenant's money. Prevent the drift structurally: a unique constraint on invoice (invoice_id, tenant_id) plus a composite foreign key from the line on (invoice_id, tenant_id) makes a mismatched pair impossible, so the database enforces agreement instead of a code review.
3. Keep amount_minor. Rounding must happen exactly once, at a named site, with a stated mode (half-even here). If readers recompute from quantity and unit_price_micros, every reader owns a rounding decision, and half-up and half-even diverge systematically across thousands of lines rather than cancelling out. A check constraint can bound the stored value but deliberately cannot re-derive it.
4. Copy the legal name and billing address onto the invoice header, written once and never updated. The statement must show what was true when it was sealed, and the tenant record will change afterwards. This is a snapshot for the same reason source_rollup_watermark is stored per line: without it, nobody can reconstruct what the customer was told.
5. Name the read pattern that pays for all of it. Rendering, dispute response and export are per-tenant, per-period reads over thousands of lines that would otherwise join back to slowly changing dimensions that no longer hold the historical value. The write side is a once-per-period batch, so the extra columns cost nothing that matters.
6. Concede the case where the reviewer is right: a mutable operational attribute such as the tenant's current plan name has no business on a line. If a report wants it, join. If a statement needs the plan as of the period, that is another snapshot and it belongs on the header with the rest.

Follow-up

• Write the composite foreign key and the unique constraint it requires on the parent. What does it cost on every line insert, and what does it do to a bulk load?
• A tenant is renamed after being invoiced. Which rows change, and what does the customer see on last quarter's PDF?
• Where does currency live, and what breaks if a tenant's billing currency changes between two periods?

Rebuild an hourly rollup with deduplication and late-arrival accounting

Hard · window functions · deduplication · event time vs ingest time · watermarks

From usage_event (event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at, ingested_at), produce the values usage_rollup_hourly should hold for one tenant over one day: per (workspace_id, sku, hour_start) the deduplicated quantity_sum, event_count and source_max_ingested_at, bucketed by occurred_at. Duplicates share (tenant_id, idempotency_key). Also report, per hour, the running total across the day and the share of quantity that arrived more than two hours after the hour began. Write the query, and state which duplicates a daily unique index cannot catch.

Approach

1. Deduplicate in its own CTE before any aggregation, because a SUM cannot be un-summed: row_number() over (partition by tenant_id, idempotency_key order by ingested_at, event_id) = 1. Include the tiebreaker. Without it the surviving row is non-deterministic when two duplicates share an ingested_at, and a rollup described as deterministically recomputable then disagrees with itself between runs.
2. Bucket on occurred_at and nothing else, and pin the timezone explicitly. date_trunc('hour', timestamptz) truncates in the session's TimeZone setting, so the same query run by a session set to a non-UTC zone buckets differently; use the three-argument date_trunc('hour', occurred_at, 'UTC') on PostgreSQL 16 or later, or date_trunc('hour', occurred_at at time zone 'UTC') at time zone 'UTC' before that. Filter environment = 'production' explicitly, since metering covers three environments and billing covers one.
3. Aggregate to the grain with sum(quantity), count(*) and max(ingested_at). The last is not decoration: it is the watermark the row consumed up to, and without it there is no way to prove afterwards what a number did and did not include.
4. Compute the late share inside the dedup-and-aggregate step as a conditional aggregate, sum(quantity) filter (where ingested_at > hour_start + interval '2 hours'), then divide by the hour's total. Compute the running total as a window over the already aggregated rows: sum(quantity_sum) over (partition by workspace_id, sku order by hour_start rows between unbounded preceding and current row). Running either over raw rows puts the duplicates back.
5. Answer the index question exactly. The unique constraint is on (ingested_day, tenant_id, idempotency_key), because a unique index on a partitioned table must contain the partition key. It therefore deduplicates only within one ingest day and admits a duplicate whose retry crosses midnight or whose replay runs a week later. That is why this CTE dedups across the whole window being recomputed, and why the dedup horizon is a correctness parameter rather than a retention cost.
6. Keep the numeric type all the way through. quantity is numeric so the sums are exact; a cast to double precision anywhere in this pipeline reintroduces drift that surfaces only as a few unreconcilable cents per tenant per month, long after the query is out of anyone's mind.

Worked solution (40 min)

1. Generate a day of events for one tenant across three workspaces and two SKUs, inject 2% duplicates sharing an idempotency_key with a later ingested_at, and inject 1% whose ingested_at is six hours past occurred_at.
2. Write the dedup CTE alone and confirm its row count drops by exactly the injected duplicate count.
3. Aggregate and compare quantity_sum against a ground truth computed outside SQL from the generator's own records.
4. Add the two window and filtered aggregates, and confirm the final hour's running total equals the ungrouped day total for each (workspace, sku).
5. Run the whole query twice over static data and diff the two outputs.
6. Re-run with the session TimeZone set to a non-UTC zone and confirm hour_start does not move.

Expected result: One query: a dedup CTE using row_number() over (partition by tenant_id, idempotency_key order by ingested_at, event_id); aggregation by (workspace_id, sku, hour_start) producing quantity_sum, event_count and max(ingested_at) plus a filtered late-quantity aggregate; a running total as a window over the aggregated rows; and an explicit statement that the partition-scoped daily unique index cannot catch a duplicate that crosses the ingest-day boundary or arrives in a later replay.

Verify it yourself

• Row count after dedup equals raw count minus injected duplicates, exactly.
• The running total at the final hour equals the ungrouped day total for that (workspace, sku).
• Two consecutive runs over static data return byte-identical rows.
• Every hour_start is hour-aligned in UTC and unchanged by the session's TimeZone setting.

Follow-up

• A dispute forces the same recompute over 40 days for one tenant. What changes about the dedup CTE's memory use and the chosen plan, and what would you do about it?
• Two runs a minute apart return different quantity_sum values for an hour that is already closed. Give two mechanisms that produce that, and the single query that distinguishes them.
• Express the same rollup incrementally so it does not re-scan the day each time the watermark advances. What does the incremental version stop being able to answer?

"Explain the internal implementation of a HashMap in Java, including collision resolution and resizing mechanisms."

Medium · system design and architecture

Approach

1. Choose a partition key and say what query it makes expensive.
2. Fix the scope first: who calls this, how often, and what they do when it fails.
3. Name the read and write paths separately; they rarely have the same bottleneck.

Follow-up

• How does this behave when that dependency is down for an hour?
• What breaks first when traffic grows ten times?

"What is the difference between an abstract class and an interface, and when would you choose one over the other?"

Medium · system design and architecture

Approach

1. Choose a partition key and say what query it makes expensive.
2. Fix the scope first: who calls this, how often, and what they do when it fails.
3. State the consistency you need, and where you are willing to be stale.

Follow-up

• What would you drop to keep the system up under load?
• What breaks first when traffic grows ten times?

"What is the key functional difference between useEffect and useLayoutEffect in React?"

Medium

Approach

1. Work from the requirement backwards to the design.
2. State your assumptions explicitly before working the problem.
3. Say what you would check first and why it is the highest-information step.

Follow-up

• What assumption would you test first?
• How would you know your answer was wrong?

Leasing sandboxed job runs without double execution

Medium · leases · fencing · state machines · metering

job-runner leases work from a queue and executes customer workloads in sandboxes with CPU, memory, wall-clock and egress limits. Run durations span 200 ms to 30 minutes, thousands are concurrent, and job_run.status must reach exactly one terminal state among succeeded, failed, timed_out, cancelled and lost. A worker can pause 45 seconds for garbage collection, or be briefly partitioned, after its sandbox has exited but before it writes the outcome. Design the lease duration, the fencing, and timeout ownership, and state how billable_seconds is computed for a run whose worker never returns.

Approach

1. Resolve the lease-duration conflict rather than picking a compromise. A lease shorter than the longest legitimate run re-dispatches work that is still executing; a lease long enough for a 30-minute run leaves a crashed 200 ms run undetected for half an hour. Use a short lease - about 30 seconds - renewed by a heartbeat every 10 seconds while the supervisor is alive, each renewal pushing leased_until to now() + 30 s. Detection latency is then one lease, not one heartbeat: a dead worker's last successful heartbeat landed at most 10 s before it died, so leased_until expires 20 to 30 s after the death, plus whatever interval the reaper scans on. What the 3:1 ratio of lease to heartbeat buys is headroom - one or two missed heartbeats change nothing, and a stall only becomes a re-dispatch once it outlives the lease remaining when it began, between 20 and 30 s here. Both costs follow from those two numbers: heartbeat write load proportional to concurrent runs (3,000 runs at a 10-second interval is about 300 updates/second), and any pause longer than the lease - the 45-second GC - re-dispatching a run that is perfectly healthy.
2. Fence the outcome write so the store, not the worker's memory, arbitrates: UPDATE job_run SET status = $1, finished_at = $2, exit_code = $3 WHERE run_id = $4 AND status = 'running' AND lease_token = $5, and require exactly one row affected. Zero rows means this worker was fenced while it was paused, and its correct behaviour is to discard the result, not to retry - the retry is how a resumed worker overwrites a newer attempt.
3. Make the status machine monotone with a check constraint or a trigger so no terminal row can move, and model a retry as a new row with parent_run_id set and attempt incremented rather than a reset of the old one. That is what keeps how many times did this actually execute answerable afterwards and keeps each attempt's resource usage attributable to itself.
4. Put the wall-clock timeout in the supervisor, never in the workload: a customer-supplied program asked to time itself out will not. Distinguish timed_out, which the supervisor observed and caused, from lost, which nobody observed at all, and leave exit_code null unless the supervisor actually saw the process exit. Recording lost as failed asserts an outcome no one witnessed and then bills and retries on that assertion.
5. Decide billable_seconds for the unobserved case explicitly, because it cannot be derived. It is not now() - started_at, which bills queue and pause time the customer never consumed. The two defensible policies are leaving it null while status = 'lost' and billing nothing, or flooring it at the last heartbeat's observed running time as a stated lower bound; pick one and write it down. Add a per-tenant concurrency cap on sandbox slots so one tenant cannot occupy the whole fleet while these questions are being answered.

Worked solution (30 min)

1. Draw the timeline: last successful heartbeat at t, sandbox exits at t, worker pauses from t to t+45 s, so leased_until = t+30 s and the lease expires there. Mark where the second worker starts and where the first worker's write arrives, then say how short the pause would have had to be to change nothing.
2. Write the fenced terminal update and state the expected rows-affected in both the healthy case and the fenced case.
3. Size the heartbeat: 3,000 concurrent runs at a 10-second interval is about 300 row updates/second on job_run - decide whether that lands on the same table and what it contends with.
4. Write the billable_seconds rule for each terminal status and say which status leaves it null.

Expected result: A short lease of roughly 30 seconds renewed by a 10-second heartbeat, so detection latency is one lease - 20 to 30 seconds after a worker dies, plus the reaper's scan interval - independent of the 200 ms to 30 minute run spread, with one or two missed heartbeats tolerated rather than fatal; a terminal write fenced by AND status = 'running' AND lease_token = $x where zero rows affected means discard; retries as new rows with parent_run_id; a supervisor-owned wall-clock kill that separates timed_out from lost; billable_seconds left null while status = 'lost', or floored at the last heartbeat as a stated policy, never inferred from wall-clock.

Verify it yourself

• In your timeline the fenced write must affect zero rows. If it affects one, the lease alone was doing the work and the token adds nothing.
• At 3,000 concurrent runs and a 10-second heartbeat, check 300 updates/second against the other write load on that table before accepting the interval; if it is the dominant writer, the heartbeat belongs somewhere else. Check the ratio too: at lease = 3 x interval a lost heartbeat is survivable, whereas at lease = interval every scheduling hiccup is a double execution.
• State whether a duplicate execution is billed, then confirm that summing billable_seconds across a parent and its retry matches that stated policy rather than matching it by accident.

Follow-up

• A worker returns from a 45-second GC pause and writes succeeded. Walk through what the database does statement by statement.
• Heartbeats are now a few hundred writes a second against job_run. How do you keep that off the path that matters, and what do you lose by moving it?
• The re-dispatched workload has an external side effect the first attempt already performed. What does your design owe the customer, and what can it not fix?

Invoice detail latency triples after an ORM relationship refactor

Easy · n+1 query · orm · latency regression

An invoice detail endpoint returned in 40 ms at p99 last week. After a refactor replaced a hand-written join with ORM relationship access it returns in 1.4 s, and the regression grows with the number of invoice_line_item rows on the invoice. Database CPU rose, but no statement in the slow-query log exceeds 3 ms. You have request traces with per-span SQL, the ORM statement log, and a staging copy of the data. Produce an ordered diagnostic checklist, the measurement that confirms the cause before any code change, and the fix.

Approach

1. Count statements per request before reading any statement duration. A slow-query log hides this class by construction, because every individual query is fast and only their number is wrong; take one trace and count SQL spans.
2. Establish proportionality rather than asserting it: sample invoices with 5, 20, 60 and 200 line items and plot statements per request against line count. A straight line of slope 1 through an intercept of one or two identifies a lazy relationship load, and no index or cache would move that line.
3. Locate the emitting attribute access in the refactored code and check whether the same shape repeats one level deeper, for instance a tax or adjustment collection hanging off each line, which turns the cost quadratic.
4. Fix with a bounded statement count: either one join that fetches invoice and lines together, or two statements where the second is WHERE invoice_id = $1 AND tenant_id = $2. Keep tenant_id in the predicate so the read stays tenant-scoped even though invoice_id already implies it.
5. Choose between the two deliberately: the join duplicates the wide parent row across N children on the wire, the two-statement form avoids that for one extra round trip. Prefer the join for narrow parents and the split for wide ones.
6. Pin it with a per-request statement-count assertion in a test that varies line count, because a latency assertion passes on a small fixture and would not have caught this.

Follow-up

• The endpoint now also needs per-line tax rows. Show the shape that keeps statement count constant instead of reintroducing the same defect one level down.
• How does this change if a transaction-pooling proxy sits between the service and the database, so each statement may land on a different backend session?
• The same page paginates invoices with LIMIT and OFFSET. Why is that a second, independent defect, and what replaces it?

Suggested practice checklist

For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.

Day 1 — Measure before reasoning

• Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
• Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
• Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.

Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.

Day 2 — References, copies, and the bugs they produce

• Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
• Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
• Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.

Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.

Day 3 — Types, once, in a language that checks them

• Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
• Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
• Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.

Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.

Day 4 — Concurrency, starting with what actually runs at the same time

• Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
• Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
• Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.

Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.

Day 5 — Debugging as a procedure rather than an instinct

• Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
• Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
• Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.

Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.

Day 6 — Tests that catch the bug you are about to write

• Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
• Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
• Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.

Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.

Day 7 — Debug something broken, out loud

• Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
• Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
• Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.

Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.

Review and mentoring: what you let through

Counting review comments or mentees proves nothing. The useful version is a specific change you approved with a reservation you stated, or one you blocked and the delay that cost. Say which standard you were holding and why it was worth the friction. A mentoring story needs the thing the other person can now do without you.

"Why do you want to work at American Express, and what do you know about how Amex generates revenue?"

Medium · behavioural and engineering judgement

Approach

1. Give the blast radius: what could have broken, and what you measured.
2. Name the disagreement and how you resolved it with evidence.
3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

• What would you do differently if you ran that again?
• How did you know your change caused the improvement?

Argue against failing open when the control plane is unreachable

Hard · revocation · cache staleness · fail-open vs fail-closed · influence

The gateway caches credential-to-context decisions with a sixty-second TTL. A design proposal says that when control-plane reads fail, pods should keep serving from expired entries indefinitely so a control-plane outage never becomes a product outage. You believe that converts every revocation into an unbounded one. Describe a design you argued against while it was still a live proposal: what you measured or modelled to make the case, what you conceded, who decided, and what happened afterwards. Say what would have changed your mind before the decision, not after it.

Approach

1. Reframe it from a values argument into a bounded-staleness argument. Both sides already accept the cache; the disagreement is only about the ceiling on how long a revoked credential keeps authorising. Put a number on the table — serve stale for up to fifteen minutes, then fail closed — and make the other side argue against a number rather than against a principle.
2. Bring arithmetic rather than adjectives: the rate of revocations with revoked_reason in ('suspected_leak','auth_version_bump'), the observed distribution of control-plane unavailability, and the product of the two, which is expected requests served by revoked credentials per outage-hour. At 30k requests/second the unbounded version is not a subtle exposure and the number says so.
3. Concede the strong half of the opposing case first, because that is what buys you the room: failing closed turns one service's outage into a total outage across three regions, and a control plane doing tens of writes per second is not engineered to the gateway's availability target. A proposal you have not steelmanned reads as reflex.
4. Propose the asymmetry that usually resolves this: stale entitlements cost bounded money (a quota fifteen minutes out of date over-serves by a computable amount), while a stale revocation costs unbounded access. Split the cached decision by what it authorises, give the two halves different staleness ceilings, and let the entitlement half fail open while the revocation half fails closed.
5. State the propagation dependency plainly, since it is the part that is missed: validity is also derived from the principal's auth_version, so password reset and sign-out-everywhere flow through this same cache. A design that bounds staleness for explicit revocation and not for auth_version bumps has only fixed half of it.
6. Say who decided, and what you did afterwards in either outcome: write the decision down with its number and a review date, and instrument the exposure you were worried about so the next round of the argument is settled by data instead of by seniority.

Follow-up

• Publish-subscribe invalidation is lossy under a partition, and a TTL is the only hard bound. What TTL do you pick, and what does it cost you at 30k requests/second?
• The key was revoked because it was found in a public repository. Does your answer change, and where does that urgency live in the design?
• You lost the argument and six weeks later the failure you predicted happens. What do you say in the review, and what do you not say?

Resolve a review disagreement over a quota check

Easy · code review · isolation levels · write skew · disagreement

A colleague's pull request enforces a per-tenant quota by selecting the current count and then inserting when it is under the limit. You flag it as a race. They reply that the transaction already runs at repeatable read, so the snapshot makes it safe, and the tests pass. Walk through taking that disagreement to a resolution: what you write in the review, what you demonstrate rather than assert, which fix you propose and why, and what you do if they still disagree after all of it.

Approach

1. Answer the claim precisely instead of restating your objection, because they have made a specific technical argument. In PostgreSQL, repeatable read is snapshot isolation; this is write skew, which snapshot isolation permits by design. Both transactions read a count that is stable within their own snapshot, insert disjoint rows that the other cannot see, and both commit, so the limit is exceeded by exactly the concurrency.
2. Demonstrate rather than cite. Two psql sessions, both BEGIN ISOLATION LEVEL REPEATABLE READ, both select the count, both insert, both commit: it succeeds. Repeat at SERIALIZABLE and the second commit fails with serialization_failure, SQLSTATE 40001. That takes two minutes, ends the argument without anyone conceding a position, and leaves an artefact for the next reviewer.
3. Offer the options with their costs rather than a verdict. Serialisable plus a retry loop on 40001 is correct but obliges every caller to retry and degrades under contention. An increment-and-compare on a counter row — update tenant_quota set used = used + 1 where tenant_id = $1 and used < limit returning used — is safe even at read committed, because a blocked updater re-evaluates the WHERE clause against the row version it finally locks, and zero rows returned means full. A unique or exclusion constraint that makes the surplus write fail is the third.
4. Name the plausible non-fix explicitly, since it is what usually gets merged instead: folding the count into the insert as insert ... select ... where (select count(*) ...) < limit is still racy under read committed, because the subquery cannot see the other transaction's uncommitted rows. It looks atomic and is not.
5. Say what you do if they still disagree: escalate the decision rather than the disagreement. Attach the reproduction, hand it to the service owner or a third reviewer, and state that you will not block the merge if the owner accepts the risk knowingly — and that you want that acceptance written down.
6. Close with the general lesson worth leaving in the review thread: a passing suite is weak evidence for a concurrency claim because it runs one request at a time. Ask for a test that runs two.

Follow-up

• Write the counter-row version. Does your answer change if the quota counts child rows rather than a column?
• Under serialisable, who performs the retry, and what does the API client see if the retry also fails?
• This is the third disagreement with the same reviewer this month. What changes in how you review?