Amazon Web Services Machine Learning Engineer Interview Questions & Guide 2026

The 60-second brief

As a Machine Learning Engineer at Amazon Web Services, you operate at the frontier of cloud computing and artificial intelligence. You are responsible for architecting, training, deploying, and optimizing large-scale machine learning systems that power global enterprise workloads. Whether you are building low-level hardware acceleration kernels for custom silicon like AWS Trainium and AWS Inferentia via Annapurna Labs, developing the AWS Neuron SDK, or engineering enterprise-grade custom Retrieval-Augmented Generation (RAG) services, your work directly defines the efficiency and capability of cloud-native AI.

The shape of the workload matters more for your prep than the industry label does. Read-heavy serving, write-heavy ingestion and scheduled batch processing have different binding constraints and fail in different places, so find out which one the team lives in before picking design topics.

Focus areas

• Choose indexes from the query's access path
• Detect concurrent edits instead of losing writes
• Make every write idempotent under retry

Overview

This role sits at the intersection of deep learning theory, distributed software engineering, and hardware-software co-design. At Amazon Web Services, machine learning is not an isolated experiment; it is integrated directly into hyperscale infrastructure. A single optimization in an ML compiler or distributed training framework can reduce compute latencies and operational costs across thousands of data centers globally, directly serving millions of developers and enterprise clients.

The expectations for a Machine Learning Engineer at AWS are exceptionally high, but so is the scope of impact. You will collaborate with elite cross-functional teams of scientists, kernel developers, hardware architects, and cloud infrastructure engineers. Success in this role requires strong foundational software engineering skills, deep understanding of distributed ML architectures, and a relentless focus on customer obsession and operational simplicity.

The Amazon Web Services interview process

Amazon Web Services candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Online Assessment

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

• In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
• In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
• Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

• Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
• Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
• For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Technical Phone Screen

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

• Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
• Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
• Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

• Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
• Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
• Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Onsite Loop

Nobody in the room with you decides this. Interviewers typically write their rounds up separately, often before seeing anyone else's, and the outcome is settled later from those write-ups. A split panel gets resolved by whichever note carries specific evidence, so what you want out of each room is one concrete thing that person could write down: a bug you caught yourself, a trade-off you named, a decision you owned. The rest is arithmetic. The project you describe in a behavioural conversation is often the same system you sketched an hour earlier, and the two accounts have to agree.

What to demonstrate

• Whether the scale, team size and timeline you attach to a project hold steady when that project resurfaces in a different round
• Whether each interviewer leaves with a specific thing to cite rather than a general impression of competence
• Whether a trade-off you defended in one round survives a challenge in another, instead of being quietly swapped for the answer the new interviewer seemed to want
• Whether a question you have already answered earlier in the day gets the same answer at the same depth, without visible impatience

How to prepare

• Write a one-page sheet per project fixing the figures you will quote — request volume, data size, team size, elapsed time, what broke — and say them aloud from the sheet until they come out identical every time
• For each round on the schedule, decide in advance the one sentence you want in that person's notes, then check in a mock that you said it outright instead of leaving it to be inferred
• Have someone ask you the same project question twice, an hour apart, and diff the two answers for numbers that moved or a trade-off that reversed

Bar Raiser

Because the format is not fixed, the first job in the room is classification. Listen to the opening question and decide what it is: a probe into work you have already described, a fresh problem to solve now, or a conversation about how you operate. Each wants a different register, and the common failure is forcing a rehearsed structure onto a question that did not ask for it. Running a full design ritual on a ten-minute debugging question reads as not listening. When you cannot tell which it is, ask how long they want to spend and answer at that depth.

What to demonstrate

• Whether the shape of your answer matches the question, so a yes-or-no gets answered before it is justified and an open prompt gets a direction before a detour
• Whether you check how much depth is wanted instead of deciding for them, and whether you stop when the answer is complete rather than continuing until someone interrupts
• Whether you can be redirected in the middle of an answer without restarting it from the beginning
• Whether a question outside your experience gets an honest boundary followed by reasoning from what you do know, instead of a confident answer with nothing behind it

How to prepare

• Rehearse one project at three lengths, roughly thirty seconds, three minutes, and a full walkthrough at the depth of a design review, and practise switching between them when someone interrupts mid-telling
• Have someone ask you five questions of deliberately mixed type in one sitting without telling you the types, and score only whether you identified each one correctly before you started answering
• Draft the sentence you will use to check depth, along the lines of asking whether the short version is useful here or they want the detail, and use it in a real conversation this week so the day of the round is not its first outing

Where candidates lose points

Paginating with LIMIT/OFFSET over a set that changes while the client is reading it

OFFSET n makes the database produce and discard n rows before returning anything, so the cost of a page grows with its depth rather than with its size and page 500 costs five hundred pages of work. The correctness problem is worse than the cost: if a row is inserted or reordered between two page fetches, rows shift across the offset boundary and are either skipped entirely or returned twice, and neither outcome leaves any trace in the response for the client to detect. Keyset pagination - WHERE (sort_key, id) < ($last_sort_key, $last_id) ORDER BY sort_key DESC, id DESC LIMIT n, backed by an index in exactly that order - reads only the rows it returns and is stable against concurrent inserts. It requires the tie-break column: a timestamp is not unique, and duplicate sort keys straddling a page boundary reintroduce the skip it was adopted to remove.

Shipping a migration and the code that depends on it as a single change

During any rolling deploy, and for as long as a rollback remains possible, old and new code execute against the same schema at the same time. A migration that drops or renames a column breaks every instance that has not restarted yet, and code that requires a column the migration has not applied breaks every instance that restarted early. The discipline is expand then contract: add the new column nullable, write both shapes, backfill in batches, move reads across once the backfill is verified, and only then stop writing the old shape and drop it - four deploys, usually spread over days. It feels disproportionate until the first rollback, at which point it is the only reason the previous version still runs.

Sharing mutable state with no stated owner

Say which thread, request or task owns each mutable structure, and what protects it when the answer is more than one: a lock, a queue that hands ownership across, or an immutable copy per reader. A structure documented as safe for concurrent reads is usually not safe for a concurrent write alongside those reads.

Assuming the bug is in the framework

Suspect your own code first: read the stack trace top to bottom, check which versions are actually installed rather than which ones you believe are, and reproduce in isolation before blaming a library that thousands of people run daily. When the fault really is upstream, you need that minimal reproduction to say so credibly anyway.

Questions & practice

Discuss your prior machine learning research experience, explaining key architectural trade-offs, theoretical assumptions, and practical deployment bottlenecks.

Medium · machine learning fundamentals

Approach

1. Name the simplest model that could work and what would make you move past it.
2. Say how you would validate it, and where leakage could enter the split.
3. Pick the metric from the cost of each error type, not from habit.

Follow-up

• How would you know the model is overfitting?
• Where could label leakage enter this setup?

Apply greedy algorithms to optimize job scheduling across heterogeneous cloud compute clusters.

Medium · machine learning fundamentals

Approach

1. State the learning problem: the label, the unit of prediction and how the model is used.
2. Pick the metric from the cost of each error type, not from habit.
3. Name the simplest model that could work and what would make you move past it.

Follow-up

• Where could label leakage enter this setup?
• How would you know the model is overfitting?

Explain the internal communication primitives (e.g., All-Reduce, All-Gather) used in multi-node ML cluster topologies.

Medium · machine learning fundamentals

Approach

1. Say how you would validate it, and where leakage could enter the split.
2. State the learning problem: the label, the unit of prediction and how the model is used.
3. Name the simplest model that could work and what would make you move past it.

Follow-up

• How would you know the model is overfitting?
• What changes if the classes are heavily imbalanced?

How do you address numerical precision loss or gradient vanishing when scaling deep neural networks across custom hardware platforms?

Medium · machine learning fundamentals

Approach

1. Pick the metric from the cost of each error type, not from habit.
2. State the learning problem: the label, the unit of prediction and how the model is used.
3. Name the simplest model that could work and what would make you move past it.

Follow-up

• What changes if the classes are heavily imbalanced?
• Where could label leakage enter this setup?

Traversal a complex dependency network or network topology using Breadth-First Search (BFS) to identify shortest paths or bottlenecks.

Medium · coding and algorithms

Approach

1. Restate the input: its shape, its size, and what is guaranteed about it.
2. Walk one small example through your approach before writing the whole thing.
3. State the target complexity and say which constraint rules the naive version out.

Follow-up

• Which test case would catch an off-by-one here?
• What is the worst case, and how likely is it on real data?

Implement an efficient solution using the sliding window pattern to compute rolling performance metrics on real-time data streams.

Medium · coding and algorithms

Approach

1. Name the brute-force solution and its complexity before improving on it.
2. Choose the data structure from the access pattern, not from familiarity.
3. State the target complexity and say which constraint rules the naive version out.

Follow-up

• Which test case would catch an off-by-one here?
• What is the worst case, and how likely is it on real data?

Track a rolling failure rate per destination for circuit decisions

Easy · sliding window · ring buffer · circuit breaker

The egress service delivers about 1,500 webhooks per second across roughly 40,000 destinations, each call bounded by a 10 second timeout. Maintain, per destination, the failure rate over the trailing 60 seconds so a caller can ask before dispatch whether the circuit should open. Attempts arrive as (destination_id, finished_at_ms, outcome). Requirement: amortised O(1) per attempt, with total memory bounded by the destination count rather than by traffic. Give the structure, its exact memory, and the rule that stops a destination with three attempts from opening a circuit.

Approach

1. Name the exact-deque version and then reject it as the default. Holding timestamps and advancing a tail pointer past anything older than now minus 60 seconds is a correct two-pointer window at amortised O(1) per attempt, but its memory tracks in-window traffic, so one destination in a retry storm holds hundreds of thousands of entries while thousands of quiet destinations hold none.
2. Use a ring of 60 one-second buckets per destination, each bucket a pair of counters for attempts and failures. On an attempt, advance the ring by the elapsed whole seconds, zeroing at most min(elapsed, 60) buckets, then increment the head. That is amortised O(1) with a fixed footprint per destination.
3. State the footprint: 60 buckets times two 4-byte counters is 480 bytes of payload per destination, so 40,000 destinations is roughly 20 to 25 MB with per-entry overhead, bounded by the catalogue rather than by the rate. The cost is granularity, since the oldest bucket ages out in whole seconds, which is far tighter than the decision needs.
4. Require a minimum sample before the circuit may open. A destination with three attempts and three failures reads as 100 percent and is not evidence; a floor of roughly 20 attempts in the window makes the ratio meaningful, and below that floor use a run of consecutive failures as the trigger instead.
5. Expire idle destinations, or memory grows with every destination ever seen rather than with the live set. Hold the rings in a bounded LRU keyed on destination_id and treat a miss as no history, which is the correct default for an endpoint that has been silent for a minute.
6. Keep the half-open probe out of the window arithmetic. After the circuit opens, one probe per interval decides whether to close it, and folding that single success into a window that still holds a 100 percent failure history would reopen the destination on one data point.

Worked solution (20 min)

1. Define the bucket struct and the advance step: take floor(finished_at_ms / 1000), compare with the ring's current second, zero min(delta, 60) buckets forward, then write into the new head.
2. Trace a destination that receives 5 attempts, goes silent for 90 seconds, then receives one more, and confirm the rate is computed from one attempt rather than six.
3. Compute total memory for 40,000 destinations at 60 buckets of two 4-byte counters, and state what changes if the window widens to 300 seconds.
4. Write the open rule as a single predicate combining the minimum-attempt floor with the rate threshold.

Expected result: A per-destination ring of 60 one-second (attempts, failures) buckets advanced lazily for amortised O(1) cost, roughly 480 bytes of counters per destination and about 20 to 25 MB for 40,000 of them, an LRU bound on live rings, and an open rule requiring both a minimum attempt count in the window and a rate above threshold.

Verify it yourself

• After 60 seconds of silence a destination's window holds zero attempts and cannot open a circuit.
• Memory measured with one hot destination and 39,999 idle ones is indistinguishable from the reverse, which the deque version would fail.
• Replaying the same attempt stream with the clock shifted by an hour yields the same final rate.

Follow-up

• The fleet is 30 instances and each sees roughly a thirtieth of a destination's traffic. Where does the rate actually live, and what does a per-instance answer get wrong?
• A destination answers in 9.5 seconds and succeeds. It is not failing but it is consuming your per-destination concurrency. What signal should open the circuit here?
• How would you make the window survive a process restart, and is it worth the cost?

Hold a per-tenant active cap against concurrent creates

Hard · write skew · isolation levels · constraints

A tenant on the standard plan may hold at most 50 resources with status='active'. The create handler runs SELECT count(*) FROM resource WHERE tenant_id = $1 AND status = 'active', compares to 50, then inserts. Two creates arrive 3 ms apart on different instances and the tenant lands at 51. Name the anomaly, say whether PostgreSQL 16 READ COMMITTED or REPEATABLE READ prevents it and why, then give an implementation that holds the cap at READ COMMITTED with the exact statements. Finally, say what changes when the cap is 'at most one running export per tenant' on job_run.

Approach

1. Name it: write skew. The two transactions read an overlapping set and write disjoint rows, so there is no row-level conflict for the engine to detect and each commit is individually legal.
2. Rule out the levels precisely. READ COMMITTED takes a fresh snapshot per statement and takes no lock on the counted rows, so both see 49. PostgreSQL's REPEATABLE READ is snapshot isolation: it removes non-repeatable reads and phantoms within the snapshot but still admits write skew, because the anomaly is not a re-read of a changed row, it is a read of a set that a concurrent transaction invalidates. Only SERIALIZABLE closes it, by tracking the read dependency and aborting one transaction with SQLSTATE 40001 — a guarantee that exists only if the application re-runs the whole transaction from the read.
3. Convert the set predicate into a single-row conflict: keep tenant.active_resource_count and run UPDATE tenant SET active_resource_count = active_resource_count + 1 WHERE tenant_id = $1 AND active_resource_count < 50 in the same transaction as the INSERT. Zero affected rows is the cap, returned as 409. The row lock serialises the decision at any isolation level, and contention is bounded to one tenant's row — which is also the fair-scheduling unit, unlike a global counter that would convoy every tenant behind one row.
4. State the cost you just took on: a counter is a second source of truth that can drift, so every path that changes status must adjust it inside the same transaction, and a periodic reconciliation has to exist, with resource_revision as the authority for what the count should have been.
5. For the job case the invariant is expressible per row, so let the database hold it: a partial unique index on job_run (tenant_id, job_type) WHERE status IN ('queued','running') makes a second running export unwritable and the loser takes 23505, mapped to 409. That is strictly better than a counter — no drift, no reconciliation — and it is available only because the cap is one rather than fifty.
6. Add the retry discipline each route demands: under SERIALIZABLE both 40001 and deadlock 40P01 are retryable and the retry must re-execute the read, while under READ COMMITTED with the counter nothing retries, because the conflict is reported to the caller rather than raised as an error.

Worked solution (35 min)

1. Reproduce with two sessions that both count 49, both insert and both commit, at READ COMMITTED and then at REPEATABLE READ; record the final active count for each.
2. Repeat both sessions at SERIALIZABLE and record which SQLSTATE the loser receives and at which statement it is raised.
3. Implement the counter form and run a 20-way concurrent create against a tenant sitting at 45 active resources.
4. Implement the partial unique index for the job case and race 20 enqueues of the same export.

Expected result: READ COMMITTED and REPEATABLE READ both finish at 51. SERIALIZABLE finishes at 50 with one transaction aborted at 40001 — usually at commit, though SSI may raise it at an earlier statement. The counter form finishes at exactly 50: of the twenty creates, the five that take the tenant from 45 to 50 return 201 and the remaining fifteen match zero rows and return 409, with no error raised. The job index admits one running export and rejects the rest with 23505.

Verify it yourself

• count(*) WHERE status='active' never exceeds the cap across 50 randomised concurrent runs.
• Remove the active_resource_count < 50 predicate and the same test must start failing; if it still passes, the race is not being exercised.
• At SERIALIZABLE, confirm every participating session is SERIALIZABLE — one session at a lower level and the guarantee is gone.
• After each run, the counter equals the real count of active rows.

Follow-up

• A resource moves from archived back to active. Which statements change, and what breaks if the counter update and the status change land in different transactions?
• The cap becomes plan-dependent and a plan can change mid-month. Where does the number 50 live, and who reads it?
• How do you detect after the fact that the counter drifted, without locking the table?

Replace offset paging on the resource feed with keyset

Easy · keyset pagination · composite index · sort order

resource holds resource_id, tenant_id, owner_user_id, title, body_ref, version, status ('draft','active','archived','deleted'), created_at, updated_at, deleted_at, with an index on (tenant_id, status, updated_at DESC, resource_id DESC). The listing endpoint returns active resources for one tenant, newest update first, 50 per page, today with LIMIT 50 OFFSET n. Tenants reach page 400 and rows are created while they read. Write the keyset query, define what the cursor carries and how it is encoded, and say which part of the index each predicate uses. Assume PostgreSQL 16.

Approach

1. Name the two failures separately. OFFSET 20000 makes the server produce and discard 20,000 rows, so page cost grows with depth rather than with page size. Independently, any write that changes how many rows sort above the offset moves the window between two fetches, and the direction decides which anomaly you get: an insert lands at the head of updated_at DESC and pushes already-returned rows down past the boundary, so they are returned a second time; a delete above the offset, or a row whose updated_at is bumped above the cursor, pulls rows up and one is never returned at all. Nothing in the response reveals either.
2. Write the seek: WHERE tenant_id = $1 AND status = 'active' AND (updated_at, resource_id) < ($2, $3) ORDER BY updated_at DESC, resource_id DESC LIMIT 50. The row-value comparison is one index range rather than a disjunction, and both columns are NOT NULL, which is what makes that comparison well defined.
3. Map each predicate onto the index: tenant_id and status are equality on the leading columns, (updated_at, resource_id) is the range, and the ORDER BY matches the index order so no Sort node appears and the scan stops after 50 rows. The DESC in the definition only matters for mixed directions — a plain ascending btree on the same columns is read backwards for this query.
4. Put both sort columns in the cursor and nothing the client can tamper with into another tenant: base64 of (updated_at, resource_id), validated server-side, with tenant_id taken from the principal.
5. State the residual honestly. Keyset is stable against concurrent inserts and deletes, but not against a row whose updated_at changes mid-scroll — that row moves in the ordering and can be seen twice. If the feed must be a snapshot, order by an immutable key or bound the page set with updated_at <= the cursor's start value.
6. Keep a total out of the page path. A tenant-wide COUNT(*) is the scan keyset just removed; fetch LIMIT 51 and return has_more instead.

Follow-up

• The client asks for 'jump to page 400'. What do you offer instead, and what does the honest version cost?
• Sort order becomes user-selectable across four columns. How many indexes is that, and which would you refuse to add?
• What does the cursor do when the row it points at has since been deleted?

Design a high-throughput, low-latency model inference service on AWS Inferentia accelerators to process streaming sensor data.

Medium · ml system design and production

Approach

1. Name what you would monitor after launch and what triggers a retrain.
2. Fix the product goal and the online metric before choosing any model.
3. Separate the offline training path from the online serving path.

Follow-up

• How would you detect drift before the metric drops?
• How would you roll the new model out safely?

Architect a distributed monitoring framework to detect training pipeline data drift and numerical instability across thousands of compute nodes.

Medium · ml system design and production

Approach

1. Fix the product goal and the online metric before choosing any model.
2. Name what you would monitor after launch and what triggers a retrain.
3. Say where features come from at serving time and how they match training.

Follow-up

• How would you detect drift before the metric drops?
• How would you roll the new model out safely?

Architect an enterprise custom RAG (Retrieval-Augmented Generation) service capable of real-time multi-modal encoding and strict tenant data isolation.

Medium · system design

Approach

1. Fix the scope first: who calls this, how often, and what they do when it fails.
2. Name the failure you are designing for, then the recovery path.
3. State the consistency you need, and where you are willing to be stale.

Follow-up

• How does this behave when that dependency is down for an hour?
• What breaks first when traffic grows ten times?

Make the resource write endpoint safe for retrying clients

Easy · idempotency · unique constraints · retries

The edge API accepts POST /v1/resources at 1.2k writes/second peak under a 400 ms p99 budget, and clients retry on timeout. idempotency_key has PRIMARY KEY (tenant_id, idempotency_key) plus request_fingerprint CHAR(64), state in ('in_flight','succeeded','failed'), response_status, response_body, resource_id, locked_until and expires_at. Specify the exact sequence the handler runs: what the second request does when it arrives while the first is still executing, what a retry carrying a different body receives, what happens when a process dies mid-request, and how the table is kept from growing without bound.

Approach

1. Start from the race rather than the happy path. Insert the key row first, in its own short transaction, with state 'in_flight' and locked_until set to now() plus the request deadline. The unique constraint on (tenant_id, idempotency_key) is the arbitration mechanism: exactly one caller commits the insert, and every other caller takes a unique violation (SQLSTATE 23505) and becomes a follower. Reading the table and then inserting is a check-then-act race that both callers pass, and they pass it most often under the load that generates the retries.
2. Give the follower a branch for every state it can observe, with no fall-through to doing the work: fingerprint mismatch means the key is being reused for a different request, so reject with 422 and never replay the stored response; 'succeeded' replays response_status and response_body verbatim; 'failed' allows a fresh attempt; 'in_flight' with locked_until in the future is answered with 409 and a retry hint, or a short bounded wait, because the only alternative is executing the effect twice.
3. Commit the effect and the record of the effect together. The resource insert, its resource_revision row, its outbox_event row and the transition of the key row to 'succeeded' all happen in one transaction, so no crash can leave the work done and the key still 'in_flight'. The earlier insert is deliberately a separate transaction: it has to be visible to a concurrent caller before the work starts, which an uncommitted row is not.
4. Close the wedged-key path. A process that dies after the insert leaves a row that would otherwise block that key forever, which is what locked_until exists for. Reclaim with a single conditional statement, UPDATE ... SET state='in_flight', locked_until=now()+interval WHERE ... AND state='in_flight' AND locked_until < now(), and treat a zero rowcount as losing the reclaim. Two statements that read and then update reintroduce the original race one layer down.
5. Size the retention before it becomes an incident. At the 1.2k/second peak a 24-hour window is on the order of 100 million rows, and the table grows with traffic rather than with data, so the sweep is load-bearing. Batched deletes driven by an index on expires_at keep it bounded. Range-partitioning by day and dropping whole partitions is cheaper, but PostgreSQL requires every partition-key column to appear in the primary key, so the key becomes (tenant_id, idempotency_key, created_at) and a retry that straddles the boundary no longer collides with its original. State that cost rather than meeting it later.

Worked solution (20 min)

1. Draw a timeline with the retry arriving 150 ms into a 900 ms first attempt, and mark the instant each row becomes visible to the other transaction.
2. Write the insert-first statement, then a five-row branch table for the follower: fingerprint mismatch, 'succeeded', 'failed', live 'in_flight', expired 'in_flight' - one action each.
3. Write the single conditional UPDATE that reclaims an expired in_flight row, and argue why two concurrent reclaimers cannot both see a non-zero rowcount.
4. Compute rows per day at peak, name the index the sweep uses, and check the sweep rate against the insert rate.

Expected result: The key row is inserted at 'in_flight' in its own transaction before any work; the unique violation identifies the follower. The follower replays the stored response on 'succeeded', returns 422 on a fingerprint mismatch, and waits or returns 409 on a live 'in_flight', never performing the work itself. The effect, its revision row, its outbox row and the transition to 'succeeded' commit in one transaction. Expired in_flight rows are reclaimed by one conditional UPDATE, and expired keys are swept in batches.

Verify it yourself

• The retry creates no second resource row, whether it arrives during the first attempt or after it.
• A retry with the same key and a changed body is rejected rather than served the first attempt's response.
• Killing the process between the key insert and the commit leaves a key that is reclaimable within locked_until rather than one that is stuck.
• For create operations, every key row in state 'succeeded' carries a resource_id that exists.

Follow-up

• The stored response body averages 200 KB and this table is now the largest in the database. What do you store instead, and what does a replay return once the body has been pruned?
• A client library generates a fresh idempotency key on every attempt. What breaks, which layer should have caught it, and does the server have any defence?
• The first attempt succeeded but its response was lost, and the client retries 30 hours later, after the key expired. What does the second attempt do, and is that acceptable?

Edge instances grow 400 MB per hour until the nightly restart

Hard · memory leak · heap analysis · metric cardinality · gc

Edge API instances start at 700 MB resident and grow about 400 MB/hour; a nightly rolling restart has hidden it for weeks. Growth continues unchanged when request rate halves overnight, p99 degrades in the last hours before an instance is recycled, and heap used immediately after a forced full GC rises monotonically. The service holds no product state. Name the discriminating measurement that separates the plausible causes, give the most likely cause, and give the fix and how you would verify it.

Approach

1. Separate resident memory from live heap first, because they fail differently. Resident size can grow from fragmentation, native buffers or thread stacks while the heap is flat; heap used after a full GC rising monotonically is the measurement that says objects are reachable and not being released. You already have it, so this is retention, not fragmentation, and that closes off half the candidate list.
2. Use the rate's independence from traffic as the discriminator. Growth that continues at half the request rate rules out per-request objects that are merely slow to collect and points at a structure that grows with distinct values observed rather than with call volume. Write the candidates that have that property: a metrics registry keyed on a high-cardinality label, an unevicted cache, an interner, a per-key lock map.
3. Take two heap snapshots an hour apart and diff by retained size, reading the dominator tree, not by allocation count or instance count. Expect one root holding a map with millions of entries, then follow the reference chain to the code that inserts and never removes. Allocation profilers point at churn, which is the wrong signal here.
4. The candidate that fits this service is an observability label carrying an identifier, such as a request path recorded before templating so that /v1/resources/48213 becomes its own metric series. That grows with distinct ids seen, is independent of rate, and explains the late p99 degradation, since GC cost rises with the size of the live set.
5. Fix by bounding cardinality at the source: template the path to /v1/resources/{id} before it becomes a label, move tenant id from a label to a log field or an exemplar, and cap the registry with a bounded map that evicts. Add a cardinality ceiling that fails loudly in a lower environment rather than growing quietly in production.
6. Verify with a soak rather than a restart. Hold one instance out of the nightly recycle for 48 hours with the fix and compare post-GC heap and series count against an unfixed control taking the same traffic.

Follow-up

• Post-GC heap is now flat but resident size still creeps. What are you looking at, and does it matter?
• How would you have detected this before an OOM, given the nightly restart masked the trend?
• That label is what makes one dashboard useful. How do you keep the dashboard and lose the leak?

Suggested practice checklist

For someone who has spent the last few years shipping features and reading other people's code, and who has not solved a timed problem from a blank file in a long time. Five days rebuild the primitives and the patterns that sit on them, working from invariants rather than remembered solutions, and the last two attach that back to the rest of the loop.

Day 1 — Rebuild the primitives by implementing them

• Implement a dynamic array with doubling growth and an operation counter, then change the growth rule to add a fixed sixteen slots instead, and time both for n of ten thousand, a hundred thousand and a million. The fixed-increment version resizes n/16 times at O(n) each, so its total work is quadratic; doubling is what makes append amortised constant.
• Implement a hash map with separate chaining and a load-factor resize, then insert ten thousand keys engineered to land in one bucket and record what happens to lookup time, so that average-case O(1) becomes a claim with a stated precondition rather than a reflex.
• For dynamic-array append and hash-map insert, write down which cost is amortised rather than worst-case, which single operation pays the whole bill, and what a system with a hard per-operation deadline would have to do instead.

Deliverable: Two working implementations plus a timing table showing the input at which each structure's advertised complexity stops holding.

Day 2 — Arrays under an invariant: two pointers, sliding window, binary search

• Solve longest-subarray-with-sum-at-most-K using a sliding window, then run it on an input containing negative numbers and watch it return the wrong answer: extending the window only moves the sum monotonically when every element is non-negative, and that precondition is the whole reason the technique works.
• Write the binary search that finds the first index satisfying a predicate rather than an exact value, put the loop invariant above the loop in a comment, and verify termination on the two inputs that break careless versions: the empty range, and a range where every element satisfies the predicate.
• Compute the midpoint as lo + (hi - lo) / 2 and write one line on why the obvious (lo + hi) / 2 is a genuine defect in a fixed-width integer type and a non-issue in a language with arbitrary-precision integers.

Deliverable: Three solved problems, each with its invariant written above the loop, plus one recorded input on which the sliding window is provably wrong.

Day 3 — Sorting, heaps, and the greedy argument that has to be proved

• Solve one top-k problem three ways, by full sort, by a size-k heap, and by quickselect, then write the values of n and k at which each becomes the right choice, along with quickselect's quadratic worst case and why a randomised pivot makes that unlikely rather than impossible.
• Implement bottom-up heapify and count sift-down steps to confirm it does linear work rather than n log n, because most nodes sit near the bottom of the tree and therefore move only a short distance.
• Take interval scheduling by earliest finishing time and write the exchange argument out in full: given any optimal schedule, swapping in the earliest-finishing interval keeps it feasible and no smaller. Then construct the weighted variant where that same greedy fails and name what has to replace it.

Deliverable: A three-way top-k comparison with measured crossover points, one written exchange argument, and one counterexample to a greedy rule that looks almost identical.

Day 4 — Recursion, memoisation, and the step to a table

• Take one problem with overlapping subproblems, such as edit distance or coin change, instrument the plain recursion with a call counter to show the blow-up, then add memoisation and re-count.
• Convert the memoised version to a bottom-up table and state the two properties you relied on: each subproblem's result depends only on its arguments, and the dependencies form a DAG you can enumerate in order.
• Rewrite one deep recursion with an explicit stack, then find the input length at which the original hits the interpreter's frame limit, which defaults to about a thousand frames in CPython, so you know when the rewrite is required rather than decorative.

Deliverable: One problem in three forms, naive, memoised and tabulated, with call counts for each and the input length at which recursion depth becomes the binding constraint.

Day 5 — Graphs, where most of the work is choosing the traversal

• Implement BFS and DFS over one adjacency list, then answer for each which finds a shortest path in an unweighted graph and which you would use to detect a cycle in a directed graph, including why the in-progress versus finished distinction matters for the second.
• Implement topological sort by in-degree, feed it a graph containing a cycle, and confirm the failure signature is that fewer than V nodes come out rather than an exception, then note that the order it produces is one of several valid ones.
• Run a shortest-path search on a graph with a single negative edge weight and show the wrong answer, then write the precondition Dijkstra actually needs, non-negative weights, because it finalises a node's distance the first time that node is popped, and name the algorithm you would switch to and its own limit.

Deliverable: A small graph library with BFS, DFS and topological sort, plus two inputs that produce documented wrong answers under the wrong algorithm choice.

Day 6 — One day for everything that is not an algorithm

• Sketch one system only to the depth a coding-heavy loop tends to reach: the endpoints, what the service stores, and the single query pattern that decides the schema. Stop at twenty-five minutes.
• Prepare the project answer for an interviewer who codes, which means rehearsing the two levels they push to: the specific thing you built, and why you chose that approach over the alternative they will name. Open with a number and be ready to say what it excludes.
• Prepare the answer to what you would do differently, choosing a real technical mistake with a specific fix rather than a complaint about process or staffing.

Deliverable: One design sketch at endpoint-and-schema depth, plus a project answer rehearsed to two levels of follow-up.

Day 7 — Solve out loud, under time

• Do three timed problems at twenty-five minutes each in a plain editor with no autocomplete and no execution until the end, then tally separately the failures that were syntax and the ones that were approach, because those two numbers call for different fixes.
• Narrate one solution from the first sentence, stating the approach and its complexity before writing any code, and rehearse the sentence you will use when you realise mid-solution that the approach is wrong.
• Re-solve from blank the two problems you were slowest on this week and compare the times against the day they first appeared.

Deliverable: A recording of one fully narrated solution and a tally that separates syntax failures from approach failures.

Disagreement needs an outcome you did not choose

Conflict answers where you were right and everyone came round are the weakest ones. Stronger: the evidence you went and collected, what would have changed your mind, and what you did in the weeks after the call went against you. Implementing a design you argued against, properly, is a specific and checkable behaviour.

How do you balance the drive for continuous innovation with the operational need for high system reliability and low maintenance overhead?

Medium · behavioural and collaboration

Approach

1. State the situation in two sentences and spend the rest on the reasoning.
2. Close with what you would do differently, concretely.
3. Give the blast radius: what could have broken, and what you measured.

Follow-up

• How did you know your change caused the improvement?
• What would you do differently if you ran that again?

Reverse your own decision and price the reversal

Medium · reversibility · measurement · migrations

Describe a technical decision you made and later reversed. Pick one that cost something: a service you split and merged back, a cache you added and removed, an index you created that pushed the planner onto a worse plan, a projection you rebuilt from scratch. State what you believed when you decided, the measurement that changed your mind, how long the wrong version ran in production, and what the reversal cost in migrations, dual writes, and a deprecation window for callers you did not own.

Approach

1. State the original rationale without irony, in the version you would still defend given what was known then. If it is not defensible, the story is about carelessness rather than judgement, and a different example serves you better.
2. Give the measurement that moved with a before and after: the p99 that did not improve, the cache hit rate that sat at 40%, the plan that flipped to a sequential scan once the table passed a size you can name.
3. Cost the reversal in steps, not adjectives: expand-and-contract deploys, the dual-write window, the callers who had to be notified, the rows already written in the wrong shape that had to be backfilled or abandoned.
4. Distinguish reversal from rewrite by naming what you kept. Most good reversals preserve the schema or the interface and undo one decision inside it, which is also why they were affordable.
5. Finish on the process change: the smallest experiment that would have produced the same measurement in a day, and why you did not run it the first time.

Follow-up

• What in that decision was irreversible, and did you know it was irreversible when you made it?
• How did you tell the people who had already built on top of the original decision?
• What do you now measure before committing to a change of this size?

Turn a code review disagreement into a decision

Easy · code review · optimistic concurrency · communication

A colleague's change updates a row with UPDATE resource SET version = version + 1 WHERE resource_id = $1 AND version = $2 and treats an affected-row count of zero as a successful no-op. You read that as a silently lost update; they think returning 200 is friendlier to clients than returning a conflict. Describe how you have handled a review disagreement of this shape: what goes in the comment, when you leave the thread, and who decides. Then write the comment you would leave here, in under 80 words.

Approach

1. Sort the disagreement before writing anything. A silently discarded write is a correctness claim about data; the choice between 409 and 412 is taste. Only the first justifies blocking a merge, and saying which one you are doing is most of the value of the comment.
2. Make the claim reproducible in the comment itself with an interleaving rather than a principle: A reads version 7, B reads version 7, B commits version 8, A's predicate matches zero rows, A is told it succeeded and A's edit is gone.
3. Offer the alternative with its cost attached: return 409 carrying the current version and the revision that won, so the client can re-read and re-apply. Note that automatic retry is not the fix, because a retry re-reads the winner's state and reapplies an intent formed against data that no longer exists.
4. Apply an escalation rule you can state: two round trips on the thread, then a call, and the service's owner decides rather than the reviewer. A reviewer who cannot be overruled is a bottleneck with extra steps.
5. Close in writing wherever the decision lands, so the next reader finds the reasoning in the code or the ticket instead of in a collapsed review thread.

Follow-up

• Where would you put the test that fails if someone reintroduces the swallowed zero rowcount?
• The author says clients cannot handle a 409. How do you check whether that is true?
• How do you handle the same review comment when the author is more senior than you and in a hurry?