Interview guides · Cruise

Cruise Machine Learning Engineer Interview Questions & Guide 2026

Cruise Machine Learning Engineer interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.

The 60-second brief

As a Machine Learning Engineer at Cruise, you are at the forefront of autonomous vehicle technology, building intelligent systems that perceive the world, predict agent behaviors, and make safe driving decisions in complex urban environments. Your work directly enables autonomous vehicles to navigate safely, share the road with pedestrians and cyclists, and scale robotaxi operations to new cities. You will design, train, and deploy production-grade machine learning models that process massive streams of sensor data in real-time under strict latency constraints.

The shape of the workload matters more for your prep than the industry label does. Read-heavy serving, write-heavy ingestion and scheduled batch processing have different binding constraints and fail in different places, so find out which one the team lives in before picking design topics.

Focus areas

Choose indexes from the query's access path
Trace a symptom to a mechanism under load
Detect concurrent edits instead of losing writes

Overview

This role sits at the intersection of deep learning, computer vision, robotics, and distributed systems. You will collaborate closely with perception, behavior planning, infrastructure, and hardware teams to solve high-impact, ambiguous problems that have no existing playbooks. Whether you are optimizing model inference for onboard compute or scaling training pipelines across GPU clusters, your contributions will redefine the future of transportation and urban mobility.

Expect a fast-paced, highly technical environment where rigorous engineering standards meet cutting-edge research. You will face unique challenges involving edge cases, safety-critical software development, and massive scale. Success in this role requires a rare blend of theoretical machine learning depth, systems-level execution, and an unwavering commitment to safety and reliability.

The Cruise interview process

Cruise candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Technical Interviews

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Behavioral Interviews

This round is deciding whether a change you make without supervision can be allowed to reach production. It is scored on what you knew at the moment you decided, not on how it turned out, so a story that opens with the result and works backwards reads as luck retold as judgement. Say what the options were, what you did not know, what you did to shrink the unknown before committing, and what you accepted as the worst plausible case. The detail that separates answers is a bound: how many users, how much data, and for how long, if you had been wrong.

What to demonstrate

Whether the reasoning you give was available at the time you decided rather than after the result came in, since a story whose deciding evidence arrived later describes an outcome and not a judgement
Whether you can put units on the exposure (users, rows, minutes of degraded service) and whether the containment you chose actually bounded it: a canary bounds the request path it fronts, while a background job writing to a shared table reaches every user regardless of which version served their requests
Whether the reversal path existed before you shipped or was improvised during the incident, and whether it restores state or only stops further damage

How to prepare

For your three largest changes, write down the one thing you would have had to be wrong about for it to fail, and what your best estimate of it was on the day you shipped. If you never held an estimate, that is the gap the follow-up questions will find
Write the undo procedure for one of those changes as it existed at the time, then mark which steps restore data and which only stop new damage. Turning a flag off or reverting a deploy ends the new writes; rows already written come back only from a copy you kept, and a dropped column comes back empty unless something outside the schema holds the values
Rehearse one story from the decision point forward and stop before the outcome, then have someone ask what you would do next. If the story only works with the ending attached, it is an anecdote rather than a decision you can defend

Where candidates lose points

Running a schema change as though the lock lasts as long as the statement

In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.

Assuming an isolation level prevents the anomaly you actually have

Isolation levels are named by the SQL standard but implemented differently, so any claim about one is only true of a named engine. PostgreSQL defaults to READ COMMITTED, where every statement takes a fresh snapshot, so two statements inside one transaction can legitimately disagree about the same row. Its REPEATABLE READ is snapshot isolation: it removes non-repeatable and phantom reads but permits write skew, where two transactions each read a set, each conclude their own write is safe, both commit, and the combined result violates a constraint that no single row expresses. Only SERIALIZABLE closes that, and it closes it by aborting a transaction with a serialization failure (SQLSTATE 40001), which means the guarantee is theoretical unless the application has a retry loop. InnoDB's REPEATABLE READ is a different mechanism again - plain SELECTs read a consistent snapshot while locking reads and writes see the latest committed row - so a read-modify-write inside one transaction can act on a value that the transaction's own earlier SELECT never returned.

Choosing a schema before the access patterns are known

Write the queries first, with their filters, sort orders, cardinalities and which ones sit on the latency-critical path, then design tables and indexes to serve them. An index nothing queries still costs write throughput and storage, and a hot query with no supporting index becomes a full scan that only hurts once the table is big.

Starting work without saying what you are about to spend time on

State the plan before executing it: the approach, roughly how long it will take, and what you intend to leave hand-waved. That gives the interviewer a chance to redirect you in ten seconds rather than watching you spend fifteen minutes on the wrong sub-problem.

Questions & practice

1–2 sentences introducing the category and what it tests.

Medium · machine learning fundamentals

Approach

Name the simplest model that could work and what would make you move past it.
Pick the metric from the cost of each error type, not from habit.
Say how you would validate it, and where leakage could enter the split.

Follow-up

How would you know the model is overfitting?
What changes if the classes are heavily imbalanced?

Identify the heaviest tenants in a five-minute window under memory pressure

Medium · top-k · heavy hitters · streaming

The edge service handles about 3,000 requests per second across roughly 50,000 tenants, peaking near 9,000. Expose the 50 heaviest tenants by request count over the trailing five minutes so limits can be tightened before one tenant's backfill starves the fleet. You may not retain five minutes of raw records. Give the exact solution and its memory, then the bounded-memory approximation with its error stated as a formula, and say which you would ship and at what tenant cardinality that choice changes.

Approach

Do the exact version first, because it is affordable at this cardinality: a ring of 300 one-second counters per tenant, advanced lazily, is 1,200 bytes of counters per tenant and roughly 60 to 90 MB for 50,000 tenants with overhead. Carry a running total and subtract the bucket you overwrite so a window read is O(1) rather than 300 adds.
Extract the top 50 with a size-k min-heap over the tenant sums: O(d log k) for d tenants, against O(d log d) to sort them all. Maintaining the heap continuously instead of on query requires a tenant-to-heap-index map, because incrementing a count already inside the heap means sifting from a known position, and without that map you rebuild the heap on every request.
State the approximation precisely rather than gesturing at sketches. Misra-Gries with m counters retains every item whose true count exceeds N/(m+1), and each retained count underestimates the truth by at most N/(m+1). With m = 1,000 and N = 900,000 requests in the window the error is roughly 900 requests, which is fine for spotting a tenant sending 50,000 and useless for ranking two tenants 200 apart.
Say what breaks when the window slides: Misra-Gries and Space-Saving are insert-only and cannot be decremented as records age out. The workable construction is one summary per sub-window, say ten seconds, with 30 summaries merged at query time, and the merged error is the sum of the per-summary errors, so the bound degrades linearly in the number of sub-windows.
Choose and defend it: at 50,000 tenants the exact rings cost under 100 MB in a process that already holds more, so ship exact. Keep the sketch for the case that actually motivates it, a per-principal or per-IP key where cardinality runs to millions and is not bounded by anything you control.
Raise the fleet problem before it is asked: each of 20 to 40 instances sees only its share, and the top 50 of one shard is not the top 50 of the fleet. Either aggregate counts centrally or accept that a per-instance threshold multiplied by instance count is the limit you are really enforcing.

Worked solution (25 min)

Size the exact structure: 300 one-second counters per tenant across 50,000 tenants, plus the running-total trick that makes a window read O(1).
Write the top-k extraction with a size-50 min-heap and compare its complexity against sorting all 50,000 sums.
Substitute N = 900,000 and m = 1,000 into N/(m+1) and state in requests what the sketch can and cannot distinguish.
Write the sub-window merge for the sliding case and state the resulting bound for 30 merged summaries.

Expected result: An exact per-tenant ring of 300 one-second counters at roughly 60 to 90 MB for 50,000 tenants with O(1) window reads, top-50 extraction by a size-k min-heap in O(d log k), a Misra-Gries bound of N/(m+1) with the numbers substituted, the sub-window merge needed to slide it, and a decision to ship exact at this cardinality with the sketch reserved for unbounded keys.

Verify it yourself

The exact window sum for one tenant equals a replay of its raw records over the same five minutes.
The heap result matches a full sort of the tenant sums on a sample.
The sketch's count for a known heavy tenant is at most the true count and no less than the true count minus N/(m+1).

Follow-up

The heaviest tenant is heavy because of one export job rather than user traffic. Should the limiter treat those as the same tenant?
Two tenants sit tied at the boundary of the top 50. Does your answer flap, and does the flapping matter?
You switch to per-principal keys and cardinality goes to 10 million. Walk through what changes.

Archive a resource graph without breaking live references or recursing

Medium · graph traversal · topological order · tenant isolation

Resources reference other resources within a tenant; for the largest tenant the reference table holds up to 2,000,000 nodes and 8,000,000 edges. Archiving a resource must archive everything reachable from it that nothing outside the set still references, refuse when a live external referrer exists, and terminate when references form cycles, which they legitimately do. Produce the archive order and the refusal list, targeting O(V+E). Say what stops the traversal crossing a tenant boundary, and why recursion is the wrong control structure at this size.

Approach

Load the subgraph with the tenant predicate on both endpoints of the edge, not only on the side you started from. Scoping the left table alone is the classic cross-tenant leak: one mis-entered edge then pulls another tenant's resources into the traversal and, worse, into the archive.
Traverse iteratively with an explicit stack. A 2,000,000-node graph can hold a chain deep enough to exhaust a native stack in the low tens of thousands of frames, and that failure is a process crash rather than an error you can return.
Treat cycles as data rather than corruption: compute strongly connected components with Tarjan in O(V+E) using its own explicit stack, then condense. The condensation is a DAG, so a topological order over it gives the archive order, and every member of a component archives in one transaction because no order within a cycle is valid.
Decide refusals with reverse edges. A candidate is archivable only if every in-edge originates inside the candidate set, so build the transpose or count in-degrees restricted to the visited set, and emit each blocked resource with the id of the external referrer, which is the only part of the answer an operator can act on.
Store the graph as CSR rather than a map of lists: an offsets array of V+1 8-byte entries plus E 8-byte targets is about 80 MB at this size, where boxed adjacency lists cost several times that and lose cache locality on every hop.
Run Kahn over the condensation for the order in O(V+E). If the emitted count is short of the component count the condensation step itself is wrong, since a condensation cannot contain a cycle, which makes the check free.

Follow-up

The graph is read in one query and the archive writes a minute later. What can change in between, and how do you make the write safe?
The candidate set is 400,000 resources. Is that one transaction, and if not, what does a half-finished archive look like to a reader?
An edge points at a resource in another tenant. Is that a refusal, an error, or an alert?

Denormalise tenant onto revisions and backfill it live

Hard · denormalisation · online migration · backfill

resource_revision (revision_id, resource_id, version, actor_user_id, change_kind, patch, request_id, created_at) has 400M rows and no tenant column; tenant_id lives only on resource. Two reads need it: a tenant-scoped audit feed ordered by created_at DESC, and an offboarding purge. Both join back to resource today. Justify adding tenant_id to resource_revision against those two reads, name the anomaly the copy introduces and the constraint that prevents it, then give the ordered migration for a live table taking 1.2k writes/second — the lock each step takes, how the backfill is batched, and where each step stops being reversible. PostgreSQL 16.

Approach

Justify from the access path rather than from taste. Without the column, the audit feed either scans resource_revision by created_at and discards other tenants' rows, or resolves the tenant's resource_ids first and probes with them — both proportional to the tenant's whole history rather than to one page. With (tenant_id, created_at DESC, revision_id DESC) it is a seek that stops at 50 rows, and the purge becomes a ranged delete instead of a join.
Name the cost exactly: a second copy of a fact can disagree with the first. Make the disagreement unwritable rather than documented — add UNIQUE (resource_id, tenant_id) on resource so it can serve as a foreign-key target, then FOREIGN KEY (resource_id, tenant_id) REFERENCES resource (resource_id, tenant_id) on the revision table. A revision can then only ever carry its parent's tenant.
Step one, expand: ALTER TABLE resource_revision ADD COLUMN tenant_id BIGINT NULL, with no default, so it is a catalogue change and no rewrite. It still needs ACCESS EXCLUSIVE for an instant, and that instant queues behind the longest open transaction on the table while every later query queues behind it — set lock_timeout to 2s and retry rather than wait.
Step two, dual-write: deploy the writer that populates tenant_id on every new revision while reads still use the join. Reversible by redeploying the previous build, because nothing reads the column yet.
Step three, backfill: batch by primary key rather than by created_at so the cursor is dense and resumable — UPDATE resource_revision rr SET tenant_id = r.tenant_id FROM resource r WHERE r.resource_id = rr.resource_id AND rr.revision_id > $1 AND rr.revision_id <= $1 + 5000 AND rr.tenant_id IS NULL — committing per batch and persisting the cursor. Throttle on replica replay lag and on dead-tuple count, since each batch writes 5,000 new row versions. Run the backfill before the index exists so those updates can stay HOT.
Step four, index then enforce then contract: CREATE INDEX CONCURRENTLY (cannot run inside a transaction block, scans the table twice, waits on open transactions, and leaves an INVALID index to drop concurrently if it fails); ADD CONSTRAINT ... CHECK (tenant_id IS NOT NULL) NOT VALID, then VALIDATE CONSTRAINT, which takes only SHARE UPDATE EXCLUSIVE, after which SET NOT NULL uses the validated check instead of re-scanning on PostgreSQL 12 and later. Only then move the audit reads onto the column and, in a later deploy, delete the join path.

Follow-up

The backfill is half finished and a rollback is required. What state is the table in, and what does the previous build do with a half-populated column?
How do you verify the backfill actually finished, given rows are still being inserted while it runs?
A resource must now be movable between tenants. What does that do to the composite foreign key and to the revisions already written?

Stop tag and share joins from fanning out a page

Medium · join fan-out · semi-joins · many-to-many

resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.

Approach

Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.
Attach the tags for display after the page has been cut — LEFT JOIN LATERAL (SELECT array_agg(rt.tag_id) FROM resource_tag rt WHERE rt.resource_id = p.resource_id) ON TRUE over the 50 returned rows. Aggregate over the page, never over the tenant.
Index both directions and say which query each serves: PK (resource_id, tag_id) serves the lateral lookup, (tag_id, resource_id) serves the EXISTS probe by tag, and resource_share needs (shared_with_user_id, resource_id) for the same reason. An index covering one direction only leaves the other as a scan.

Worked solution (30 min)

Build a tenant where each resource carries 0-5 tags from a 20-tag vocabulary and is shared with 0-4 distinct users, then bind $2 to three tags and $3 to a user holding shares on about half the resources. Run the joined query and compare its row count to the distinct resource count on page one.
Run COUNT(*) on the joined shape and on the EXISTS shape and compare both to a ground truth computed from distinct ids; then give $3 a second permission row on 10% of resources and record which of the two counts moves.
EXPLAIN both page queries and compare rows-read plus the presence of a Sort or HashAggregate node above the join.
Add (tag_id, resource_id), re-run the EXISTS probe, and record the plan change on the inner side.

Expected result: The EXISTS page returns exactly 50 distinct resources and reads about 50 driving rows with no aggregate above the join. The EXISTS count matches the ground truth. The joined COUNT(*) is larger by roughly the average number of matching tags per qualifying resource — the tags it holds from $2, not its total tag count — times its share rows for $3, which is 1 until the duplicate permission rows exist and rises with them afterwards. The EXISTS count is unchanged by those duplicates.

Verify it yourself

The page's resource_id list has no duplicates and is 50 long whenever at least 50 rows match.
count(*) from the rewrite equals count(DISTINCT r.resource_id) over the joined shape.
EXPLAIN shows a semi-join node on the inner side rather than a full join followed by a unique or aggregate step.

Follow-up

The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
A resource can be shared with the same user twice under different permissions. Does your count change, and should it?
Where does the correct total come from when the tenant holds 4M resources and the header must not cost 200 ms?

1–2 sentences introducing the category and what it tests.

Medium · ml system design and production

Approach

Separate the offline training path from the online serving path.
Name what you would monitor after launch and what triggers a retrain.
Fix the product goal and the online metric before choosing any model.

Follow-up

How would you detect drift before the metric drops?
What happens when a feature is missing at serving time?

Make resource creation safe for a client that retried a timeout

Medium · idempotency · retries · race conditions

POST /v1/resources creates a resource row and, in the same transaction, a resource_revision and an outbox_event. A partner's HTTP client times out at 2 seconds and retries twice; under load the first request is still executing when the retry arrives, and duplicate resources are appearing. Using idempotency_key (tenant_id, idempotency_key, request_fingerprint, state, response_status, response_body, resource_id, locked_until, expires_at), specify the whole contract: first call, concurrent retry, retry after completion, a key reused with a different body, and what the client does with each response.

Approach

Name why the plain REST shape fails here. POST is not idempotent, a timeout leaves the client unable to distinguish a lost request from a lost response, so it must retry - and the retry is a second create. Checking whether the key exists and then inserting it is check-then-act: under exactly the concurrency that caused the timeout, both attempts look, both see nothing, both proceed.
Let the unique constraint arbitrate instead. INSERT the key row with state 'in_flight' and locked_until set to now plus the request deadline, and commit that reservation on its own before any work is attempted - an uncommitted row is invisible to the concurrent retry, which is the entire point of writing it first. One caller wins the insert and proceeds; the other takes the unique violation and reads the row rather than falling through to the work.
Branch the loser on what it found. 'succeeded' replays response_status and response_body verbatim, with a header marking it a replay. 'in_flight' with locked_until in the future returns 409 and a Retry-After - do not park a request thread waiting on another request, that is how a concurrency bound is consumed by retries. 'in_flight' past locked_until is reclaimable by a conditional UPDATE predicated on the old locked_until, so exactly one reclaimer wins. A fingerprint mismatch is 422 and never the stored response: a different body under the same key is a client defect, and serving the old response hides it forever.
Commit the effect and the key's completion together - resource insert, revision row, outbox row, and the UPDATE of the reserved key to 'succeeded' with the stored response, in one transaction. That makes a crash binary: either all of it is durable and the key reads 'succeeded', or the transaction rolls back and the only durable trace is the reservation row, still 'in_flight', with no resource behind it. The reclaim branch finishes that case - once locked_until passes, one retry takes the row by conditional UPDATE and runs the work for the first time. Splitting the two commits is what creates the window with no recovery: the effect durable, the key still 'in_flight', and a reclaimer that cannot distinguish that from work never started, so it creates a second resource.
State the lifetime explicitly. Keys expire - 24 hours is a common choice - and are swept, so the client's total retry window must be shorter than the TTL, or a late retry re-executes with no record that it is a duplicate. The key is scoped by tenant so two tenants choosing the same string do not collide, and the client generates one key per logical attempt, not per network send.
Write the client rules: on timeout, retry the same key with capped backoff; on 409, wait Retry-After and retry the same key; on 422, stop and fix the caller; on 2xx, treat the body as authoritative whether it was executed or replayed.

Worked solution (30 min)

Write the two-request interleaving that defeats read-then-insert, with the line each request is executing at each step.
Write the insert-first sequence and the four branches taken on unique violation, as a table of state, locked_until and response.
Put the completion UPDATE inside the same transaction as the resource, revision and outbox writes, then state what a crash before that commit leaves durable and which branch cleans it up.
Define the key's scope, lifetime and generation rule, then bound the client's retry budget by the TTL.
Fire two concurrent requests carrying one key and count rows in resource and resource_revision.

Expected result: One resource row, one revision row and one outbox row; the second caller either replays the first response or receives 409 with Retry-After; a different body under the same key is rejected 422; no request thread blocks waiting on another request.

Verify it yourself

A hundred runs of two simultaneous same-key requests produce exactly one resource and one revision every time.
Killing the process mid-work leaves zero resource rows and the reservation still 'in_flight'; after locked_until passes a retry reclaims the key and executes, so the crash and the retry together produce exactly one resource row.
The documented client retry budget is strictly shorter than expires_at minus created_at.

Follow-up

Response bodies are pruned at 24 hours but resource_id is kept. What do you return to a replay that arrives after pruning?
The endpoint now also sends a welcome email. Which part of this contract prevents a second send, and which part cannot?
What exactly breaks if the key is scoped globally rather than per tenant?

Every query on one table stalls for forty seconds mid-deploy

Medium · lock contention · ddl · migrations · postgres

During a release on PostgreSQL, every query touching resource times out for about 40 seconds and then recovers with no intervention. The release ran one migration, ALTER TABLE resource ADD COLUMN archived_reason TEXT, and the migration log shows it completing in 6 ms. Unrelated tables showed no change in error rate. Explain how a 6 ms statement caused a 40-second stall, give the ordered checks you would run on a live system to confirm it, and give the migration procedure that prevents a repeat.

Approach

Separate the statement's duration from the lock's duration. ADD COLUMN with no default is a catalogue-only change and genuinely runs in milliseconds, but it requires ACCESS EXCLUSIVE, and it cannot acquire that until every transaction already touching the table has finished.
Account for the queueing, which is the part that surprises people. A lock request that is waiting blocks later requests for conflicting modes behind it rather than letting them overtake, so one long-open transaction holds the DDL and the DDL holds all the traffic. The stall length is set by the longest open transaction, not by the size of the change.
Confirm on a live system in this order: pg_stat_activity for that table ordered by xact_start, looking for the oldest transaction and specifically for state = idle in transaction; then pg_locks where granted = false to find the waiter; then join them on pid to name blocker and blocked. pg_blocking_pids() does that join for you and is the fastest single call.
Prevent rather than merely time it better. Set lock_timeout to a second or two on the migration session so the DDL abandons the queue after a bounded wait and is retried, instead of holding it for as long as the oldest transaction lives. Be exact about what that buys: queries arriving during the wait still queue behind the pending ACCESS EXCLUSIVE request, so each attempt costs them up to one lock_timeout of added latency. The outage goes from 40 seconds to about one second per attempt, not to zero. Also run migrations away from deploy-time peaks, and put a statement timeout and an idle-in-transaction timeout on the analytics role that opens the long transactions.
Know the lock each change takes, since the mitigation differs by change. A column with a non-volatile default is a metadata-only change from PostgreSQL 11 and still needs the brief ACCESS EXCLUSIVE; an index needs CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block and leaves an INVALID index to drop if it fails; a check or foreign key is added NOT VALID and then VALIDATE CONSTRAINT as a separate statement under a weaker lock.

Follow-up

The same release also wants NOT NULL on that column. What is the sequence that gets there without a long lock?
Your lock_timeout retry fails ten times in a row because the analytics transaction is always open. What do you change?
How does this differ on MySQL with InnoDB online DDL, and what is the equivalent of the waiting-lock queue there?

Suggested practice checklist

Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.

Day 1 — Coding, one pass at shallow depth

Solve one problem from each of six families, an array with two pointers, hash counting, binary search, a tree traversal, a graph traversal and one dynamic program, under a hard twenty-minute cap with no extensions, marking each finished, late, or stalled.
For every stall, write the exact move you could not make rather than the subject, so the note reads could not turn the recurrence into a loop rather than bad at dynamic programming.
Fix nothing today. The value of the pass is the unfixed record.

Deliverable: Six timed attempts marked finished, late or stalled, each stall carrying a named blocking move.

Day 2 — Design, one pass at shallow depth

Spend twenty minutes each on three different shapes, a read-heavy feed, a write-heavy ingest path, and something needing a transaction across two entities, stopping each at requirements, interface and data model.
After each, write the first question you could not answer, which is usually a number you could not estimate or a failure mode you had no vocabulary for.
Mark which of the three you would be most relieved not to be asked, and treat that as data rather than as a preference.

Deliverable: Three shallow designs, each with the first unanswerable question written at the bottom.

Day 3 — Fundamentals and the practical rounds

Answer eight short questions in writing at four minutes each, covering the material that fills the gaps between the big rounds: what happens between a URL and a rendered page, what an index costs on write, when a process is preferable to a thread, and what conditions a deadlock requires.
Do one thirty-minute practical task of the kind a take-home compresses: read an unfamiliar two-hundred-line file and write what it does, what you would change, and the one thing you remain unsure of.
Score every answer fluent, correct but slow, or absent, and keep the absent ones visible.

Deliverable: Eight scored short answers and one written reading of unfamiliar code.

Day 4 — The rounds that are about you, and the map

Deliver three behavioural answers aloud against a timer, a conflict, a failure you owned, and a decision made without enough information, marking any that ran past three minutes or contained no number.
Assemble the map: every marked item from days one to three on a single page, sorted by how likely it is to appear in your loop rather than by how uncomfortable it felt.
Choose exactly two areas for the remaining three days and write down what you are deliberately abandoning.

Deliverable: A one-page scored map of the whole surface area with two areas chosen and the rest explicitly abandoned.

Day 5 — First chosen area, to the depth you skipped

Work the higher-ranked area in four focused blocks, choosing items one level above where you stalled rather than repeating what already works.
After each block write the rule you extracted in one sentence with its precondition attached, since a rule carrying no precondition is exactly what fails under a variation.
Re-attempt the day-one or day-two item that exposed this area and compare against the original timing.

Deliverable: Four worked blocks, a timed re-attempt against the original, and three one-sentence rules with preconditions.

Day 6 — Second chosen area, where the gap is coverage rather than speed

Treat the second area differently from the first. Day five drilled something you could already half-do; this one is usually a topic you had simply never met, so build one worked reference example end to end and keep it, rather than attempting six problems badly.
Write down the vocabulary you were missing on day two or three, five terms at most, each with the one sentence that makes it usable in an answer rather than the textbook definition.
Redo the shallow attempt that exposed this area and note whether you now fail later in the problem, because moving the failure point is the realistic gain from a single day and is worth more than a score that did not change.

Deliverable: One worked reference example for the newly covered area, a five-term vocabulary list, and a note on where the failure point moved.

Day 7 — Reassemble the loop

Sit two rounds back to back with no gap, ordering them so the area you chose second comes last, because the map was built from rested, isolated attempts and the loop will reach your weaker area when you are already spent.
Write where the second round suffered from the first, which is normally the point at which structure collapses into narration.
Reduce the week to one page holding only the rules you can state without reading them.

Deliverable: Mock notes on cross-round carryover plus a one-page card of rules you can recite from memory.

Review and mentoring: what you let through

Counting review comments or mentees proves nothing. The useful version is a specific change you approved with a reservation you stated, or one you blocked and the delay that cost. Say which standard you were holding and why it was worth the friction. A mentoring story needs the thing the other person can now do without you.

1–2 sentences introducing the category and what it tests.

Medium · behavioural and collaboration

Approach

Give the blast radius: what could have broken, and what you measured.
State the situation in two sentences and spend the rest on the reasoning.
Close with what you would do differently, concretely.

Follow-up

What did you decide not to do, and why?
How did you know your change caused the improvement?

Turn a code review disagreement into a decision

Easy · code review · optimistic concurrency · communication

A colleague's change updates a row with UPDATE resource SET version = version + 1 WHERE resource_id = $1 AND version = $2 and treats an affected-row count of zero as a successful no-op. You read that as a silently lost update; they think returning 200 is friendlier to clients than returning a conflict. Describe how you have handled a review disagreement of this shape: what goes in the comment, when you leave the thread, and who decides. Then write the comment you would leave here, in under 80 words.

Approach

Sort the disagreement before writing anything. A silently discarded write is a correctness claim about data; the choice between 409 and 412 is taste. Only the first justifies blocking a merge, and saying which one you are doing is most of the value of the comment.
Make the claim reproducible in the comment itself with an interleaving rather than a principle: A reads version 7, B reads version 7, B commits version 8, A's predicate matches zero rows, A is told it succeeded and A's edit is gone.
Offer the alternative with its cost attached: return 409 carrying the current version and the revision that won, so the client can re-read and re-apply. Note that automatic retry is not the fix, because a retry re-reads the winner's state and reapplies an intent formed against data that no longer exists.
Apply an escalation rule you can state: two round trips on the thread, then a call, and the service's owner decides rather than the reviewer. A reviewer who cannot be overruled is a bottleneck with extra steps.
Close in writing wherever the decision lands, so the next reader finds the reasoning in the code or the ticket instead of in a collapsed review thread.

Follow-up

Where would you put the test that fails if someone reintroduces the swallowed zero rowcount?
The author says clients cannot handle a 409. How do you check whether that is true?
How do you handle the same review comment when the author is more senior than you and in a hurry?

Unblock an engineer without taking the keyboard

Easy · mentoring · leases · at-least-once

A teammate has spent two days on a job handler that occasionally writes duplicate rows. They are certain the queue is delivering twice by mistake. You suspect a lease expiring under a slow handler, so the job is running concurrently with itself. Describe how you have unblocked someone in this position: what you asked before offering a hypothesis, what you showed them rather than told them, and what you left them owning. Then say what you would do if their theory turned out to be the right one.

Approach

Ask before diagnosing, and ask for things answerable from data they already have: the attempt count on the job rows that produced duplicates, the handler's observed duration against its lease expiry, and whether the duplicate rows share a natural key that a unique constraint could have caught.
Teach the shape rather than the answer. A lease cannot distinguish a dead worker from a slow one, so a handler that outruns its lease is running twice by design, and deploys deliver the other half by killing handlers mid-run on every rollout. Both of their candidate theories produce identical duplicate rows, which is why the evidence has to come from timings rather than from argument.
Hand over a checklist they execute: a natural key on every write the handler performs so the second copy collides rather than appends, the record of intent written before any external effect, a lease heartbeat while running, and the metric that shows it working.
Keep ownership with them deliberately. Pair on the first write, then step back; if you finish it yourself you have closed one ticket and left the same person stuck on the next redelivery.
Close on the systemic gap that let two days pass, which is usually a missing dashboard for attempt counts or an undocumented at-least-once contract, and fix that rather than only the bug.

Follow-up

How would you distinguish a genuine double-delivery from a lease expiry using only the data already stored?
Their handler calls an external endpoint before recording that it did. What do you tell them to change first?
What do you do the third time the same person brings you the same class of bug?