Toyota Machine Learning Engineer Interview Questions & Guide 2026

The 60-second brief

As a Machine Learning Engineer at Toyota, you operate at the intersection of traditional automotive excellence and cutting-edge software innovation. You are not merely building models; you are defining the future of mobility, safety, and vehicle autonomy. Your work directly impacts how millions of drivers interact with their vehicles, ranging from advanced driver-assistance systems (ADAS) to predictive maintenance and personalized in-cabin experiences.

Most rounds test how you handle an under-specified problem more than what you recall. A wrong first approach you correct out loud is recoverable; a constraint you assumed silently and were never given is the expensive mistake.

Focus areas

• Make every write idempotent under retry
• Trace a symptom to a mechanism under load
• Paginate large result sets with keyset cursors

Overview

This role is critical to Toyota’s transition into a mobility-first organization. You will navigate the complexity of high-stakes, real-world deployments where reliability and precision are non-negotiable. Whether you are optimizing computer vision pipelines or developing scalable data architectures, you will be expected to balance academic rigor with the pragmatic demands of a global manufacturing leader. Success here requires a blend of deep technical expertise and the ability to thrive in a highly collaborative, cross-functional environment.

The Toyota interview process

Toyota candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

• Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
• Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
• Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

• Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
• Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
• Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Technical Assessments

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

• In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
• In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
• Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

• Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
• Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
• For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Collaborative Rounds

An unlabelled round is first an information problem, and the cheapest information is free. Whoever schedules it can usually tell you how long it runs, who will be in the room and what they work on, whether you will be writing code and in what environment, and whether anything is being sent beforehand. Ask in writing so the answer is on record, then prepare for the two or three formats those answers still leave open instead of betting on one. What separates a strong candidate is not guessing right; it is having an opening that works whichever one it turns out to be.

What to demonstrate

• Whether you can start work from an ambiguous brief, since tolerating a vague scope without stalling is the same thing the job asks for
• Whether the questions you asked beforehand were ones that change your preparation, such as duration, medium and who is joining, rather than ones whose answers you could not have acted on
• Whether you adapt when the round turns out to be something other than what you were told, instead of spending the first ten minutes visibly recalibrating

How to prepare

• Send one short scheduling message asking four things: how long, who is joining and what they work on, whether you will be writing code and where, and whether to prepare anything in advance. Treat a vague reply as real information, since it means the round is loosely structured and you will be shaping it yourself.
• Write one opening that works in any of the formats still open: restate in your own words what you have been asked to do, then ask which of two directions is more useful to them. Say it aloud until it stops sounding recited.
• Set up for the two most likely formats before the call starts, with a blank editor in the language you would choose and a shared document you can type into, so a format surprise costs you nothing in the first minutes

Behavioral Rounds

Your first answer is not really what is scored. It buys the follow-up questions, and those decide the round. An interviewer with fifteen minutes takes one thread and pushes on it four or five times, so a story you can only tell at a single level of detail collapses under the third why. That is an argument for fewer stories known deeply rather than one prepared per prompt. Four or five pieces of work you can still explain down to the code you changed and the argument you had about it will cover nearly anything asked in this round.

What to demonstrate

• Whether a story holds as the questioning moves from what you did to why that instead of the alternative, and then to what you would change knowing what you know now
• Whether you can re-cut a project to answer the question actually asked rather than delivering a rehearsed block that answers an adjacent one
• Whether your level of detail is chosen rather than habitual: going down to the schema when the question is about the data model, staying out of it when the question is about the person who disagreed with you

How to prepare

• Pick four projects and write the chain out four levels deep for each: what you did, why that, why not the alternative, and what would have to be true for the alternative to have won. Where you cannot reach the fourth level, you have a placeholder rather than a story
• Have someone ask why three times in a row on a single thread with nothing else added, and mark the point where you start repeating a sentence you already said. That point is where the interviewer stops learning anything
• Build a one-page index instead of an answer bank: the common prompts in this round (disagreement, a failure that was yours, thin requirements, a deadline you missed, work you inherited) mapped to which of your four projects you would use for each, so the choosing is done now rather than while an interviewer waits

Final Decision

Nobody in the room with you decides this. Interviewers typically write their rounds up separately, often before seeing anyone else's, and the outcome is settled later from those write-ups. A split panel gets resolved by whichever note carries specific evidence, so what you want out of each room is one concrete thing that person could write down: a bug you caught yourself, a trade-off you named, a decision you owned. The rest is arithmetic. The project you describe in a behavioural conversation is often the same system you sketched an hour earlier, and the two accounts have to agree.

What to demonstrate

• Whether the scale, team size and timeline you attach to a project hold steady when that project resurfaces in a different round
• Whether each interviewer leaves with a specific thing to cite rather than a general impression of competence
• Whether a trade-off you defended in one round survives a challenge in another, instead of being quietly swapped for the answer the new interviewer seemed to want
• Whether a question you have already answered earlier in the day gets the same answer at the same depth, without visible impatience

How to prepare

• Write a one-page sheet per project fixing the figures you will quote — request volume, data size, team size, elapsed time, what broke — and say them aloud from the sheet until they come out identical every time
• For each round on the schedule, decide in advance the one sentence you want in that person's notes, then check in a mock that you said it outright instead of leaving it to be inferred
• Have someone ask you the same project question twice, an hour apart, and diff the two answers for numbers that moved or a trade-off that reversed

Where candidates lose points

Assuming an isolation level prevents the anomaly you actually have

Isolation levels are named by the SQL standard but implemented differently, so any claim about one is only true of a named engine. PostgreSQL defaults to READ COMMITTED, where every statement takes a fresh snapshot, so two statements inside one transaction can legitimately disagree about the same row. Its REPEATABLE READ is snapshot isolation: it removes non-repeatable and phantom reads but permits write skew, where two transactions each read a set, each conclude their own write is safe, both commit, and the combined result violates a constraint that no single row expresses. Only SERIALIZABLE closes that, and it closes it by aborting a transaction with a serialization failure (SQLSTATE 40001), which means the guarantee is theoretical unless the application has a retry loop. InnoDB's REPEATABLE READ is a different mechanism again - plain SELECTs read a consistent snapshot while locking reads and writes see the latest committed row - so a read-modify-write inside one transaction can act on a value that the transaction's own earlier SELECT never returned.

Paginating with LIMIT/OFFSET over a set that changes while the client is reading it

OFFSET n makes the database produce and discard n rows before returning anything, so the cost of a page grows with its depth rather than with its size and page 500 costs five hundred pages of work. The correctness problem is worse than the cost: if a row is inserted or reordered between two page fetches, rows shift across the offset boundary and are either skipped entirely or returned twice, and neither outcome leaves any trace in the response for the client to detect. Keyset pagination - WHERE (sort_key, id) < ($last_sort_key, $last_id) ORDER BY sort_key DESC, id DESC LIMIT n, backed by an index in exactly that order - reads only the rows it returns and is stable against concurrent inserts. It requires the tie-break column: a timestamp is not unique, and duplicate sort keys straddling a page boundary reintroduce the skip it was adopted to remove.

Check-then-act on shared state

Read, decide, write is not safe under concurrency unless the decision and the write are one atomic step: a unique constraint with conflict handling, a compare-and-set, or a row lock held for the whole transaction. Two requests can both pass the existence check before either inserts, which shows up as duplicate rows under load and never in a single-threaded test.

Going silent while thinking

Narrate the candidates and why you are discarding them, even in fragments: sorting first would make this a two-pointer scan, but it destroys the original indices, which the output needs. From the other side of the table, a candidate thinking hard and a candidate stuck are indistinguishable until one of them speaks.

Questions & practice

What strategies do you employ to prevent overfitting when working with limited high-quality labeled data?

Medium · machine learning fundamentals

Approach

1. Say how you would validate it, and where leakage could enter the split.
2. Pick the metric from the cost of each error type, not from habit.
3. Name the simplest model that could work and what would make you move past it.

Follow-up

• Where could label leakage enter this setup?
• What changes if the classes are heavily imbalanced?

Explain the trade-offs between different loss functions in a classification task.

Medium · machine learning fundamentals

Approach

1. Say how you would validate it, and where leakage could enter the split.
2. State the learning problem: the label, the unit of prediction and how the model is used.
3. Name the simplest model that could work and what would make you move past it.

Follow-up

• How would you know the model is overfitting?
• What changes if the classes are heavily imbalanced?

What are the challenges of maintaining model versioning and data lineage in a large-scale enterprise?

Medium · machine learning fundamentals

Approach

1. Say how you would validate it, and where leakage could enter the split.
2. State the learning problem: the label, the unit of prediction and how the model is used.
3. Pick the metric from the cost of each error type, not from habit.

Follow-up

• How would you know the model is overfitting?
• Where could label leakage enter this setup?

How do you evaluate the performance of a model beyond standard accuracy metrics?

Medium · machine learning fundamentals

Approach

1. Say how you would validate it, and where leakage could enter the split.
2. Pick the metric from the cost of each error type, not from habit.
3. Name the simplest model that could work and what would make you move past it.

Follow-up

• How would you know the model is overfitting?
• What changes if the classes are heavily imbalanced?

Diff a projection against the primary without per-row point reads

Hard · reconciliation · range hashing · throttling

The listing projection has drifted and some rows show a stale version. The primary holds 40,000,000 resource rows across 12,000 tenants while serving 1,200 writes and 14,000 reads per second. The obvious repair, reading each resource row and comparing its version against the projection, is correct and would eventually finish. Explain precisely why it is unacceptable here, then give a diff that finds the differing rows, state its complexity, and make it safe to run against a live primary. Replication lag is usually under 100 ms and is not bounded.

Approach

1. Quantify the naive cost rather than calling it slow: 40,000,000 point reads at even 0.5 ms each is over five hours serialised, and the only lever is concurrency, which is exactly what you cannot spend. The primary's pool is sized for the write path, and 40,000,000 random reads evict the buffer cache that sustains the 85 percent cache hit rate, so the audit degrades the system it is auditing.
2. Replace random access with one ordered pass per side. Both sides can be read in (tenant_id, resource_id) order, which is a sequential scan on each and a merge join in O(n) time and O(1) memory. For a dense diff that is the whole answer, and it reads the primary once instead of 40,000,000 times.
3. For the expected sparse case, compare range hashes instead of rows: partition the key space, compute per range an order-independent aggregate over hash(resource_id, version), compare aggregates, and descend only into ranges that differ. With d differing rows and branching factor B, at most d ranges mismatch per level, so the drill-down examines O(d log_B(n/d)) ranges and reads full rows only in mismatching leaves.
4. Aggregate with a sum modulo 2^64 or a multiset hash, never XOR. XOR is order-independent but self-cancelling, so two rows wrong in the same way, or a row duplicated on one side, leave the range aggregate matching and the range is declared clean.
5. Pin the comparison to a point in time or it reports lag as drift: consider only rows whose updated_at is older than now minus a lag margin, and re-check each candidate mismatch individually before repairing. At 1,200 writes per second a diff without this reports thousands of false positives, and an unattended repairer would then overwrite live rows with stale values.
6. Make the run resumable and throttled: batch by range key, persist the last completed range, and watch a signal such as replica lag or primary CPU, pausing rather than pressing on. A reconciliation that cannot be stopped and resumed gets killed halfway and restarted from zero, which is how a repair becomes an incident.

Worked solution (35 min)

1. Compute the naive cost explicitly at 40,000,000 reads and 0.5 ms each, then at 100 concurrent, and state what those connections do to a pool already carrying 1,200 writes per second.
2. Write the merge-join version over (tenant_id, resource_id) and state its memory.
3. Define the range aggregate: the range key, the per-row hash input, and the combining function, with one sentence excluding XOR.
4. Work an example with 40,000,000 rows, branching factor 256 and 5 differing rows, and count the ranges examined.
5. Add the watermark filter and the resume point, and name the throttle signal the loop watches.

Expected result: A rejection of per-row point reads backed by the time cost and the cache-eviction argument, a single ordered merge join as the dense-case answer at O(n) time and O(1) memory, a range-hash drill-down examining O(d log_B(n/d)) ranges using a sum or multiset hash rather than XOR, a watermark excluding recently written rows, and a resumable throttled run loop.

Verify it yourself

• Injecting one known stale row makes exactly one leaf range mismatch and the drill-down reaches it.
• Duplicating a row on one side is detected, which is precisely the case XOR would miss.
• Running the diff twice against an idle system reports zero mismatches both times.

Follow-up

• The diff reports 900 stale rows. How do you decide between patching those rows and rebuilding the projection from resource_revision?
• Same job, but the projection lives in a search index that cannot be scanned in key order. What changes?
• How would you run this continuously at low cost instead of only as incident response?

Keep soft-deleted accounts from blocking re-registration

Easy · soft delete · unique constraints · engine differences

app_user holds user_id, tenant_id, email CITEXT, password_hash (NULL for SSO principals), email_verified_at, auth_version, status ('invited','active','suspended','deactivated'), created_at, updated_at, deleted_at. Two live accounts for one address inside a tenant must be impossible, but an address freed by a soft delete must be reusable, and the same tenant may delete and re-register it repeatedly. Write the uniqueness DDL for PostgreSQL 16, then the equivalent for MySQL 8 where partial indexes do not exist, and say what each permits once three deleted rows already hold that address.

Approach

1. Start from what is actually unique: not (tenant_id, email), but (tenant_id, email) among live rows. PostgreSQL says that directly — CREATE UNIQUE INDEX app_user_live_email ON app_user (tenant_id, email) WHERE deleted_at IS NULL. A full constraint over the same two columns burns the address permanently the first time someone deletes an account.
2. Keep case-insensitivity in the type or the index, never in the application: CITEXT as given, or UNIQUE (tenant_id, lower(email)) as an expression index where the extension is unavailable. A case-sensitive unique column is exactly how two accounts for one human appear.
3. For MySQL 8 the predicate has to move inside the key: add a discriminator column that is a constant 0 while the row is live and is set to user_id on delete, with UNIQUE (tenant_id, email, deleted_marker). Live rows share the constant and still collide; deleted rows differ from each other and stop colliding.
4. State the NULL variant and its dependency: leaving the marker NULL for deleted rows also works, because a unique index treats NULLs as distinct — true in MySQL, and true in PostgreSQL only under the default NULLS DISTINCT, which PostgreSQL 15 lets you reverse. Check the polarity against the three existing deleted rows: constant-on-live is what preserves the collision you want, and reversing it silently admits duplicate live accounts.
5. Say what a soft delete must do besides setting deleted_at: increment auth_version so existing tokens stop validating, leave resource.owner_user_id and resource_revision.actor_user_id intact, and accept that the address is retained — erasure is a different requirement answered by scrubbing the column, not by a DELETE that would break those references.

Follow-up

• A deleted account re-registers with the same address the next day. Do the old resource rows follow the new user_id, and how does the API keep the two principals apart?
• How do you honour an erasure request while resource_revision.actor_user_id still references this table?
• What changes if a user may hold membership in two tenants?

Find version gaps and relay lag with window functions

Medium · window functions · event tables · frame semantics

outbox_event holds event_id, aggregate_type, aggregate_id, aggregate_version, event_type, payload, status ('pending','published','dead'), attempts, created_at, published_at. A projection is missing rows and you must decide whether the relay skipped events or the consumer dropped them. Write three queries over the last seven days: one listing every aggregate_id whose published aggregate_version sequence has a hole, one giving per-day counts with a running total, and one returning the newest published event per aggregate. For each, say where the window function is evaluated relative to WHERE and LIMIT. PostgreSQL 16.

Approach

1. Gaps: compute lead(aggregate_version) OVER (PARTITION BY aggregate_id ORDER BY aggregate_version) in a subquery, then filter next_version <> aggregate_version + 1 in the outer query. Window functions are evaluated after WHERE, GROUP BY and HAVING and before the outer ORDER BY and LIMIT, so the predicate cannot sit in the same WHERE clause and PostgreSQL 16 has no QUALIFY.
2. Say what the seven-day filter does to the answer: it truncates every partition, so the first row per aggregate has no predecessor inside the window and a hole spanning the boundary is invisible. Widen the window, or join to resource.version as the authority for the true maximum.
3. Running total: SELECT date_trunc('day', created_at) AS d, count() AS n, sum(count()) OVER (ORDER BY date_trunc('day', created_at) ROWS UNBOUNDED PRECEDING). An aggregate inside a window call is legal because grouping runs before windowing. The grouping key is unique per row here so ROWS and RANGE agree, but write the frame anyway — over ungrouped rows with tied timestamps the default RANGE frame pulls in every peer row and the total jumps.
4. Newest per aggregate: DISTINCT ON (aggregate_id) ... ORDER BY aggregate_id, aggregate_version DESC is the cheap PostgreSQL-only form when an index matches that order; row_number() OVER (PARTITION BY aggregate_id ORDER BY aggregate_version DESC) = 1 is the portable form and needs a subquery for the same evaluation-order reason as the gap query.
5. Interpret rather than report: no gaps plus a normal p95 of published_at - created_at points at the consumer; gaps or a fat lag tail point at the relay; rows still 'pending' with attempts > 0 point at neither, because they never left the database.
6. Be explicit that the partial index on (created_at, event_id) WHERE status = 'pending' does not serve any of these — they read published rows. Name the index a recurring monitor would need, and say why a query run twice a year may not deserve one.

Worked solution (30 min)

1. Write the three queries against seven days of data and confirm each returns without error.
2. In a scratch copy, delete one middle event for a single aggregate and confirm the gap query names that aggregate and the versions either side.
3. Run a running total over ungrouped rows ordered by date_trunc('second', created_at), once with the default frame and once with ROWS, and record where the two series diverge.
4. Compare the DISTINCT ON and row_number() plans on the same data and record rows-read for each.

Expected result: The gap query returns one row per hole with the version on each side; the daily query returns seven rows whose running total ends at the seven-day count; the newest-per-aggregate query returns exactly one row per aggregate_id present in the window. The RANGE running total is higher than the ROWS one at every tied second.

Verify it yourself

• The per-day counts sum to COUNT(*) over the same predicate.
• Re-running the gap query after reinserting the deleted event returns zero rows.
• count(*) of the newest-per-aggregate result equals count(DISTINCT aggregate_id) over the same filter.

Follow-up

• Relay failover redelivers events. Does a duplicate break the gap query, and how would you detect one from this table alone?
• Turn the gap check into a continuous monitor rather than a query someone runs after an incident. What does it watch?
• The consumer claims it never received event 4,812,006. What do you look at, in what order?

How would you architect a pipeline to retrain models based on edge-case feedback from the field?

Medium · ml system design and production

Approach

1. Fix the product goal and the online metric before choosing any model.
2. Name what you would monitor after launch and what triggers a retrain.
3. Say where features come from at serving time and how they match training.

Follow-up

• How would you roll the new model out safely?
• What happens when a feature is missing at serving time?

Explain how you would deploy a low-latency inference model to an embedded environment.

Medium · ml system design and production

Approach

1. Say where features come from at serving time and how they match training.
2. Separate the offline training path from the online serving path.
3. Fix the product goal and the online metric before choosing any model.

Follow-up

• How would you roll the new model out safely?
• What happens when a feature is missing at serving time?

Design a system for processing and storing massive streams of vehicle telemetry data.

Medium · system design

Approach

1. Choose a partition key and say what query it makes expensive.
2. Name the read and write paths separately; they rarely have the same bottleneck.
3. State the consistency you need, and where you are willing to be stale.

Follow-up

• What would you drop to keep the system up under load?
• What breaks first when traffic grows ten times?

Relay committed events to the log without gaps or reordering

Medium · transactional outbox · at-least-once · ordering · leases

outbox_event rows are written in the same transaction as the state change and carry aggregate_type, aggregate_id, aggregate_version, payload and status, with a partial index on (created_at, event_id) WHERE status = 'pending'. The relay publishes about 4k events/second to a partitioned append-only log keyed by aggregate_id, with one leader per partition range holding a lease. Consumers must never miss an event; they may see one twice. Design the claim-publish-mark loop, and state exactly what consumers observe when a leader's lease expires while it is mid-batch.

Approach

1. Claim with SELECT ... WHERE status='pending' ORDER BY created_at, event_id LIMIT $batch FOR UPDATE SKIP LOCKED inside a transaction. SKIP LOCKED lets several relay workers share a range without serialising on each other's rows, and the partial index keeps the claim proportional to the backlog rather than to a table that is overwhelmingly published rows. At 4k/second a batch of 500 is eight claims per second, each an index scan of 500 entries.
2. Publish before marking, never the reverse, and say why it is a choice. Marking first loses the event outright if the process dies in the gap, and the loss is silent - nothing remains to retry, and it surfaces later as a projection missing a row. Publishing first can repeat the event, and repetition is what every consumer is already built to survive. That single ordering is the whole at-least-once guarantee.
3. Preserve the only ordering on offer. Partition by aggregate_id and never publish two events for one aggregate concurrently: claim in (created_at, event_id) order and publish sequentially within an aggregate, or hash aggregate_id to a worker slot. Order across aggregates is not available at any price here, which is why the event carries aggregate_version and the full fact rather than a delta - a consumer can then discard what it has already applied without coordinating with anyone.
4. State the failover behaviour precisely, because it is the consistency-versus-availability decision in this design. A lease expires because the holder is slow, and no mechanism distinguishes that from dead, so for the length of the lease window two leaders can publish the same claimed batch. The system accepts duplicates to avoid stalling publication for every aggregate in the range whenever one worker pauses. Consumers deduplicate on (aggregate_id, aggregate_version) and drop anything at or below what they have applied.
5. Bound the failure paths and pick the right alarm. A row that fails to publish increments attempts, records last_error, and moves to 'dead' after a limit so one poison payload cannot block the backlog behind it. Alert on the age of the oldest pending row, not on the relay's error rate: the failure worth catching is a relay reporting itself healthy while nothing is being published.

Worked solution (25 min)

1. Write the claim statement and check it against the partial index: which columns it seeks on, how many entries it touches, and what two concurrent workers do to each other.
2. Write both orderings of publish and mark, and for each state what exists after a crash at every point in the loop.
3. Write the consumer's dedupe rule on (aggregate_id, aggregate_version) and test it against a replayed batch of 500.
4. Compute the backlog after a 40-minute outage and the batch rate needed to drain it while 4k/second continues to arrive.

Expected result: A claim using FOR UPDATE SKIP LOCKED over the partial index in (created_at, event_id) order, publish to the aggregate_id partition, then mark published - in that order, accepting duplicates on lease expiry. Per-aggregate order is preserved by never publishing an aggregate concurrently; cross-aggregate order is explicitly not guaranteed. Failures increment attempts and terminate at 'dead'. The alarm is the age of the oldest pending row.

Verify it yourself

• Killing the relay mid-batch produces duplicates on restart and no gap.
• The claim query's plan uses the partial index and reads at most batch-size entries regardless of table size.
• Two relay workers on the same range never block on each other's rows.
• Events for one aggregate_id arrive in aggregate_version order, and no consumer relies on order across aggregates.

Follow-up

• The relay is down 40 minutes and 9.6 million rows are pending. What does catch-up do to the primary, and what changes in the claim loop to survive it?
• A consumer insists it never received an event. Which single query settles whether the relay lost it, and what does each answer look like?
• Delivery is at-least-once. What would exactly-once require end to end, and why is that a property of the consumer rather than of the relay?

Every query on one table stalls for forty seconds mid-deploy

Medium · lock contention · ddl · migrations · postgres

During a release on PostgreSQL, every query touching resource times out for about 40 seconds and then recovers with no intervention. The release ran one migration, ALTER TABLE resource ADD COLUMN archived_reason TEXT, and the migration log shows it completing in 6 ms. Unrelated tables showed no change in error rate. Explain how a 6 ms statement caused a 40-second stall, give the ordered checks you would run on a live system to confirm it, and give the migration procedure that prevents a repeat.

Approach

1. Separate the statement's duration from the lock's duration. ADD COLUMN with no default is a catalogue-only change and genuinely runs in milliseconds, but it requires ACCESS EXCLUSIVE, and it cannot acquire that until every transaction already touching the table has finished.
2. Account for the queueing, which is the part that surprises people. A lock request that is waiting blocks later requests for conflicting modes behind it rather than letting them overtake, so one long-open transaction holds the DDL and the DDL holds all the traffic. The stall length is set by the longest open transaction, not by the size of the change.
3. Confirm on a live system in this order: pg_stat_activity for that table ordered by xact_start, looking for the oldest transaction and specifically for state = idle in transaction; then pg_locks where granted = false to find the waiter; then join them on pid to name blocker and blocked. pg_blocking_pids() does that join for you and is the fastest single call.
4. Prevent rather than merely time it better. Set lock_timeout to a second or two on the migration session so the DDL abandons the queue after a bounded wait and is retried, instead of holding it for as long as the oldest transaction lives. Be exact about what that buys: queries arriving during the wait still queue behind the pending ACCESS EXCLUSIVE request, so each attempt costs them up to one lock_timeout of added latency. The outage goes from 40 seconds to about one second per attempt, not to zero. Also run migrations away from deploy-time peaks, and put a statement timeout and an idle-in-transaction timeout on the analytics role that opens the long transactions.
5. Know the lock each change takes, since the mitigation differs by change. A column with a non-volatile default is a metadata-only change from PostgreSQL 11 and still needs the brief ACCESS EXCLUSIVE; an index needs CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block and leaves an INVALID index to drop if it fails; a check or foreign key is added NOT VALID and then VALIDATE CONSTRAINT as a separate statement under a weaker lock.

Follow-up

• The same release also wants NOT NULL on that column. What is the sequence that gets there without a long lock?
• Your lock_timeout retry fails ten times in a row because the analytics transaction is always open. What do you change?
• How does this differ on MySQL with InnoDB online DDL, and what is the equivalent of the waiting-lock queue there?

Suggested practice checklist

Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.

Day 1 — Diagnostic, scored before you study anything

• Sit a 110-minute diagnostic in four blocks: forty-five minutes on two coding problems, twenty-five on one design prompt taken to interface and data model, twenty of short-answer fundamentals, and twenty delivering two behavioural answers aloud.
• Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct and 0 is stuck, grading the artifact rather than how the attempt felt.
• Allocate days two to five in proportion to 3 minus each block's score, write the allocation down, and commit to leaving it alone.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

• Split the weakest area into named sub-skills and rate each separately. For coding those are restating the problem, choosing the structure, stating the invariant, turning the invariant into loop bounds, handling empty and single-element input, and accounting for complexity out loud.
• Attempt three items positioned just above where the rating drops off, and for each write the first move you failed to make.
• Re-attempt one of them from blank four hours later with nothing open.

Deliverable: A sub-skill map with the two blocking sub-skills circled.

Day 3 — Drill the blocking sub-skill by repeating the shape

• Do eight short repetitions of the same shape rather than eight different problems, so what gets practised is the pattern and not the puzzle.
• State the rule you now hold in one sentence, then test it against a case built to break it, a sliding window over an array containing negative values, or a cache-aside read path whose invalidation message is dropped.
• Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on the strongest area

• Run the same sub-skill decomposition on the second-largest gap in half the time.
• Spend twenty-five timed minutes on the block you scored highest, choosing the hardest item you can still finish rather than a warm-up.
• Write whether each area fails you on recall, on setup, or on execution, and set the fix accordingly: repetition for recall, a written checklist for setup, timed work for execution.

Deliverable: A second sub-skill map plus a one-line failure diagnosis for each area.

Day 5 — The gap that is not a skill

• Record one technical and one behavioural answer, then count two things in the playback: seconds before your first clarifying question, and sentences you began without knowing where they would end.
• Practise saying that you do not know, followed by how you would find out, without letting it soften into a guess, and practise stating a complexity or an estimate before being asked for it.
• Redeliver one answer under a hard ninety-second cap, which forces structure ahead of detail.

Deliverable: Two recordings with a counted reduction in time-to-first-question.

Day 6 — Retest under day-one conditions

• Sit the same 110-minute structure with new prompts of comparable difficulty and score it on the identical rubric.
• For any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
• Write down which single block you would still lose the offer on.

Deliverable: A second scored rubric placed beside the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

• Run a sixty-minute mock over the two blocks that moved least, with an interviewer briefed to interrupt and change direction mid-answer.
• Write the recovery script for going blank: restate the question, state your assumption, name the first thing you would check.
• Say every rule from the week aloud without reading it, and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive an interruption.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

'We rewrote it in a better framework' is not a story

A migration is a cost you chose to pay, not an achievement. The story is what the old system made expensive, what you measured before committing, what kept serving traffic during the cutover, and what you would have done if the numbers had come back flat. Without those, a rewrite reads as taste.

How do you prioritize tasks when faced with conflicting requirements from hardware and software teams?

Medium · behavioural and collaboration

Approach

1. Pick a story where you made the decision, not one where you watched it.
2. Give the blast radius: what could have broken, and what you measured.
3. Close with what you would do differently, concretely.

Follow-up

• What did you decide not to do, and why?
• How did you know your change caused the improvement?

Narrate an outage you owned from page to postmortem

Hard · incident response · blast radius · postmortems

Pick an incident you personally drove, ideally one where writes were affected rather than reads. In six to eight minutes: state the symptom as it first appeared on a dashboard, the blast radius you established before you knew the cause, the mitigation you applied and when, the mechanism you eventually proved, and the follow-up that would prevent a repeat. Bring numbers: error rate, tenants affected, minutes to mitigate, minutes to resolve. If you cannot name what you measured, choose a different incident.

Approach

1. Open on the signal rather than the cause: which metric at which percentile moved, on which service, at what time, so the listener follows the same evidence you had rather than a conclusion you already reached.
2. Separate mitigation from diagnosis out loud. State what you did to stop the bleeding (flag off, shed traffic, drain a lease, roll back a deploy) and say plainly that you did it before the mechanism was known, because those are two jobs with different deadlines.
3. Establish blast radius in countable terms: how many tenants, how many writes, and crucially whether the effect was loss or only delay. An append-only revision table or a pending outbox row means the change survived and the projection was merely behind, which is a repair rather than a data-loss incident.
4. Prove the mechanism instead of asserting it. Name the trace span that grew, the plan that flipped to a sequential scan, the lease that expired, plus one alternative you ruled out and the signal that stayed flat while you ruled it out.
5. Close on the durable fix and its cost, distinguishing what landed that week from what needed an expand-and-contract migration across several deploys, and say which of the two you actually finished.

Follow-up

• What would you do differently in the first five minutes, given the same dashboard and no more information?
• Which follow-up action did you deliberately not take, and why was dropping it the right call?
• How did you convince yourself the mitigation was safe to apply while the cause was still unknown?

Estimate work you have never done and defend the range

Hard · estimation · backfills · expand-contract

You are asked to estimate a change you have never attempted: add a column to a 100-million-row table, populate it, move reads across, and drop the old shape. Give a range with the assumptions that generate it, including batch size, the signal your backfill throttles on, and wall-clock hours, and name the three unknowns that would move the number most. Then describe a real estimate you gave under comparable ignorance: how you expressed its uncertainty, what you committed to, and how wrong you turned out to be.

Approach

1. Decompose into independently deployable steps before estimating anything: add the column nullable, write both shapes, backfill in batches, verify, move reads, stop writing the old shape, drop it. That is four deploys spread over days, and the calendar estimate is dominated by them rather than by the loop's runtime.
2. Do the arithmetic aloud for the part that has arithmetic in it: batch size times number of batches times per-batch duration, at a write rate the primary can absorb alongside roughly 1.2k writes per second of production traffic. The loop is throttled by replication lag and lock waits, not by how fast it can issue statements.
3. Price the schema step by its lock rather than its statement duration. In PostgreSQL an ALTER TABLE taking ACCESS EXCLUSIVE waits for every open transaction on that table while later queries queue behind it, so a millisecond change issued during a thirty-second analytics query stalls that table for thirty seconds. Adding a nullable column with a non-volatile default avoids a rewrite from version 11; a new index wants CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block and leaves an invalid index behind if it fails.
4. Express the answer as a range whose endpoints each trace to a stated assumption, then name the cheapest experiment that collapses it, which is almost always running one real batch against the real table and multiplying.
5. Commit to a checkpoint rather than a completion date: the day you report a measured number from that first batch. That is a promise you can keep under uncertainty, and it is what the asker actually needs in order to plan.

Follow-up

• How do you verify the backfill genuinely finished, given rows written by production traffic while it ran?
• Where does the backfill resume from after a worker is killed mid-batch, and what makes that resume point trustworthy?
• Your first batch comes back ten times slower than assumed. What do you tell the person waiting on the estimate, and when?