Interview guides · Weride.Ai
Weride.Ai Machine Learning Engineer Interview Questions & Guide 2026
Weride.Ai Machine Learning Engineer interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.
The 60-second brief
A Machine Learning Engineer at Weride.Ai occupies a mission-critical role at the absolute forefront of autonomous driving technology. Weride.Ai is pioneering Level 4 autonomous driving, which means the systems you design and deploy must operate with near-perfect reliability in complex, dynamic, and unpredictable real-world environments. The machine learning models developed here do not run in isolated, high-latency cloud environments; they are deployed directly onto autonomous vehicles where millisecond-level latency and safety-critical precision are non-negotiable.
Getting the code to run is the floor. What usually separates answers is the case checked without prompting: empty input, a single element, duplicate keys, or a value that overflows the integer type you chose.
Focus areas
- Choose indexes from the query's access path
- Make every write idempotent under retry
- Bound every outbound call with a timeout
Overview
In this role, your work directly impacts the safety and efficiency of autonomous robotaxis, robobuses, and robovans. You will be tasked with solving some of the most challenging problems in computer vision, sensor fusion, path planning, and behavior prediction. This involves processing massive streams of high-dimensional sensor data—including LiDAR, camera, and radar inputs—to help the vehicle perceive its surroundings, predict the movements of other road users, and make safe driving decisions.
What makes this position exceptionally challenging and rewarding is the scale of the data and the necessity of handling the "long tail" of driving edge cases. You will not just be training off-the-shelf models; you will be designing custom architectures, optimizing real-time inference on edge hardware, and building sophisticated data pipelines to systematically discover and learn from rare driving scenarios. It is a role that demands a rare combination of deep mathematical intuition, robust software engineering, and a passion for physical-world robotics.
The Weride.Ai interview process
Weride.Ai candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.
Recruiter Phone Screen
The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.
What to demonstrate
- Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
- Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
- Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural
How to prepare
- Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
- Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
- Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one
Technical Phone Screen
The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.
What to demonstrate
- Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
- Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
- Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural
How to prepare
- Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
- Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
- Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one
Virtual Onsite Rounds
Nobody in the room with you decides this. Interviewers typically write their rounds up separately, often before seeing anyone else's, and the outcome is settled later from those write-ups. A split panel gets resolved by whichever note carries specific evidence, so what you want out of each room is one concrete thing that person could write down: a bug you caught yourself, a trade-off you named, a decision you owned. The rest is arithmetic. The project you describe in a behavioural conversation is often the same system you sketched an hour earlier, and the two accounts have to agree.
What to demonstrate
- Whether the scale, team size and timeline you attach to a project hold steady when that project resurfaces in a different round
- Whether each interviewer leaves with a specific thing to cite rather than a general impression of competence
- Whether a trade-off you defended in one round survives a challenge in another, instead of being quietly swapped for the answer the new interviewer seemed to want
- Whether a question you have already answered earlier in the day gets the same answer at the same depth, without visible impatience
How to prepare
- Write a one-page sheet per project fixing the figures you will quote — request volume, data size, team size, elapsed time, what broke — and say them aloud from the sheet until they come out identical every time
- For each round on the schedule, decide in advance the one sentence you want in that person's notes, then check in a mock that you said it outright instead of leaving it to be inferred
- Have someone ask you the same project question twice, an hour apart, and diff the two answers for numbers that moved or a trade-off that reversed
Resume Deep Dive
The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.
What to demonstrate
- Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
- Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
- Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural
How to prepare
- Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
- Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
- Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one
Technical Evaluations
The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.
What to demonstrate
- In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
- In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
- Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes
How to prepare
- Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
- Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
- For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first
Where candidates lose points
Paginating with LIMIT/OFFSET over a set that changes while the client is reading it
OFFSET n makes the database produce and discard n rows before returning anything, so the cost of a page grows with its depth rather than with its size and page 500 costs five hundred pages of work. The correctness problem is worse than the cost: if a row is inserted or reordered between two page fetches, rows shift across the offset boundary and are either skipped entirely or returned twice, and neither outcome leaves any trace in the response for the client to detect. Keyset pagination - WHERE (sort_key, id) < ($last_sort_key, $last_id) ORDER BY sort_key DESC, id DESC LIMIT n, backed by an index in exactly that order - reads only the rows it returns and is stable against concurrent inserts. It requires the tie-break column: a timestamp is not unique, and duplicate sort keys straddling a page boundary reintroduce the skip it was adopted to remove.
Running a schema change as though the lock lasts as long as the statement
In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.
Saying 'eventually consistent' without naming the anomaly a user would see
Describe the concrete symptom you are choosing to accept: the author reloads and their own comment is missing for two seconds, or two devices show different balances for a minute. The class of consistency model is a technical label; the tolerable anomaly is the actual product decision.
Assuming the input fits in memory
Ask how large the input is in bytes before committing to an in-memory algorithm; beyond that point the options are a single streaming pass, an external sort with bounded buffers, or a sketch that trades exactness for constant memory. An algorithm that assumes random access to the whole input is a different algorithm from one that sees each element once.
Questions & practice
How would you design and evaluate a prediction model that forecasts the trajectories of surrounding pedestrians and cyclists in a dense urban environment?
Medium · machine learning fundamentals
Approach
- Name the simplest model that could work and what would make you move past it.
- Say how you would validate it, and where leakage could enter the split.
- State the learning problem: the label, the unit of prediction and how the model is used.
Follow-up
- Where could label leakage enter this setup?
- How would you know the model is overfitting?
Write a complete, optimized implementation of Non-Maximum Suppression (NMS) from scratch using native Python/NumPy.
Medium · machine learning fundamentals
Approach
- Say how you would validate it, and where leakage could enter the split.
- State the learning problem: the label, the unit of prediction and how the model is used.
- Pick the metric from the cost of each error type, not from habit.
Follow-up
- What changes if the classes are heavily imbalanced?
- How would you know the model is overfitting?
Implement a custom loss function (such as Focal Loss) designed to handle extreme class imbalance in object detection datasets.
Medium · machine learning fundamentals
Approach
- State the learning problem: the label, the unit of prediction and how the model is used.
- Name the simplest model that could work and what would make you move past it.
- Say how you would validate it, and where leakage could enter the split.
Follow-up
- Where could label leakage enter this setup?
- How would you know the model is overfitting?
Implement an Intersection over Union (IoU) calculation function for 3D bounding boxes, accounting for rotation and yaw.
Medium · machine learning fundamentals
Approach
- Pick the metric from the cost of each error type, not from habit.
- State the learning problem: the label, the unit of prediction and how the model is used.
- Say how you would validate it, and where leakage could enter the split.
Follow-up
- How would you know the model is overfitting?
- Where could label leakage enter this setup?
Solve a hard-level string manipulation and dynamic programming problem to find the longest valid trajectory path under dynamic constraints.
Medium · coding and algorithms
Approach
- Restate the input: its shape, its size, and what is guaranteed about it.
- Choose the data structure from the access pattern, not from familiarity.
- Name the brute-force solution and its complexity before improving on it.
Follow-up
- Which test case would catch an off-by-one here?
- What is the worst case, and how likely is it on real data?
Implement an efficient sliding window algorithm to detect anomalies in continuous sensor data streams.
Medium · coding and algorithms
Approach
- State the target complexity and say which constraint rules the naive version out.
- Walk one small example through your approach before writing the whole thing.
- Name the brute-force solution and its complexity before improving on it.
Follow-up
- What is the worst case, and how likely is it on real data?
- Which test case would catch an off-by-one here?
Archive a resource graph without breaking live references or recursing
Medium · graph traversal · topological order · tenant isolation
Resources reference other resources within a tenant; for the largest tenant the reference table holds up to 2,000,000 nodes and 8,000,000 edges. Archiving a resource must archive everything reachable from it that nothing outside the set still references, refuse when a live external referrer exists, and terminate when references form cycles, which they legitimately do. Produce the archive order and the refusal list, targeting O(V+E). Say what stops the traversal crossing a tenant boundary, and why recursion is the wrong control structure at this size.
Approach
- Load the subgraph with the tenant predicate on both endpoints of the edge, not only on the side you started from. Scoping the left table alone is the classic cross-tenant leak: one mis-entered edge then pulls another tenant's resources into the traversal and, worse, into the archive.
- Traverse iteratively with an explicit stack. A 2,000,000-node graph can hold a chain deep enough to exhaust a native stack in the low tens of thousands of frames, and that failure is a process crash rather than an error you can return.
- Treat cycles as data rather than corruption: compute strongly connected components with Tarjan in O(V+E) using its own explicit stack, then condense. The condensation is a DAG, so a topological order over it gives the archive order, and every member of a component archives in one transaction because no order within a cycle is valid.
- Decide refusals with reverse edges. A candidate is archivable only if every in-edge originates inside the candidate set, so build the transpose or count in-degrees restricted to the visited set, and emit each blocked resource with the id of the external referrer, which is the only part of the answer an operator can act on.
- Store the graph as CSR rather than a map of lists: an offsets array of V+1 8-byte entries plus E 8-byte targets is about 80 MB at this size, where boxed adjacency lists cost several times that and lose cache locality on every hop.
- Run Kahn over the condensation for the order in O(V+E). If the emitted count is short of the component count the condensation step itself is wrong, since a condensation cannot contain a cycle, which makes the check free.
Worked solution (30 min)
- Write the edge-loading query with the tenant predicate on both endpoints and state what it does with a cross-tenant edge.
- Implement iterative Tarjan with an explicit stack and confirm on a three-node cycle that it emits one component of size three.
- Build the transpose restricted to the visited set and mark every node with an in-edge from outside it as refused, carrying the referrer id.
- Run Kahn over the condensation and verify the emitted order against the referrer-before-referenced rule.
- Size the CSR arrays for 2,000,000 nodes and 8,000,000 edges and compare against a boxed adjacency map.
Expected result: An iterative O(V+E) traversal over a tenant-scoped CSR subgraph, SCC condensation so cycles archive atomically as one component, a transpose-based refusal list naming the external referrer for each blocked resource, and a Kahn topological order over the condensation, with recursion replaced by an explicit stack because of graph depth rather than style.
Verify it yourself
- A two-node cycle with no external referrers archives as one unit, and adding a single external referrer moves both nodes to the refusal list.
- No resource appears in the archive order before something that references it, except within its own component.
- A counter on edge visits matches the loaded edge count exactly, confirming each edge is traversed once.
Follow-up
- The graph is read in one query and the archive writes a minute later. What can change in between, and how do you make the write safe?
- The candidate set is 400,000 resources. Is that one transaction, and if not, what does a half-finished archive look like to a reader?
- An edge points at a resource in another tenant. Is that a refusal, an error, or an alert?
Find version gaps and relay lag with window functions
Medium · window functions · event tables · frame semantics
outbox_event holds event_id, aggregate_type, aggregate_id, aggregate_version, event_type, payload, status ('pending','published','dead'), attempts, created_at, published_at. A projection is missing rows and you must decide whether the relay skipped events or the consumer dropped them. Write three queries over the last seven days: one listing every aggregate_id whose published aggregate_version sequence has a hole, one giving per-day counts with a running total, and one returning the newest published event per aggregate. For each, say where the window function is evaluated relative to WHERE and LIMIT. PostgreSQL 16.
Approach
- Gaps: compute lead(aggregate_version) OVER (PARTITION BY aggregate_id ORDER BY aggregate_version) in a subquery, then filter next_version <> aggregate_version + 1 in the outer query. Window functions are evaluated after WHERE, GROUP BY and HAVING and before the outer ORDER BY and LIMIT, so the predicate cannot sit in the same WHERE clause and PostgreSQL 16 has no QUALIFY.
- Say what the seven-day filter does to the answer: it truncates every partition, so the first row per aggregate has no predecessor inside the window and a hole spanning the boundary is invisible. Widen the window, or join to resource.version as the authority for the true maximum.
- Running total: SELECT date_trunc('day', created_at) AS d, count() AS n, sum(count()) OVER (ORDER BY date_trunc('day', created_at) ROWS UNBOUNDED PRECEDING). An aggregate inside a window call is legal because grouping runs before windowing. The grouping key is unique per row here so ROWS and RANGE agree, but write the frame anyway — over ungrouped rows with tied timestamps the default RANGE frame pulls in every peer row and the total jumps.
- Newest per aggregate: DISTINCT ON (aggregate_id) ... ORDER BY aggregate_id, aggregate_version DESC is the cheap PostgreSQL-only form when an index matches that order; row_number() OVER (PARTITION BY aggregate_id ORDER BY aggregate_version DESC) = 1 is the portable form and needs a subquery for the same evaluation-order reason as the gap query.
- Interpret rather than report: no gaps plus a normal p95 of published_at - created_at points at the consumer; gaps or a fat lag tail point at the relay; rows still 'pending' with attempts > 0 point at neither, because they never left the database.
- Be explicit that the partial index on (created_at, event_id) WHERE status = 'pending' does not serve any of these — they read published rows. Name the index a recurring monitor would need, and say why a query run twice a year may not deserve one.
Follow-up
- Relay failover redelivers events. Does a duplicate break the gap query, and how would you detect one from this table alone?
- Turn the gap check into a continuous monitor rather than a query someone runs after an incident. What does it watch?
- The consumer claims it never received event 4,812,006. What do you look at, in what order?
Denormalise tenant onto revisions and backfill it live
Hard · denormalisation · online migration · backfill
resource_revision (revision_id, resource_id, version, actor_user_id, change_kind, patch, request_id, created_at) has 400M rows and no tenant column; tenant_id lives only on resource. Two reads need it: a tenant-scoped audit feed ordered by created_at DESC, and an offboarding purge. Both join back to resource today. Justify adding tenant_id to resource_revision against those two reads, name the anomaly the copy introduces and the constraint that prevents it, then give the ordered migration for a live table taking 1.2k writes/second — the lock each step takes, how the backfill is batched, and where each step stops being reversible. PostgreSQL 16.
Approach
- Justify from the access path rather than from taste. Without the column, the audit feed either scans resource_revision by created_at and discards other tenants' rows, or resolves the tenant's resource_ids first and probes with them — both proportional to the tenant's whole history rather than to one page. With (tenant_id, created_at DESC, revision_id DESC) it is a seek that stops at 50 rows, and the purge becomes a ranged delete instead of a join.
- Name the cost exactly: a second copy of a fact can disagree with the first. Make the disagreement unwritable rather than documented — add UNIQUE (resource_id, tenant_id) on resource so it can serve as a foreign-key target, then FOREIGN KEY (resource_id, tenant_id) REFERENCES resource (resource_id, tenant_id) on the revision table. A revision can then only ever carry its parent's tenant.
- Step one, expand: ALTER TABLE resource_revision ADD COLUMN tenant_id BIGINT NULL, with no default, so it is a catalogue change and no rewrite. It still needs ACCESS EXCLUSIVE for an instant, and that instant queues behind the longest open transaction on the table while every later query queues behind it — set lock_timeout to 2s and retry rather than wait.
- Step two, dual-write: deploy the writer that populates tenant_id on every new revision while reads still use the join. Reversible by redeploying the previous build, because nothing reads the column yet.
- Step three, backfill: batch by primary key rather than by created_at so the cursor is dense and resumable — UPDATE resource_revision rr SET tenant_id = r.tenant_id FROM resource r WHERE r.resource_id = rr.resource_id AND rr.revision_id > $1 AND rr.revision_id <= $1 + 5000 AND rr.tenant_id IS NULL — committing per batch and persisting the cursor. Throttle on replica replay lag and on dead-tuple count, since each batch writes 5,000 new row versions. Run the backfill before the index exists so those updates can stay HOT.
- Step four, index then enforce then contract: CREATE INDEX CONCURRENTLY (cannot run inside a transaction block, scans the table twice, waits on open transactions, and leaves an INVALID index to drop concurrently if it fails); ADD CONSTRAINT ... CHECK (tenant_id IS NOT NULL) NOT VALID, then VALIDATE CONSTRAINT, which takes only SHARE UPDATE EXCLUSIVE, after which SET NOT NULL uses the validated check instead of re-scanning on PostgreSQL 12 and later. Only then move the audit reads onto the column and, in a later deploy, delete the join path.
Worked solution (40 min)
- Write the five steps as separate scripts and state, for each, the lock mode it acquires and the deploy it pairs with.
- On a 20M-row copy, run the ADD COLUMN while a 30-second transaction holds a lock on the table, and record how long unrelated queries queue behind it.
- Run the batched backfill at 5,000 rows, kill it mid-run, restart from the persisted cursor, and confirm no row is processed twice and none is skipped.
- Build the index concurrently under concurrent write load, then add the CHECK ... NOT VALID, VALIDATE it and SET NOT NULL, timing each.
- Compare the audit-feed plan before and after: join-and-filter versus an index seek with no Sort.
Expected result: ADD COLUMN completes in milliseconds once it holds the lock, but with a 30-second transaction open it stalls all traffic on that table for the full 30 seconds unless lock_timeout cuts it off. The backfill resumes exactly at the persisted revision_id. After VALIDATE, SET NOT NULL returns without a table scan. The audit feed drops from scanning the tenant's history to reading about 50 rows.
Verify it yourself
- SELECT count(*) FROM resource_revision WHERE tenant_id IS NULL returns 0, re-checked after the dual-writer has been live for a full deploy cycle.
- A sampled join confirms rr.tenant_id = r.tenant_id across 100k random revisions; the composite foreign key makes a mismatch unwritable from then on.
- pg_stat_activity shows no transaction older than one batch's duration while the backfill runs.
- EXPLAIN of the audit feed shows an Index Scan with no Sort node.
Follow-up
- The backfill is half finished and a rollback is required. What state is the table in, and what does the previous build do with a half-populated column?
- How do you verify the backfill actually finished, given rows are still being inserted while it runs?
- A resource must now be movable between tenants. What does that do to the composite foreign key and to the revisions already written?
Design a real-time perception pipeline that fuses camera and LiDAR inputs, detailing how you handle latency constraints and sensor synchronization.
Medium · ml system design and production
Approach
- Separate the offline training path from the online serving path.
- Say where features come from at serving time and how they match training.
- Fix the product goal and the online metric before choosing any model.
Follow-up
- What happens when a feature is missing at serving time?
- How would you roll the new model out safely?
How would you design an end-to-end machine learning system and data pipeline to systematically obtain, curate, and label datasets for detecting highly uncommon or rare objects on the road?
Medium · ml system design and production
Approach
- Say where features come from at serving time and how they match training.
- Fix the product goal and the online metric before choosing any model.
- Separate the offline training path from the online serving path.
Follow-up
- How would you roll the new model out safely?
- What happens when a feature is missing at serving time?
Keep one tenant's bulk export from starving projection updates
Medium · backpressure · fair scheduling · head-of-line blocking · leases
The worker fleet runs about 600 jobs/second out of job_run, claimed under a lease and heartbeated while running. Most handlers finish under 200 ms; exports run for minutes. One tenant enqueues 50,000 exports. Design the scheduling so that tenant's exports slow down and nothing else does: the queue split, the concurrency caps and where they are enforced, the claim query, and the backpressure signal that stops accepting new work. State the in-flight worker count each class needs at its arrival rate, and what the cap costs the tenant that hits it.
Approach
- Do the concurrency arithmetic first, because it determines the pool shapes. Required in-flight work is arrival rate times service time: 600/second of 200 ms handlers needs about 120 workers, while one export per second at 180 seconds needs 180 on its own. The long tail dominates any pool it shares, so a single pool sized from the mean is consumed by exports while 50 ms projections queue behind them. That is head-of-line blocking, and adding capacity does not fix it because the ratio is what is wrong.
- Split by job class rather than by priority. Priority inside one pool still lets a running export hold its worker for the next four minutes - there is no preemption for a handler already executing. Separate queues and separate worker processes give each class a floor the other cannot take.
- Enforce the per-tenant cap inside the claim, not at enqueue. Claim with SELECT ... WHERE status='queued' AND run_after <= now() AND job_type=$1 ORDER BY run_after, job_run_id FOR UPDATE SKIP LOCKED LIMIT 1, restricted to tenants currently under their cap. Derive the running count from rows in status 'running' with a live lease rather than from a counter incremented at claim and decremented at completion: the derived count self-heals when a worker dies, and the counter leaks a slot every time one does.
- Make the ordering fair over tenants instead of over jobs. Strict FIFO with a cap still forces the claim to scan past 50,000 rows belonging to a capped tenant before it finds anyone else's work, so cost grows with backlog depth. Pick the tenant first from a small per-tenant queue-depth summary, then claim within that tenant, which makes the scan proportional to the number of active tenants instead.
- Close the loop with backpressure and deduplication. Reject or defer enqueues once a class's depth passes a bound, and rely on UNIQUE (job_type, dedupe_key) WHERE status IN ('queued','running') so a retrying producer collapses into one row instead of multiplying the backlog. Watch oldest-queued-age per class rather than depth: depth means nothing without a service rate, while age is the user-visible latency directly.
Worked solution (25 min)
- Compute required in-flight workers per class from arrival rate times service time, and show what one shared pool does when both classes contend.
- Write the claim statement with the per-tenant restriction, and state how the running count is derived so it survives a worker being killed.
- Trace what the claim scans when one tenant holds 50,000 queued rows, under FIFO and under tenant-first selection.
- Choose the backpressure threshold and the signal it watches, then state what the producer receives when it trips.
Expected result: Separate queues and worker pools per job class, sized from rate times service time (about 120 for 200 ms work, about 180 for exports). A per-(job_type, tenant) cap enforced inside the claim with FOR UPDATE SKIP LOCKED, with the running count derived from live leases so it self-heals. Tenant-first selection keeps claim cost proportional to active tenants rather than backlog depth. Enqueue is rejected on class depth, duplicates collapse on dedupe_key, and the alarm is oldest-queued-age per class.
Verify it yourself
- A tenant with 50,000 queued exports never occupies more than its cap's worth of workers.
- Claim cost does not grow as that tenant's backlog grows.
- Killing a worker mid-job returns its cap slot within the lease interval with no manual step.
- Projection p99 is unchanged while the export backlog drains.
Follow-up
- An export outruns its lease, so two copies run. What must the handler do to make the second copy harmless, and what does it write to guarantee that?
- With a cap of 4, the tenant's 50,000 exports now take hours. What do you tell them, and does the answer change if they are the largest tenant on the platform?
- Queue depth is flat but oldest-queued-age is climbing. Name the mechanisms that produce that exact pair of signals.
One customer endpoint stalls deliveries to every other destination
Medium · bulkhead · timeouts · saturation · retry amplification
The egress service delivers about 1.5k webhooks/second across 40,000 destinations, with a per-destination concurrency cap of 4 and a 10-second connect-plus-read timeout. Throughput falls to 300/second, queue depth climbs, and p99 delivery latency for unaffected destinations goes from 200 ms to minutes, while the error rate barely moves. One tenant holds 900 destination rows whose URLs share a hostname that now answers in 9.5 seconds. Explain the mechanism with the arithmetic, then give the containment in the order you would apply it.
Approach
- Look at saturation before errors. A flat error rate with collapsing throughput says nothing is failing, things are waiting, so the first signal to pull is in-flight request count or pool wait time rather than the error counter. This is the distinction that decides the whole investigation.
- Group in-flight work by resolved host, not by destination id. The cap is keyed per destination row, so 900 rows sharing one hostname buy 3,600 concurrent slots against a single host, each held for 9.5 seconds. The bulkhead was never a bulkhead for that host, and grouping by the wrong dimension is why the dashboard looked healthy.
- Do the arithmetic in both directions. Required concurrency is arrival rate times latency, so 1.5k/second at 200 ms needs about 300 in flight, which is entirely consumed by 3,600 slow slots; conversely whatever concurrency is left sustains rate equals concurrency divided by 9.5 seconds, which is the 300/second you are seeing. Matching both numbers is what promotes this from a plausible story to the mechanism.
- Explain why the circuit breaker never helped. It opens on consecutive failures, and a 9.5-second response inside a 10-second timeout is a success. Slow is not failing, so an error-rate breaker cannot see this; you need a slow-call ratio, a deadline propagated from the caller's remaining budget, or a concurrency limiter.
- Contain in order: park the offending host so the shared pool drains, add a per-resolved-host concurrency cap alongside the per-destination one, give slow hosts their own queue so they cannot occupy the general pool, derive the timeout from the delivery deadline rather than a round number, and check the retry policy is not tripling load on a host that is already slow. Use backoff with full jitter so retries do not resynchronise on recovery.
- State the invariant you are restoring: one tenant's endpoints degrade only that tenant's deliveries. That is a property to load-test for, not to assume from a config value.
Follow-up
- The host recovers to 80 ms. How long does the queue take to drain, and what does the drain do to the recovered host?
- Where should the 10-second timeout number actually come from?
- If that tenant had one destination row instead of 900, would the cap of 4 have saved you? What would you measure to be sure?
Suggested practice checklist
For someone who has spent the last few years shipping features and reading other people's code, and who has not solved a timed problem from a blank file in a long time. Five days rebuild the primitives and the patterns that sit on them, working from invariants rather than remembered solutions, and the last two attach that back to the rest of the loop.
Day 1 — Rebuild the primitives by implementing them
- Implement a dynamic array with doubling growth and an operation counter, then change the growth rule to add a fixed sixteen slots instead, and time both for n of ten thousand, a hundred thousand and a million. The fixed-increment version resizes n/16 times at O(n) each, so its total work is quadratic; doubling is what makes append amortised constant.
- Implement a hash map with separate chaining and a load-factor resize, then insert ten thousand keys engineered to land in one bucket and record what happens to lookup time, so that average-case O(1) becomes a claim with a stated precondition rather than a reflex.
- For dynamic-array append and hash-map insert, write down which cost is amortised rather than worst-case, which single operation pays the whole bill, and what a system with a hard per-operation deadline would have to do instead.
Deliverable: Two working implementations plus a timing table showing the input at which each structure's advertised complexity stops holding.
Day 2 — Arrays under an invariant: two pointers, sliding window, binary search
- Solve longest-subarray-with-sum-at-most-K using a sliding window, then run it on an input containing negative numbers and watch it return the wrong answer: extending the window only moves the sum monotonically when every element is non-negative, and that precondition is the whole reason the technique works.
- Write the binary search that finds the first index satisfying a predicate rather than an exact value, put the loop invariant above the loop in a comment, and verify termination on the two inputs that break careless versions: the empty range, and a range where every element satisfies the predicate.
- Compute the midpoint as lo + (hi - lo) / 2 and write one line on why the obvious (lo + hi) / 2 is a genuine defect in a fixed-width integer type and a non-issue in a language with arbitrary-precision integers.
Deliverable: Three solved problems, each with its invariant written above the loop, plus one recorded input on which the sliding window is provably wrong.
Day 3 — Sorting, heaps, and the greedy argument that has to be proved
- Solve one top-k problem three ways, by full sort, by a size-k heap, and by quickselect, then write the values of n and k at which each becomes the right choice, along with quickselect's quadratic worst case and why a randomised pivot makes that unlikely rather than impossible.
- Implement bottom-up heapify and count sift-down steps to confirm it does linear work rather than n log n, because most nodes sit near the bottom of the tree and therefore move only a short distance.
- Take interval scheduling by earliest finishing time and write the exchange argument out in full: given any optimal schedule, swapping in the earliest-finishing interval keeps it feasible and no smaller. Then construct the weighted variant where that same greedy fails and name what has to replace it.
Deliverable: A three-way top-k comparison with measured crossover points, one written exchange argument, and one counterexample to a greedy rule that looks almost identical.
Day 4 — Recursion, memoisation, and the step to a table
- Take one problem with overlapping subproblems, such as edit distance or coin change, instrument the plain recursion with a call counter to show the blow-up, then add memoisation and re-count.
- Convert the memoised version to a bottom-up table and state the two properties you relied on: each subproblem's result depends only on its arguments, and the dependencies form a DAG you can enumerate in order.
- Rewrite one deep recursion with an explicit stack, then find the input length at which the original hits the interpreter's frame limit, which defaults to about a thousand frames in CPython, so you know when the rewrite is required rather than decorative.
Deliverable: One problem in three forms, naive, memoised and tabulated, with call counts for each and the input length at which recursion depth becomes the binding constraint.
Day 5 — Graphs, where most of the work is choosing the traversal
- Implement BFS and DFS over one adjacency list, then answer for each which finds a shortest path in an unweighted graph and which you would use to detect a cycle in a directed graph, including why the in-progress versus finished distinction matters for the second.
- Implement topological sort by in-degree, feed it a graph containing a cycle, and confirm the failure signature is that fewer than V nodes come out rather than an exception, then note that the order it produces is one of several valid ones.
- Run a shortest-path search on a graph with a single negative edge weight and show the wrong answer, then write the precondition Dijkstra actually needs, non-negative weights, because it finalises a node's distance the first time that node is popped, and name the algorithm you would switch to and its own limit.
Deliverable: A small graph library with BFS, DFS and topological sort, plus two inputs that produce documented wrong answers under the wrong algorithm choice.
Day 6 — One day for everything that is not an algorithm
- Sketch one system only to the depth a coding-heavy loop tends to reach: the endpoints, what the service stores, and the single query pattern that decides the schema. Stop at twenty-five minutes.
- Prepare the project answer for an interviewer who codes, which means rehearsing the two levels they push to: the specific thing you built, and why you chose that approach over the alternative they will name. Open with a number and be ready to say what it excludes.
- Prepare the answer to what you would do differently, choosing a real technical mistake with a specific fix rather than a complaint about process or staffing.
Deliverable: One design sketch at endpoint-and-schema depth, plus a project answer rehearsed to two levels of follow-up.
Day 7 — Solve out loud, under time
- Do three timed problems at twenty-five minutes each in a plain editor with no autocomplete and no execution until the end, then tally separately the failures that were syntax and the ones that were approach, because those two numbers call for different fixes.
- Narrate one solution from the first sentence, stating the approach and its complexity before writing any code, and rehearse the sentence you will use when you realise mid-solution that the approach is wrong.
- Re-solve from blank the two problems you were slowest on this week and compare the times against the day they first appeared.
Deliverable: A recording of one fully narrated solution and a tally that separates syntax failures from approach failures.
The bug you decided not to fix
Engineers over-index on what they repaired. A stronger answer covers something you knowingly left broken: the alert you tuned down, the data inconsistency you documented instead of chasing, the cleanup you deferred past two quarters. Give the reasoning and the condition that would have reopened it, so it reads as a decision and not as neglect.
Ship under a deadline and bound the debt you chose
Medium · pagination · technical debt · tradeoffs
You have four days to ship a tenant-facing listing endpoint. The version you would defend uses keyset pagination over (tenant_id, status, updated_at DESC, resource_id DESC); the version you can finish uses LIMIT/OFFSET with no matching index. Describe a deadline call you actually made of this shape: what you shipped, what you knowingly deferred, how you bounded the damage with a mechanism rather than an intention, and the specific numeric condition that would force the follow-up. Name who you told and where you wrote it down.
Approach
- Name the deferred failure precisely instead of calling it slow. OFFSET n makes the database produce and discard n rows, so cost grows with page depth; without an index matching the sort, every matching row is read and sorted before the limit applies; and rows inserted between two page fetches shift across the boundary so items are skipped or repeated with nothing in the response to signal it.
- Bound the blast radius with something mechanical rather than a promise: cap maximum page depth, cap page size, restrict the endpoint to one internal caller, or keep it behind a flag. State which failure each cap removes and which it leaves standing.
- Attach a number to the trigger and wire it to an alarm: the first tenant crossing N resources, or the endpoint's p99 crossing its share of the 400 ms budget, so the debt announces itself instead of waiting to be remembered.
- Write it where the next engineer looks, which is the code and the ticket, not a chat message: what was deferred, why, the cap, and the trigger.
- Report what actually happened in your real example, including the case where the trigger never fired and the debt was correctly never repaid.
Follow-up
- At what page depth does the offset version breach your latency budget, given your page size and row counts?
- What breaks first when you switch to keyset pagination later, and what does a client holding an old page token see?
- Who would have overruled you if you had asked for two more days, and did you ask?
Tell callers you do not own that their integration breaks
Medium · deprecation · compatibility · stakeholders
A field in a write endpoint's response must change shape. You own the endpoint; you do not own the four internal callers or the outbound webhook consumers who read it. Describe a deprecation you were responsible for: what you shipped first, how you established who was actually reading the field, the window you gave and what set its length, what you did about the consumer who never moved, and how you decided removal was safe. Name the signal you used, not the announcement you sent.
Approach
- Establish the reader set empirically rather than from a wiki of owners: per-field usage counters keyed by principal, or access logs attributed to a consumer. State the blind spot of whichever you pick, since a consumer that reads the field only on a monthly job will not appear in a week of logs.
- Ship additive first. Populate the new field alongside the old one so no reader is forced to move, which is also what keeps a rolling deploy safe, because old and new instances answer the same requests at the same time and a rollback must still find the old shape present.
- Set the window from the slowest legitimate consumer's release cadence, not from your calendar, and decide separately what to do for a consumer with no release process at all, such as an external webhook endpoint you can only email.
- Convert silence into evidence before you rely on it: a short, low-traffic removal window that makes a still-dependent consumer fail visibly and loudly while you are watching, rather than at three in the morning after you have moved on.
- State the removal criterion as a measurement with a duration attached, such as observed reads at zero across a full billing cycle, and keep the change reversible for one release after removal.
Follow-up
- How would you detect a consumer that reads the field only during a monthly export?
- One caller refuses to move and has a commercial relationship behind it. What changes in your plan and what does not?
- After removal, what makes the change irreversible, and how long before you cross that line?
Unblock an engineer without taking the keyboard
Easy · mentoring · leases · at-least-once
A teammate has spent two days on a job handler that occasionally writes duplicate rows. They are certain the queue is delivering twice by mistake. You suspect a lease expiring under a slow handler, so the job is running concurrently with itself. Describe how you have unblocked someone in this position: what you asked before offering a hypothesis, what you showed them rather than told them, and what you left them owning. Then say what you would do if their theory turned out to be the right one.
Approach
- Ask before diagnosing, and ask for things answerable from data they already have: the attempt count on the job rows that produced duplicates, the handler's observed duration against its lease expiry, and whether the duplicate rows share a natural key that a unique constraint could have caught.
- Teach the shape rather than the answer. A lease cannot distinguish a dead worker from a slow one, so a handler that outruns its lease is running twice by design, and deploys deliver the other half by killing handlers mid-run on every rollout. Both of their candidate theories produce identical duplicate rows, which is why the evidence has to come from timings rather than from argument.
- Hand over a checklist they execute: a natural key on every write the handler performs so the second copy collides rather than appends, the record of intent written before any external effect, a lease heartbeat while running, and the metric that shows it working.
- Keep ownership with them deliberately. Pair on the first write, then step back; if you finish it yourself you have closed one ticket and left the same person stuck on the next redelivery.
- Close on the systemic gap that let two days pass, which is usually a missing dashboard for attempt counts or an undocumented at-least-once contract, and fix that rather than only the bug.
Follow-up
- How would you distinguish a genuine double-delivery from a lease expiry using only the data already stored?
- Their handler calls an external endpoint before recording that it did. What do you tell them to change first?
- What do you do the third time the same person brings you the same class of bug?