cswork / libraryOpen practice workspace

Interview guides · Meta Logistics

Meta Logistics Machine Learning Engineer Interview Questions & Guide 2026

Meta Logistics Machine Learning Engineer interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.

The 60-second brief

As a Machine Learning Engineer at Meta Logistics, you play a critical role in harnessing the power of data to optimize logistics operations and improve user experiences. Your work will directly impact how products are delivered, enhancing efficiency and effectiveness in the supply chain. By developing advanced machine learning models and algorithms, you contribute to innovative solutions that drive decision-making processes and operational strategies.

Find out what you will be typing into. A shared plain-text editor with no autocomplete, compiler or test runner changes what you have to hold in your head, and practising inside your own configured environment hides exactly that gap.

Focus areas

  • Choose indexes from the query's access path
  • Bound every outbound call with a timeout
  • Make every write idempotent under retry

Overview

This role is not just about coding; it involves understanding complex systems and applying deep learning techniques to real-world problems. You will collaborate with cross-functional teams, including data scientists, software engineers, and product managers, to create scalable models that can handle vast amounts of data. Projects may include building recommendation systems for inventory management, optimizing routing algorithms, or developing predictive analytics tools that enhance operational visibility.

Expect to work in a fast-paced environment where your contributions have a tangible impact on the company's success. As logistics becomes increasingly data-driven, your role as a Machine Learning Engineer will be central to shaping the future of.

The Meta Logistics interview process

Meta Logistics candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screening

Half of this call is the part candidates treat as small talk: start date, notice period, work authorisation and its timing, location and time zone, on-call, and the number. Those are what kill offers late, after several engineers have each spent a day. Surfacing a hard constraint now costs you nothing and occasionally buys you something, since a loop compressed to fit a competing deadline can usually only be arranged if it is asked for early. The common failure is deflecting the compensation question twice, then discovering at offer stage that the band never reached your number.

What to demonstrate

  • Whether your hard constraints are compatible with the role before a loop gets booked: earliest start, notice period, what authorisation you hold and when it needs action, days on site, willingness to carry a pager
  • Whether you give a compensation range with something behind it, such as current total compensation or a competing timeline, rather than leaving the band untested
  • Whether your stated timeline is real, since a competing deadline raised now is something scheduling can sometimes work around and the same deadline raised at offer stage usually is not

How to prepare

  • Write each constraint down in one line before the call and state them as facts rather than negotiating them live under a question you were not expecting
  • Set your range from two or three current data points for that level and location, and name the structure you are quoting in, so the number is comparable to the one they are holding
  • If another process is running, say where it stands and by when, and ask directly whether this loop can be scheduled inside that window

Technical Phone Interviews

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

  • Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
  • Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
  • Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
  • Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

  • For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
  • For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
  • Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Machine Learning System Design

You cannot drill a format you do not know, so put the preparation into material that travels. Three pieces of your own work, each rehearsed until you can take a follow-up you did not anticipate, will carry a conversation or a code walkthrough equally well. Specificity is what separates that from filler. A number needs its definition before it means anything: a p99 is over some window and measured at some hop, and a server-side figure excludes the queueing and network time a client would see. The number you cannot qualify is the one to leave out.

What to demonstrate

  • Whether your examples carry detail only someone who did the work would hold, such as what the binding constraint actually was, which alternative you rejected and why it was worse, and what you measured on each side of the change
  • Whether a number survives one follow-up, meaning you can say what it was measured over and whether it moved because of your change or merely alongside it
  • Whether a failure is described with the specific change that followed it, rather than a lesson stated in general terms
  • Whether your part in a team effort is stated accurately, including what other people did

How to prepare

  • Write a page on each of three projects covering the constraint, the option you rejected, the measurement before and after, and what went wrong. Cut any line you cannot take a follow-up on, since you are writing the parts you will be pressed on rather than a summary.
  • Recover the real figures while you still have access: request volume, data size, latency with its percentile and window, team size, timeline. Note where each came from, whether a dashboard, a design document or memory, and mark the estimates so you can say which they are out loud.
  • Take your weakest project story to someone who works in a different area and have them ask why four times in succession. The point where you run out of answer is the part to go and re-read before the round.

Onsite Interviews

A day like this is several different games in a row, and the expensive mistake is carrying the previous one into the next room. Coding rewards narrow precision and finishing inside a timer. Design rewards breadth, stated assumptions and naming what you are deliberately not building. Behavioural rewards specificity about people and decisions. Candidates who over-engineer a coding problem they were supposed to finish, or who start sketching class hierarchies before anyone has agreed what the system has to do, are usually still playing the last round. Between rooms, name out loud which game the next one is.

What to demonstrate

  • Whether the coding round ends with something that runs and has been traced against a degenerate input, rather than an extensible design that was never finished
  • Whether a design discussion opens by agreeing on traffic shape, read-to-write ratio and what is allowed to be stale, instead of proceeding from an architecture you arrived with
  • Whether a behavioural answer names a person, a disagreement and what you did about it, rather than describing the system the story happened inside
  • Whether the opening habits still appear late in the day: restating the problem, asking for constraints, saying the plan before typing

How to prepare

  • Book three mocks of different types back to back on one afternoon and ask each interviewer afterwards which round you answered in the wrong mode
  • Write a three-line opening script per round type — coding: restate, name the approach and its cost, then type; design: ask for scale, read-write mix and what must not break; behavioural: name the person, the stakes and the decision — and run it off a card so the switch is mechanical rather than remembered
  • Practise coding with a timer you do not extend, stopping when it stops, so the trained reflex is to finish a correct solution rather than to keep improving one

Where candidates lose points

Choosing an index from the columns a query mentions rather than from how it filters and orders

A composite B-tree index on (a, b, c) can be seeked only as a left prefix: equality on a, then equality on b, then a range or an ordering on c. A query that filters on b alone cannot seek into it at all and at best gets a full scan of the index; a query that filters a and ranges on b gets no benefit from c, because the index is only sorted by c within a fixed (a, b) pair. The practical consequence is that one index per column is close to useless for multi-predicate queries while a single correctly ordered composite index turns a scan into a lookup. The ordering half is what gets missed: if the index cannot satisfy the ORDER BY, the database must read every matching row and sort before the limit can apply, so a LIMIT 20 over a million matching rows still reads a million rows.

Letting a slow dependency consume unbounded concurrency

The failure that takes a service down is usually not an error but a delay. A dependency answering in thirty seconds instead of fifty milliseconds holds each request's worker or connection six hundred times longer, and since required concurrency is arrival rate times latency, a fleet sized for sixty in-flight requests now needs thirty-six thousand to sustain the same rate - so it queues, and requests whose clients have already abandoned them still occupy resources. Retries make it precisely worse: a policy of three attempts triples the load on a dependency at the exact moment it is least able to serve, which is how one slow dependency becomes an outage of everything sharing that pool. Containment is four specific things - a timeout on every outbound call shorter than the caller's remaining budget, a bounded pool per dependency so one cannot starve the others, backoff with full jitter rather than a fixed delay so retries do not resynchronise, and a circuit that stops sending once the failure rate makes an attempt pointless.

Sorting when the problem never required a total order

Match the algorithm to the guarantee actually needed: the top k comes from a size-k heap in O(n log k) time and O(k) space, distinctness needs a set rather than an ordering, and a small bounded integer key range admits a linear counting pass. A full O(n log n) sort is the right default only when you genuinely need everything in order.

Comparing floating-point values for equality, or holding money in them

Binary floating point cannot represent 0.1 exactly, so repeated addition drifts and an equality check fails on values that are mathematically equal. Store currency as integer minor units or a decimal type, and compare floats against a tolerance you chose for a stated reason.

Questions & practice

Discuss how you would approach developing a machine learning model for demand forecasting.

Medium · machine learning fundamentals

Approach

  1. Pick the metric from the cost of each error type, not from habit.
  2. Say how you would validate it, and where leakage could enter the split.
  3. State the learning problem: the label, the unit of prediction and how the model is used.

Follow-up

  • How would you know the model is overfitting?
  • Where could label leakage enter this setup?

Describe a machine learning project you worked on and the impact it had.

Medium · machine learning fundamentals

Approach

  1. Name the simplest model that could work and what would make you move past it.
  2. State the learning problem: the label, the unit of prediction and how the model is used.
  3. Say how you would validate it, and where leakage could enter the split.

Follow-up

  • How would you know the model is overfitting?
  • What changes if the classes are heavily imbalanced?

How do you handle overfitting in a model?

Medium · machine learning fundamentals

Approach

  1. Name the simplest model that could work and what would make you move past it.
  2. Pick the metric from the cost of each error type, not from habit.
  3. State the learning problem: the label, the unit of prediction and how the model is used.

Follow-up

  • How would you know the model is overfitting?
  • What changes if the classes are heavily imbalanced?

Write a function to find the k-th smallest element in a sorted matrix.

Medium · coding and algorithms

Approach

  1. Restate the input: its shape, its size, and what is guaranteed about it.
  2. Name the brute-force solution and its complexity before improving on it.
  3. State the target complexity and say which constraint rules the naive version out.

Follow-up

  • What is the worst case, and how likely is it on real data?
  • Which test case would catch an off-by-one here?

Implement a basic calculator that can evaluate expressions.

Medium · coding and algorithms

Approach

  1. Walk one small example through your approach before writing the whole thing.
  2. Name the brute-force solution and its complexity before improving on it.
  3. Choose the data structure from the access pattern, not from familiarity.

Follow-up

  • Which test case would catch an off-by-one here?
  • What is the worst case, and how likely is it on real data?

Find overlapping job attempts and peak concurrency from lease records

Medium · sweep line · intervals · leases

A day of job_run history yields about 50,000,000 attempt records: (job_run_id, job_type, attempt, started_at, finished_at which is NULL when the worker died, lease_expires_at). Leases expire on a clock, so a job that outran its lease ran twice. Produce (a) every job_run_id whose attempts overlapped in wall-clock time and (b) the peak number of simultaneously running attempts per job_type with the minute it occurred. Target O(n log n). State how you treat a NULL finished_at and what clock skew does to your answer.

Approach

  1. Define the interval before sorting anything: an attempt occupies [started_at, COALESCE(finished_at, lease_expires_at)). finished_at is observed and lease_expires_at is only a promise, so every attempt without a finish contributes an estimate and the whole result is a lower bound on overlap rather than an exact count.
  2. For peak concurrency, sweep: emit 2n endpoints, sort by (timestamp, kind) with ends ordered before starts at equal timestamps, then walk the sequence maintaining a counter per job_type and record each type's maximum with its timestamp. O(n log n) dominated by the sort, O(n) space, or O(1) extra if the sort is external and the walk streams.
  3. For overlap detection, do not compare attempts pairwise. A single global sort by (job_run_id, started_at) gives both the grouping and the order; within a group, keep the maximum end seen so far and report an overlap exactly when the next start is less than that running maximum, which is one linear pass after the sort.
  4. Half-open intervals matter and are easy to get wrong: with closed intervals an attempt ending at the same millisecond another begins reads as concurrency two, and across 50,000,000 records that artefact swamps the real signal.
  5. State the clock caveat: started_at and finished_at are written by different workers, so under skew of a few hundred milliseconds an apparent overlap shorter than that bound is not evidence. Filter reported overlaps by a minimum duration, or prefer timestamps written by whichever component heartbeats the lease.
  6. Scale the sort rather than assuming it fits: the sweep emits two endpoints per attempt, so 50,000,000 records become 100,000,000 endpoints, and at roughly 24 bytes each, an 8-byte timestamp plus a 4-byte job_type plus a kind flag padded to alignment, that is about 2.4 GB of sort keys before any scratch space. Either push the ordering into the database behind an index on (job_type, started_at) or run an external merge sort in chunks; the overlap pass sorts n records rather than 2n, so it is the cheaper of the two.

Worked solution (30 min)

  1. Write the interval derivation with the COALESCE and state in one line which of the two end sources is observed and which is assumed.
  2. Write the concurrency sweep: the endpoint tuples, the sort key including the end-before-start tie-break, and the per-job_type counter.
  3. Hand-trace four attempts of one job, two disjoint and two overlapping by three seconds, and confirm the overlap detector fires exactly once.
  4. Add the skew filter as a minimum overlap duration, state the value you chose, and justify it from how the timestamps are written.

Expected result: One global sort by (job_run_id, started_at) driving a running-maximum-end check for overlaps, plus a sweep over 2n endpoints with ends ordered before starts for per-job_type peak concurrency, both O(n log n), with half-open intervals, NULL finished_at falling back to lease_expires_at, and overlaps shorter than the clock-skew bound excluded.

Verify it yourself

  • Two attempts touching at exactly the same instant report concurrency 1, not 2.
  • A job with a single attempt never appears in the overlap output regardless of its lease.
  • The reported peak timestamp for a job_type lies inside at least one of that type's intervals.

Follow-up

  • A handler is not idempotent and you have found 400 overlapping jobs. Which of them actually caused damage, and what would you query to find out?
  • Peak concurrency for one job_type is 4 against a configured cap of 4. Is the cap working, or is the data hiding attempts that never started?
  • How would you compute both answers incrementally as records arrive rather than in a daily batch?

Denormalise tenant onto revisions and backfill it live

Hard · denormalisation · online migration · backfill

resource_revision (revision_id, resource_id, version, actor_user_id, change_kind, patch, request_id, created_at) has 400M rows and no tenant column; tenant_id lives only on resource. Two reads need it: a tenant-scoped audit feed ordered by created_at DESC, and an offboarding purge. Both join back to resource today. Justify adding tenant_id to resource_revision against those two reads, name the anomaly the copy introduces and the constraint that prevents it, then give the ordered migration for a live table taking 1.2k writes/second — the lock each step takes, how the backfill is batched, and where each step stops being reversible. PostgreSQL 16.

Approach

  1. Justify from the access path rather than from taste. Without the column, the audit feed either scans resource_revision by created_at and discards other tenants' rows, or resolves the tenant's resource_ids first and probes with them — both proportional to the tenant's whole history rather than to one page. With (tenant_id, created_at DESC, revision_id DESC) it is a seek that stops at 50 rows, and the purge becomes a ranged delete instead of a join.
  2. Name the cost exactly: a second copy of a fact can disagree with the first. Make the disagreement unwritable rather than documented — add UNIQUE (resource_id, tenant_id) on resource so it can serve as a foreign-key target, then FOREIGN KEY (resource_id, tenant_id) REFERENCES resource (resource_id, tenant_id) on the revision table. A revision can then only ever carry its parent's tenant.
  3. Step one, expand: ALTER TABLE resource_revision ADD COLUMN tenant_id BIGINT NULL, with no default, so it is a catalogue change and no rewrite. It still needs ACCESS EXCLUSIVE for an instant, and that instant queues behind the longest open transaction on the table while every later query queues behind it — set lock_timeout to 2s and retry rather than wait.
  4. Step two, dual-write: deploy the writer that populates tenant_id on every new revision while reads still use the join. Reversible by redeploying the previous build, because nothing reads the column yet.
  5. Step three, backfill: batch by primary key rather than by created_at so the cursor is dense and resumable — UPDATE resource_revision rr SET tenant_id = r.tenant_id FROM resource r WHERE r.resource_id = rr.resource_id AND rr.revision_id > $1 AND rr.revision_id <= $1 + 5000 AND rr.tenant_id IS NULL — committing per batch and persisting the cursor. Throttle on replica replay lag and on dead-tuple count, since each batch writes 5,000 new row versions. Run the backfill before the index exists so those updates can stay HOT.
  6. Step four, index then enforce then contract: CREATE INDEX CONCURRENTLY (cannot run inside a transaction block, scans the table twice, waits on open transactions, and leaves an INVALID index to drop concurrently if it fails); ADD CONSTRAINT ... CHECK (tenant_id IS NOT NULL) NOT VALID, then VALIDATE CONSTRAINT, which takes only SHARE UPDATE EXCLUSIVE, after which SET NOT NULL uses the validated check instead of re-scanning on PostgreSQL 12 and later. Only then move the audit reads onto the column and, in a later deploy, delete the join path.

Worked solution (40 min)

  1. Write the five steps as separate scripts and state, for each, the lock mode it acquires and the deploy it pairs with.
  2. On a 20M-row copy, run the ADD COLUMN while a 30-second transaction holds a lock on the table, and record how long unrelated queries queue behind it.
  3. Run the batched backfill at 5,000 rows, kill it mid-run, restart from the persisted cursor, and confirm no row is processed twice and none is skipped.
  4. Build the index concurrently under concurrent write load, then add the CHECK ... NOT VALID, VALIDATE it and SET NOT NULL, timing each.
  5. Compare the audit-feed plan before and after: join-and-filter versus an index seek with no Sort.

Expected result: ADD COLUMN completes in milliseconds once it holds the lock, but with a 30-second transaction open it stalls all traffic on that table for the full 30 seconds unless lock_timeout cuts it off. The backfill resumes exactly at the persisted revision_id. After VALIDATE, SET NOT NULL returns without a table scan. The audit feed drops from scanning the tenant's history to reading about 50 rows.

Verify it yourself

  • SELECT count(*) FROM resource_revision WHERE tenant_id IS NULL returns 0, re-checked after the dual-writer has been live for a full deploy cycle.
  • A sampled join confirms rr.tenant_id = r.tenant_id across 100k random revisions; the composite foreign key makes a mismatch unwritable from then on.
  • pg_stat_activity shows no transaction older than one batch's duration while the backfill runs.
  • EXPLAIN of the audit feed shows an Index Scan with no Sort node.

Follow-up

  • The backfill is half finished and a rollback is required. What state is the table in, and what does the previous build do with a half-populated column?
  • How do you verify the backfill actually finished, given rows are still being inserted while it runs?
  • A resource must now be movable between tenants. What does that do to the composite foreign key and to the revisions already written?

Hold a per-tenant active cap against concurrent creates

Hard · write skew · isolation levels · constraints

A tenant on the standard plan may hold at most 50 resources with status='active'. The create handler runs SELECT count(*) FROM resource WHERE tenant_id = $1 AND status = 'active', compares to 50, then inserts. Two creates arrive 3 ms apart on different instances and the tenant lands at 51. Name the anomaly, say whether PostgreSQL 16 READ COMMITTED or REPEATABLE READ prevents it and why, then give an implementation that holds the cap at READ COMMITTED with the exact statements. Finally, say what changes when the cap is 'at most one running export per tenant' on job_run.

Approach

  1. Name it: write skew. The two transactions read an overlapping set and write disjoint rows, so there is no row-level conflict for the engine to detect and each commit is individually legal.
  2. Rule out the levels precisely. READ COMMITTED takes a fresh snapshot per statement and takes no lock on the counted rows, so both see 49. PostgreSQL's REPEATABLE READ is snapshot isolation: it removes non-repeatable reads and phantoms within the snapshot but still admits write skew, because the anomaly is not a re-read of a changed row, it is a read of a set that a concurrent transaction invalidates. Only SERIALIZABLE closes it, by tracking the read dependency and aborting one transaction with SQLSTATE 40001 — a guarantee that exists only if the application re-runs the whole transaction from the read.
  3. Convert the set predicate into a single-row conflict: keep tenant.active_resource_count and run UPDATE tenant SET active_resource_count = active_resource_count + 1 WHERE tenant_id = $1 AND active_resource_count < 50 in the same transaction as the INSERT. Zero affected rows is the cap, returned as 409. The row lock serialises the decision at any isolation level, and contention is bounded to one tenant's row — which is also the fair-scheduling unit, unlike a global counter that would convoy every tenant behind one row.
  4. State the cost you just took on: a counter is a second source of truth that can drift, so every path that changes status must adjust it inside the same transaction, and a periodic reconciliation has to exist, with resource_revision as the authority for what the count should have been.
  5. For the job case the invariant is expressible per row, so let the database hold it: a partial unique index on job_run (tenant_id, job_type) WHERE status IN ('queued','running') makes a second running export unwritable and the loser takes 23505, mapped to 409. That is strictly better than a counter — no drift, no reconciliation — and it is available only because the cap is one rather than fifty.
  6. Add the retry discipline each route demands: under SERIALIZABLE both 40001 and deadlock 40P01 are retryable and the retry must re-execute the read, while under READ COMMITTED with the counter nothing retries, because the conflict is reported to the caller rather than raised as an error.

Follow-up

  • A resource moves from archived back to active. Which statements change, and what breaks if the counter update and the status change land in different transactions?
  • The cap becomes plan-dependent and a plan can change mid-month. Where does the number 50 live, and who reads it?
  • How do you detect after the fact that the counter drifted, without locking the table?

Design a recommendation system for a logistics platform.

Medium · ml system design and production

Approach

  1. Fix the product goal and the online metric before choosing any model.
  2. Separate the offline training path from the online serving path.
  3. Name what you would monitor after launch and what triggers a retrain.

Follow-up

  • What happens when a feature is missing at serving time?
  • How would you roll the new model out safely?

How would you architect a data pipeline for real-time analytics?

Medium · system design

Approach

  1. Choose a partition key and say what query it makes expensive.
  2. Name the read and write paths separately; they rarely have the same bottleneck.
  3. Fix the scope first: who calls this, how often, and what they do when it fails.

Follow-up

  • What breaks first when traffic grows ten times?
  • How does this behave when that dependency is down for an hour?

Discuss the trade-offs involved in different data storage solutions.

Medium · system design

Approach

  1. State the consistency you need, and where you are willing to be stale.
  2. Fix the scope first: who calls this, how often, and what they do when it fails.
  3. Name the failure you are designing for, then the recovery path.

Follow-up

  • What breaks first when traffic grows ten times?
  • What would you drop to keep the system up under load?

If tasked with improving delivery times, what factors would you analyze?

Medium

Approach

  1. Work from the requirement backwards to the design.
  2. Clarify what is being asked and what a complete answer contains.
  3. Say what you would check first and why it is the highest-information step.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Design the async export contract a client can resume safely

Hard · async operations · idempotency · leases

A tenant asks for a CSV of every resource. The work runs for minutes on the worker fleet through a job_run row carrying a lease, an attempt count and a dedupe_key, far past the edge's 400 ms budget. Callers are a browser that polls and a script that walks away and checks later. Specify what the submit call returns, the operation resource and its states, how a duplicate submit is handled, how a client learns about completion, what cancellation means given that a lease can expire mid-run, and how the result is fetched and when it expires.

Approach

  1. Split the API in two. Submit returns 202 with an operation id and a location to poll, and never blocks on the work. The operation is a real resource with its own lifecycle - queued, running, succeeded, failed, cancelled - plus attempt, a monotonic progress figure, and a terminal error drawn from the same code taxonomy the synchronous endpoints use, so a client needs one error vocabulary rather than two.
  2. Deduplicate at submit using job_run.dedupe_key, unique over (job_type, dedupe_key) while status is 'queued' or 'running': a repeat submit of the same logical export returns 200 with the existing operation instead of 202 with a new one, and the partial index deliberately permits a legitimate re-run once the first has finished. Pair it with the request's idempotency key so an HTTP-level retry of the submit is exact rather than merely similar.
  3. Tell the poller how to poll: Retry-After on the polling response, a minimum interval enforced at the edge, and a documented maximum lifetime after which an operation is reaped. Polling is the contract of record; the webhook is the fast path, and both must lead to the same terminal state, so a client that receives the completion event and then polls anyway sees no contradiction.
  4. Be exact about cancellation. A cancel request records intent; it cannot stop work already executing. The handler reads the flag at checkpoints, and because a lease expires on a clock that cannot distinguish a dead worker from a slow one, a second copy may start after the cancel was recorded - so the handler re-reads the flag immediately after claiming the lease. 'cancelled' becomes terminal only when no lease is outstanding; reporting it earlier shows a client a stopped job while a worker is still writing output.
  5. Make the handler safe to run twice, because the lease guarantees that it will be. Write output to a deterministic object key derived from the operation id so a second copy overwrites its own work instead of appending a second file, and record completion with a conditional update that only the copy holding the current lease can win.
  6. Treat result fetch as a separate authorised read: a short-lived signed URL, the tenant checked when it is issued rather than only when the file was produced, and a documented retention after which the operation remains terminal but the bytes are gone - a state the client must be able to tell apart from a failure.

Worked solution (40 min)

  1. Write the submit request and its two possible responses, 202 for new and 200 for a duplicate, and the dedupe_key construction.
  2. Draw the operation state machine, marking which transitions a client may observe and which are terminal.
  3. Write the cancellation sequence across a lease expiry, showing where the second copy reads the flag.
  4. Define the output key, the completion update's predicate, and why both are needed for a double run.
  5. Specify result fetch: URL lifetime, authorisation point, retention, and the distinct response once the bytes are gone.

Expected result: A 202-plus-operation contract with deduplicated submits, a documented poll interval alongside an optional completion webhook, cancellation defined as recorded intent with a terminal state gated on lease ownership, an idempotent handler writing to a deterministic key, and a separately authorised expiring result.

Verify it yourself

  • Submitting the same export twice while the first is running produces one job_run row and one operation id.
  • Killing a worker mid-export and letting the lease expire yields one output file, not two, and one terminal transition.
  • A client that receives the completion webhook and then polls sees the same terminal state and the same result location.

Follow-up

  • An operation has said 'running' for 40 minutes and the worker is gone. What does the client see, and which columns in job_run decide that?
  • Two tenants each submit 50 exports at once. What in this contract stops one of them delaying the other?
  • The customer wants the export emailed instead. What changes, and what becomes harder to make exactly-once?

One customer endpoint stalls deliveries to every other destination

Medium · bulkhead · timeouts · saturation · retry amplification

The egress service delivers about 1.5k webhooks/second across 40,000 destinations, with a per-destination concurrency cap of 4 and a 10-second connect-plus-read timeout. Throughput falls to 300/second, queue depth climbs, and p99 delivery latency for unaffected destinations goes from 200 ms to minutes, while the error rate barely moves. One tenant holds 900 destination rows whose URLs share a hostname that now answers in 9.5 seconds. Explain the mechanism with the arithmetic, then give the containment in the order you would apply it.

Approach

  1. Look at saturation before errors. A flat error rate with collapsing throughput says nothing is failing, things are waiting, so the first signal to pull is in-flight request count or pool wait time rather than the error counter. This is the distinction that decides the whole investigation.
  2. Group in-flight work by resolved host, not by destination id. The cap is keyed per destination row, so 900 rows sharing one hostname buy 3,600 concurrent slots against a single host, each held for 9.5 seconds. The bulkhead was never a bulkhead for that host, and grouping by the wrong dimension is why the dashboard looked healthy.
  3. Do the arithmetic in both directions. Required concurrency is arrival rate times latency, so 1.5k/second at 200 ms needs about 300 in flight, which is entirely consumed by 3,600 slow slots; conversely whatever concurrency is left sustains rate equals concurrency divided by 9.5 seconds, which is the 300/second you are seeing. Matching both numbers is what promotes this from a plausible story to the mechanism.
  4. Explain why the circuit breaker never helped. It opens on consecutive failures, and a 9.5-second response inside a 10-second timeout is a success. Slow is not failing, so an error-rate breaker cannot see this; you need a slow-call ratio, a deadline propagated from the caller's remaining budget, or a concurrency limiter.
  5. Contain in order: park the offending host so the shared pool drains, add a per-resolved-host concurrency cap alongside the per-destination one, give slow hosts their own queue so they cannot occupy the general pool, derive the timeout from the delivery deadline rather than a round number, and check the retry policy is not tripling load on a host that is already slow. Use backoff with full jitter so retries do not resynchronise on recovery.
  6. State the invariant you are restoring: one tenant's endpoints degrade only that tenant's deliveries. That is a property to load-test for, not to assume from a config value.

Follow-up

  • The host recovers to 80 ms. How long does the queue take to drain, and what does the drain do to the recovered host?
  • Where should the 10-second timeout number actually come from?
  • If that tenant had one destination row instead of 900, would the cap of 4 have saved you? What would you measure to be sure?

Suggested practice checklist

Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.

Day 1 — Coding, one pass at shallow depth

  • Solve one problem from each of six families, an array with two pointers, hash counting, binary search, a tree traversal, a graph traversal and one dynamic program, under a hard twenty-minute cap with no extensions, marking each finished, late, or stalled.
  • For every stall, write the exact move you could not make rather than the subject, so the note reads could not turn the recurrence into a loop rather than bad at dynamic programming.
  • Fix nothing today. The value of the pass is the unfixed record.

Deliverable: Six timed attempts marked finished, late or stalled, each stall carrying a named blocking move.

Day 2 — Design, one pass at shallow depth

  • Spend twenty minutes each on three different shapes, a read-heavy feed, a write-heavy ingest path, and something needing a transaction across two entities, stopping each at requirements, interface and data model.
  • After each, write the first question you could not answer, which is usually a number you could not estimate or a failure mode you had no vocabulary for.
  • Mark which of the three you would be most relieved not to be asked, and treat that as data rather than as a preference.

Deliverable: Three shallow designs, each with the first unanswerable question written at the bottom.

Day 3 — Fundamentals and the practical rounds

  • Answer eight short questions in writing at four minutes each, covering the material that fills the gaps between the big rounds: what happens between a URL and a rendered page, what an index costs on write, when a process is preferable to a thread, and what conditions a deadlock requires.
  • Do one thirty-minute practical task of the kind a take-home compresses: read an unfamiliar two-hundred-line file and write what it does, what you would change, and the one thing you remain unsure of.
  • Score every answer fluent, correct but slow, or absent, and keep the absent ones visible.

Deliverable: Eight scored short answers and one written reading of unfamiliar code.

Day 4 — The rounds that are about you, and the map

  • Deliver three behavioural answers aloud against a timer, a conflict, a failure you owned, and a decision made without enough information, marking any that ran past three minutes or contained no number.
  • Assemble the map: every marked item from days one to three on a single page, sorted by how likely it is to appear in your loop rather than by how uncomfortable it felt.
  • Choose exactly two areas for the remaining three days and write down what you are deliberately abandoning.

Deliverable: A one-page scored map of the whole surface area with two areas chosen and the rest explicitly abandoned.

Day 5 — First chosen area, to the depth you skipped

  • Work the higher-ranked area in four focused blocks, choosing items one level above where you stalled rather than repeating what already works.
  • After each block write the rule you extracted in one sentence with its precondition attached, since a rule carrying no precondition is exactly what fails under a variation.
  • Re-attempt the day-one or day-two item that exposed this area and compare against the original timing.

Deliverable: Four worked blocks, a timed re-attempt against the original, and three one-sentence rules with preconditions.

Day 6 — Second chosen area, where the gap is coverage rather than speed

  • Treat the second area differently from the first. Day five drilled something you could already half-do; this one is usually a topic you had simply never met, so build one worked reference example end to end and keep it, rather than attempting six problems badly.
  • Write down the vocabulary you were missing on day two or three, five terms at most, each with the one sentence that makes it usable in an answer rather than the textbook definition.
  • Redo the shallow attempt that exposed this area and note whether you now fail later in the problem, because moving the failure point is the realistic gain from a single day and is worth more than a score that did not change.

Deliverable: One worked reference example for the newly covered area, a five-term vocabulary list, and a note on where the failure point moved.

Day 7 — Reassemble the loop

  • Sit two rounds back to back with no gap, ordering them so the area you chose second comes last, because the map was built from rested, isolated attempts and the loop will reach your weaker area when you are already spent.
  • Write where the second round suffered from the first, which is normally the point at which structure collapses into narration.
  • Reduce the week to one page holding only the rules you can state without reading them.

Deliverable: Mock notes on cross-round carryover plus a one-page card of rules you can recite from memory.

What the behavioural round is actually measuring

Every story you tell gets read for blast radius and judgement: what could have broken, who else it touched, what you knew at the moment you decided. Nobody can audit your code in an hour, so they audit your reasoning instead. Pick work where the call was genuinely yours and the consequences were real enough to remember.

Describe a challenge you faced in a team project and how you overcame it.

Medium · behavioural and collaboration

Approach

  1. Name the disagreement and how you resolved it with evidence.
  2. Pick a story where you made the decision, not one where you watched it.
  3. Close with what you would do differently, concretely.

Follow-up

  • How did you know your change caused the improvement?
  • What did you decide not to do, and why?

Unblock an engineer without taking the keyboard

Easy · mentoring · leases · at-least-once

A teammate has spent two days on a job handler that occasionally writes duplicate rows. They are certain the queue is delivering twice by mistake. You suspect a lease expiring under a slow handler, so the job is running concurrently with itself. Describe how you have unblocked someone in this position: what you asked before offering a hypothesis, what you showed them rather than told them, and what you left them owning. Then say what you would do if their theory turned out to be the right one.

Approach

  1. Ask before diagnosing, and ask for things answerable from data they already have: the attempt count on the job rows that produced duplicates, the handler's observed duration against its lease expiry, and whether the duplicate rows share a natural key that a unique constraint could have caught.
  2. Teach the shape rather than the answer. A lease cannot distinguish a dead worker from a slow one, so a handler that outruns its lease is running twice by design, and deploys deliver the other half by killing handlers mid-run on every rollout. Both of their candidate theories produce identical duplicate rows, which is why the evidence has to come from timings rather than from argument.
  3. Hand over a checklist they execute: a natural key on every write the handler performs so the second copy collides rather than appends, the record of intent written before any external effect, a lease heartbeat while running, and the metric that shows it working.
  4. Keep ownership with them deliberately. Pair on the first write, then step back; if you finish it yourself you have closed one ticket and left the same person stuck on the next redelivery.
  5. Close on the systemic gap that let two days pass, which is usually a missing dashboard for attempt counts or an undocumented at-least-once contract, and fix that rather than only the bug.

Follow-up

  • How would you distinguish a genuine double-delivery from a lease expiry using only the data already stored?
  • Their handler calls an external endpoint before recording that it did. What do you tell them to change first?
  • What do you do the third time the same person brings you the same class of bug?

Tell callers you do not own that their integration breaks

Medium · deprecation · compatibility · stakeholders

A field in a write endpoint's response must change shape. You own the endpoint; you do not own the four internal callers or the outbound webhook consumers who read it. Describe a deprecation you were responsible for: what you shipped first, how you established who was actually reading the field, the window you gave and what set its length, what you did about the consumer who never moved, and how you decided removal was safe. Name the signal you used, not the announcement you sent.

Approach

  1. Establish the reader set empirically rather than from a wiki of owners: per-field usage counters keyed by principal, or access logs attributed to a consumer. State the blind spot of whichever you pick, since a consumer that reads the field only on a monthly job will not appear in a week of logs.
  2. Ship additive first. Populate the new field alongside the old one so no reader is forced to move, which is also what keeps a rolling deploy safe, because old and new instances answer the same requests at the same time and a rollback must still find the old shape present.
  3. Set the window from the slowest legitimate consumer's release cadence, not from your calendar, and decide separately what to do for a consumer with no release process at all, such as an external webhook endpoint you can only email.
  4. Convert silence into evidence before you rely on it: a short, low-traffic removal window that makes a still-dependent consumer fail visibly and loudly while you are watching, rather than at three in the morning after you have moved on.
  5. State the removal criterion as a measurement with a duration attached, such as observed reads at zero across a full billing cycle, and keep the change reversible for one release after removal.

Follow-up

  • How would you detect a consumer that reads the field only during a monthly export?
  • One caller refuses to move and has a commercial relationship behind it. What changes in your plan and what does not?
  • After removal, what makes the change irreversible, and how long before you cross that line?