Interview guides · CHEP

CHEP Machine Learning Engineer Interview Questions & Guide 2026

CHEP Machine Learning Engineer interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.

The 60-second brief

As a Machine Learning Engineer at CHEP, you are stepping into a pivotal role at the heart of the global supply chain. CHEP is a worldwide leader in pooling solutions, managing millions of pallets and containers that form the invisible backbone of global commerce. By joining this team, your work directly influences the efficiency, sustainability, and reliability of how goods move around the world.

Allocate preparation against your weakest link rather than your favourite topic. A strong algorithm habit usually comes with weak out-loud explanation of tradeoffs, and years of shipping usually come with rusty from-scratch implementation under a clock.

Focus areas

Paginate large result sets with keyset cursors
Detect concurrent edits instead of losing writes
Bound every outbound call with a timeout

Overview

In this role, you will tackle complex, large-scale logistical challenges using advanced predictive modeling and data science. Your impact extends from optimizing asset tracking and predicting demand surges to minimizing transportation miles and reducing carbon footprints. Because CHEP operates on a circular economy model, the machine learning solutions you build are fundamentally tied to reducing waste and improving operational efficiency for thousands of enterprise customers globally.

The problems you will solve here are highly strategic and technically demanding. You will navigate massive datasets generated by IoT sensors, global transit networks, and regional distribution centers. This role requires a unique blend of deep machine learning expertise, robust software engineering, and a strong product mindset to ensure that theoretical models translate into resilient, production-ready systems that drive tangible business value.

The CHEP interview process

CHEP candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

HR Screening

Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.

What to demonstrate

Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled

How to prepare

Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market

Technical Video Interviews

Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.

What to demonstrate

Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
Whether a failing case is isolated and explained before any edit is made to the code

How to prepare

From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.

Take-Home Assignment

The README is read before the code, and a follow-up conversation is usually built from it, so treat every sentence you put there as a question you have agreed to answer. It needs the command that runs the thing, the assumptions you made where the prompt was ambiguous, and the limits of what you built stated with the preconditions that make them true. Overclaiming is the expensive mistake here. Writing that something is thread-safe, or constant-time, or handles files larger than memory invites a reader to check that exact line, and a claim the code cannot support costs more than silence would have.

What to demonstrate

Whether the run instructions work from a clean clone, naming the exact commands, the language version you tested on, and any environment variable the program expects
Whether ambiguities in the prompt are resolved in writing, with the interpretation you picked and the reason, rather than settled silently in the code
Whether documented limits match the implementation, so a stated input bound is one the code enforces or at least does not contradict
Whether the trade-offs you list come with the condition that would make you choose the other way, instead of reading as a list of alternatives you happened to consider

How to prepare

Write the README before the final hour, then read the code against it claim by claim and correct or delete every statement the implementation does not back
For each ambiguity in the prompt, write one sentence fixing your interpretation and keep it; those sentences become the assumptions section and your answer when someone asks why you did it that way
Give the repository to someone who has not seen the prompt and ask them to run it using only what is written down, treating every question they have to ask you as a gap in the document

Where candidates lose points

Choosing an index from the columns a query mentions rather than from how it filters and orders

A composite B-tree index on (a, b, c) can be seeked only as a left prefix: equality on a, then equality on b, then a range or an ordering on c. A query that filters on b alone cannot seek into it at all and at best gets a full scan of the index; a query that filters a and ranges on b gets no benefit from c, because the index is only sorted by c within a fixed (a, b) pair. The practical consequence is that one index per column is close to useless for multi-predicate queries while a single correctly ordered composite index turns a scan into a lookup. The ordering half is what gets missed: if the index cannot satisfy the ORDER BY, the database must read every matching row and sort before the limit can apply, so a LIMIT 20 over a million matching rows still reads a million rows.

Running a schema change as though the lock lasts as long as the statement

In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.

Tests that assert on the implementation rather than the behaviour

Assert on what a caller can observe, not on the number of internal calls or the shape of a private field. A test that breaks on every refactor but still passes when the answer is wrong costs more than it protects.

Abandoning working code to chase the optimal solution

Get the straightforward version correct, state its complexity, and only then optimise, keeping the working version until the faster one passes the same cases. A correct quadratic solution with a stated path to linear beats a half-written optimal one that never ran.

Questions & practice

How do you handle multicollinearity in a dataset before training a linear regression model?

Medium · machine learning fundamentals

Approach

Pick the metric from the cost of each error type, not from habit.
Name the simplest model that could work and what would make you move past it.
Say how you would validate it, and where leakage could enter the split.

Follow-up

How would you know the model is overfitting?
Where could label leakage enter this setup?

Given a list of transit times, write a Python function to identify and remove statistical outliers.

Medium · machine learning fundamentals

Approach

State the learning problem: the label, the unit of prediction and how the model is used.
Name the simplest model that could work and what would make you move past it.
Pick the metric from the cost of each error type, not from habit.

Follow-up

Where could label leakage enter this setup?
What changes if the classes are heavily imbalanced?

Explain the difference between bagging and boosting, and provide an example of an algorithm for each.

Medium · machine learning fundamentals

Approach

Name the simplest model that could work and what would make you move past it.
State the learning problem: the label, the unit of prediction and how the model is used.
Pick the metric from the cost of each error type, not from habit.

Follow-up

How would you know the model is overfitting?
Where could label leakage enter this setup?

What techniques would you use to prevent a deep neural network from overfitting?

Medium · machine learning fundamentals

Approach

Name the simplest model that could work and what would make you move past it.
Pick the metric from the cost of each error type, not from habit.
State the learning problem: the label, the unit of prediction and how the model is used.

Follow-up

Where could label leakage enter this setup?
How would you know the model is overfitting?

Diff a projection against the primary without per-row point reads

Hard · reconciliation · range hashing · throttling

The listing projection has drifted and some rows show a stale version. The primary holds 40,000,000 resource rows across 12,000 tenants while serving 1,200 writes and 14,000 reads per second. The obvious repair, reading each resource row and comparing its version against the projection, is correct and would eventually finish. Explain precisely why it is unacceptable here, then give a diff that finds the differing rows, state its complexity, and make it safe to run against a live primary. Replication lag is usually under 100 ms and is not bounded.

Approach

Quantify the naive cost rather than calling it slow: 40,000,000 point reads at even 0.5 ms each is over five hours serialised, and the only lever is concurrency, which is exactly what you cannot spend. The primary's pool is sized for the write path, and 40,000,000 random reads evict the buffer cache that sustains the 85 percent cache hit rate, so the audit degrades the system it is auditing.
Replace random access with one ordered pass per side. Both sides can be read in (tenant_id, resource_id) order, which is a sequential scan on each and a merge join in O(n) time and O(1) memory. For a dense diff that is the whole answer, and it reads the primary once instead of 40,000,000 times.
For the expected sparse case, compare range hashes instead of rows: partition the key space, compute per range an order-independent aggregate over hash(resource_id, version), compare aggregates, and descend only into ranges that differ. With d differing rows and branching factor B, at most d ranges mismatch per level, so the drill-down examines O(d log_B(n/d)) ranges and reads full rows only in mismatching leaves.
Aggregate with a sum modulo 2^64 or a multiset hash, never XOR. XOR is order-independent but self-cancelling, so two rows wrong in the same way, or a row duplicated on one side, leave the range aggregate matching and the range is declared clean.
Pin the comparison to a point in time or it reports lag as drift: consider only rows whose updated_at is older than now minus a lag margin, and re-check each candidate mismatch individually before repairing. At 1,200 writes per second a diff without this reports thousands of false positives, and an unattended repairer would then overwrite live rows with stale values.
Make the run resumable and throttled: batch by range key, persist the last completed range, and watch a signal such as replica lag or primary CPU, pausing rather than pressing on. A reconciliation that cannot be stopped and resumed gets killed halfway and restarted from zero, which is how a repair becomes an incident.

Worked solution (35 min)

Compute the naive cost explicitly at 40,000,000 reads and 0.5 ms each, then at 100 concurrent, and state what those connections do to a pool already carrying 1,200 writes per second.
Write the merge-join version over (tenant_id, resource_id) and state its memory.
Define the range aggregate: the range key, the per-row hash input, and the combining function, with one sentence excluding XOR.
Work an example with 40,000,000 rows, branching factor 256 and 5 differing rows, and count the ranges examined.
Add the watermark filter and the resume point, and name the throttle signal the loop watches.

Expected result: A rejection of per-row point reads backed by the time cost and the cache-eviction argument, a single ordered merge join as the dense-case answer at O(n) time and O(1) memory, a range-hash drill-down examining O(d log_B(n/d)) ranges using a sum or multiset hash rather than XOR, a watermark excluding recently written rows, and a resumable throttled run loop.

Verify it yourself

Injecting one known stale row makes exactly one leaf range mismatch and the drill-down reaches it.
Duplicating a row on one side is detected, which is precisely the case XOR would miss.
Running the diff twice against an idle system reports zero mismatches both times.

Follow-up

The diff reports 900 stale rows. How do you decide between patching those rows and rebuilding the projection from resource_revision?
Same job, but the projection lives in a search index that cannot be scanned in key order. What changes?
How would you run this continuously at low cost instead of only as incident response?

Write a SQL query to find the rolling 7-day average of pallet deliveries per regional hub.

Medium · data, sql and pipelines

Approach

State the isolation you are assuming and the anomaly it still allows.
Check whether any join is one-to-many before aggregating, or the sums inflate.
Say which index the query would use, and what makes it unusable.

Follow-up

What happens to this when the table is ten times larger?
How does the query change if that join becomes one-to-many?

Write a function to merge two large datasets with mismatched timestamp frequencies.

Medium · data, sql and pipelines

Approach

State the isolation you are assuming and the anomaly it still allows.
Handle the rows that do not match: that is usually the actual question.
Say which index the query would use, and what makes it unusable.

Follow-up

How would you run this migration without downtime?
What happens to this when the table is ten times larger?

Stop tag and share joins from fanning out a page

Medium · join fan-out · semi-joins · many-to-many

resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.

Approach

Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.
Attach the tags for display after the page has been cut — LEFT JOIN LATERAL (SELECT array_agg(rt.tag_id) FROM resource_tag rt WHERE rt.resource_id = p.resource_id) ON TRUE over the 50 returned rows. Aggregate over the page, never over the tenant.
Index both directions and say which query each serves: PK (resource_id, tag_id) serves the lateral lookup, (tag_id, resource_id) serves the EXISTS probe by tag, and resource_share needs (shared_with_user_id, resource_id) for the same reason. An index covering one direction only leaves the other as a scan.

Worked solution (30 min)

Build a tenant where each resource carries 0-5 tags from a 20-tag vocabulary and is shared with 0-4 distinct users, then bind $2 to three tags and $3 to a user holding shares on about half the resources. Run the joined query and compare its row count to the distinct resource count on page one.
Run COUNT(*) on the joined shape and on the EXISTS shape and compare both to a ground truth computed from distinct ids; then give $3 a second permission row on 10% of resources and record which of the two counts moves.
EXPLAIN both page queries and compare rows-read plus the presence of a Sort or HashAggregate node above the join.
Add (tag_id, resource_id), re-run the EXISTS probe, and record the plan change on the inner side.

Expected result: The EXISTS page returns exactly 50 distinct resources and reads about 50 driving rows with no aggregate above the join. The EXISTS count matches the ground truth. The joined COUNT(*) is larger by roughly the average number of matching tags per qualifying resource — the tags it holds from $2, not its total tag count — times its share rows for $3, which is 1 until the duplicate permission rows exist and rises with them afterwards. The EXISTS count is unchanged by those duplicates.

Verify it yourself

The page's resource_id list has no duplicates and is 50 long whenever at least 50 rows match.
count(*) from the rewrite equals count(DISTINCT r.resource_id) over the joined shape.
EXPLAIN shows a semi-join node on the inner side rather than a full join followed by a unique or aggregate step.

Follow-up

The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
A resource can be shared with the same user twice under different permissions. Does your count change, and should it?
Where does the correct total come from when the tenant holds 4M resources and the header must not cost 200 ms?

Cache the tenant listing feed with a bounded staleness window

Easy · caching · cache invalidation · keyset pagination

GET /v1/resources returns one tenant's resources ordered by updated_at DESC, 20 per page, at 14k requests/second peak against a 120 ms p99. The table carries the index (tenant_id, status, updated_at DESC, resource_id DESC) and writes land on the primary at 1.2k/second. Design the read path: the pagination contract, the cache key and value, what a write invalidates, and the staleness a user can observe. State the request rate that actually reaches the database, and the one repopulation race that deleting on write does not close.

Approach

Settle the pagination contract first, because it decides what is cacheable. OFFSET makes the database produce and discard the skipped rows, so page 500 costs five hundred pages of work, and rows inserted between two fetches shift across the boundary and are skipped or repeated with nothing in the response to reveal it. The cursor is the row value of the last row returned: WHERE tenant_id = $1 AND status = $2 AND (updated_at, resource_id) < ($3, $4) ORDER BY updated_at DESC, resource_id DESC LIMIT 21. That is a row-value comparison, not updated_at < $3 AND resource_id < $4, which is a different and wrong predicate.
Confirm the index actually serves it: equality on the two leading columns, then a range on the pair that follows in exactly the index's sort order, so the plan is an index scan that touches 21 entries with no sort node. Requesting 21 to return 20 is how has_more is answered without a count. resource_id is not decoration - updated_at is not unique, and without the tie-break two rows sharing a timestamp at a page boundary are the skip that keyset pagination was adopted to remove.
Key the cache on every value the predicate reads: tenant_id, status, cursor and page size. A key that omits tenant_id is a cross-tenant disclosure, and no test running against a single tenant's data will show it.
Be honest that a write does not invalidate one key. An update moves its row to the head of the ordering, so it invalidates the first page and every cursor page whose range spans the row's old and new position, which is not enumerable. Cache the first page per (tenant_id, status) - that is where the traffic is - with a short TTL, invalidate it on write, and serve deep cursor pages uncached from a replica, since each is already a 21-row index scan and they are rare.
Name the residual race and the real bound. A reader that loaded rows before the write can populate the cache after the invalidation deleted the key, so the delete is not a staleness bound; the TTL is. Choose the TTL as the staleness a listing can tolerate, and jitter expiries so a busy tenant's keys do not all expire together and stampede the replica. Then do the arithmetic: at an 85% hit rate, 14k requests/second is about 2.1k database reads/second across two replicas, and that is the number capacity planning uses.

Worked solution (20 min)

Write the keyset predicate and match it column by column against the index, marking which columns are equality, which is the range, and which satisfies the ORDER BY.
Compare rows examined for page 1 and page 500 under OFFSET and under keyset, and state both numbers.
Write the cache key template and the first-page invalidation the write path performs.
Write the interleaving in which a stale value is written into the cache after the invalidation, and identify what bounds it.

Expected result: Keyset pagination on the row value (updated_at, resource_id) served as a 21-entry index scan with no sort at any page depth. The first page per (tenant_id, status) is cached under a short jittered TTL and deleted on write; deeper cursor pages go to a replica uncached. Observable staleness is the TTL, not the invalidation, because a concurrent reader can repopulate the key afterwards. At an 85% hit rate roughly 2.1k reads/second reach the two replicas.

Verify it yourself

The query plan shows an index scan with no Sort node, and rows examined equal to the limit, at page 1 and at page 500.
Two rows sharing an updated_at value across a page boundary are returned exactly once across the two pages.
Every cache key contains tenant_id.
Removing the TTL leaves staleness unbounded even though every write still invalidates.

Follow-up

The tenant writes and immediately lists. What does it see, and what is the smallest change that makes its own write visible without sending all 14k requests/second to the primary?
A tenant has 4 million resources and a client walks every page nightly. What does that do to the cache hit rate, and should that traffic share this path at all?
Sort order becomes configurable - by title, by created_at. What happens to the index set and to the cache key space?

Make the resource write endpoint safe for retrying clients

Easy · idempotency · unique constraints · retries

The edge API accepts POST /v1/resources at 1.2k writes/second peak under a 400 ms p99 budget, and clients retry on timeout. idempotency_key has PRIMARY KEY (tenant_id, idempotency_key) plus request_fingerprint CHAR(64), state in ('in_flight','succeeded','failed'), response_status, response_body, resource_id, locked_until and expires_at. Specify the exact sequence the handler runs: what the second request does when it arrives while the first is still executing, what a retry carrying a different body receives, what happens when a process dies mid-request, and how the table is kept from growing without bound.

Approach

Start from the race rather than the happy path. Insert the key row first, in its own short transaction, with state 'in_flight' and locked_until set to now() plus the request deadline. The unique constraint on (tenant_id, idempotency_key) is the arbitration mechanism: exactly one caller commits the insert, and every other caller takes a unique violation (SQLSTATE 23505) and becomes a follower. Reading the table and then inserting is a check-then-act race that both callers pass, and they pass it most often under the load that generates the retries.
Give the follower a branch for every state it can observe, with no fall-through to doing the work: fingerprint mismatch means the key is being reused for a different request, so reject with 422 and never replay the stored response; 'succeeded' replays response_status and response_body verbatim; 'failed' allows a fresh attempt; 'in_flight' with locked_until in the future is answered with 409 and a retry hint, or a short bounded wait, because the only alternative is executing the effect twice.
Commit the effect and the record of the effect together. The resource insert, its resource_revision row, its outbox_event row and the transition of the key row to 'succeeded' all happen in one transaction, so no crash can leave the work done and the key still 'in_flight'. The earlier insert is deliberately a separate transaction: it has to be visible to a concurrent caller before the work starts, which an uncommitted row is not.
Close the wedged-key path. A process that dies after the insert leaves a row that would otherwise block that key forever, which is what locked_until exists for. Reclaim with a single conditional statement, UPDATE ... SET state='in_flight', locked_until=now()+interval WHERE ... AND state='in_flight' AND locked_until < now(), and treat a zero rowcount as losing the reclaim. Two statements that read and then update reintroduce the original race one layer down.
Size the retention before it becomes an incident. At the 1.2k/second peak a 24-hour window is on the order of 100 million rows, and the table grows with traffic rather than with data, so the sweep is load-bearing. Batched deletes driven by an index on expires_at keep it bounded. Range-partitioning by day and dropping whole partitions is cheaper, but PostgreSQL requires every partition-key column to appear in the primary key, so the key becomes (tenant_id, idempotency_key, created_at) and a retry that straddles the boundary no longer collides with its original. State that cost rather than meeting it later.

Follow-up

The stored response body averages 200 KB and this table is now the largest in the database. What do you store instead, and what does a replay return once the body has been pruned?
A client library generates a fresh idempotency key on every attempt. What breaks, which layer should have caught it, and does the server have any defence?
The first attempt succeeded but its response was lost, and the client retries 30 hours later, after the key expired. What does the second attempt do, and is that acceptable?

Exports duplicate a row range about once a week

Hard · leases · clock skew · idempotency · fencing

Roughly once a week an export writes a file containing a duplicated range of rows. The affected job_run rows show attempt = 1, status = succeeded, one started_at, and a lease_owner naming a different host from the one whose logs show the job starting. Leases last 30 seconds and are heartbeated every 10 from inside the handler; lease_expires_at is computed on the worker and compared against the database's now(). Find the mechanism, and give a fix that holds even if you cannot fix the clocks.

Approach

Start from the fact that eliminates the obvious answer. attempt = 1 means no retry was recorded, so this is not a re-run after failure; two workers ran the same row concurrently and the takeover path never touched the counter. lease_owner naming a host other than the one that started the job is the same statement from the other side.
Enumerate the mechanisms that cause a premature takeover, then find the signal that separates them. Either the lease genuinely expired because the heartbeat did not fire, which is what happens when the heartbeat runs on the handler's own thread and the handler makes a long blocking call, or it only appeared expired because two clocks disagree, since lease_expires_at is written from the worker's clock and evaluated against the database's. The discriminator is the distribution: incidents clustered on the longest exports indict the heartbeat, incidents clustered on one host indict skew. Measure both, and measure each host's offset against the database directly.
Read the reclaim query precisely. In PostgreSQL now() is transaction start time, not statement time, so a reclaimer holding a long transaction compares against an older timestamp than expected; clock_timestamp() is the statement-time function. This is worth ruling in or out before you redesign anything, because it changes which rows look expired.
Remove the second clock rather than trying to synchronise it. Issue and extend the lease in the database, with lease_expires_at = now() + interval '30 seconds' in both the claim and the heartbeat, so exactly one clock is ever compared and worker skew stops mattering to this predicate.
Accept that a lease can still expire under a slow worker, because a lease cannot distinguish slow from dead, and fence the work. Carry a monotonically increasing lease generation and make every write the handler performs conditional on still holding it, as UPDATE ... WHERE job_run_id = $1 AND lease_owner = $2 AND lease_generation = $3, so a displaced worker's writes affect zero rows and it aborts instead of duplicating.
Make the handler's writes idempotent independently of all that: give each exported chunk a natural key of (job_run_id, batch_start) with a unique constraint so a second copy conflicts rather than appends, move the heartbeat off the handler's thread, and increment attempt on takeover so the event is visible in a metric.

Follow-up

The displaced worker has already streamed half the file to object storage. What makes that side effect safe to repeat?
You now count takeovers. What alert fires on that counter, and at what threshold?
What breaks if you simply raise the lease to five minutes?

Suggested practice checklist

For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.

Day 1 — Measure before reasoning

Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.

Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.

Day 2 — References, copies, and the bugs they produce

Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.

Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.

Day 3 — Types, once, in a language that checks them

Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.

Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.

Day 4 — Concurrency, starting with what actually runs at the same time

Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.

Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.

Day 5 — Debugging as a procedure rather than an instinct

Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.

Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.

Day 6 — Tests that catch the bug you are about to write

Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.

Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.

Day 7 — Debug something broken, out loud

Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.

Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.

Know where your numbers came from

Bring the two or three numbers the story rests on and know how they were collected. A p99 whose timer starts inside your handler excludes the time a request spent queued, so it can sit flat while users wait longer. Give the window, the percentile and what the measurement left out, or drop the number.

Tell me about a time you had to learn a new technology on the fly to complete a project.

Medium · behavioural and collaboration

Approach

Pick a story where you made the decision, not one where you watched it.
Name the disagreement and how you resolved it with evidence.
State the situation in two sentences and spend the rest on the reasoning.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?

Tell me about a time your model performed well in testing but failed in production. How did you handle it?

Medium · behavioural and collaboration

Approach

Name the disagreement and how you resolved it with evidence.
Give the blast radius: what could have broken, and what you measured.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?

Unblock an engineer without taking the keyboard

Easy · mentoring · leases · at-least-once

A teammate has spent two days on a job handler that occasionally writes duplicate rows. They are certain the queue is delivering twice by mistake. You suspect a lease expiring under a slow handler, so the job is running concurrently with itself. Describe how you have unblocked someone in this position: what you asked before offering a hypothesis, what you showed them rather than told them, and what you left them owning. Then say what you would do if their theory turned out to be the right one.

Approach

Ask before diagnosing, and ask for things answerable from data they already have: the attempt count on the job rows that produced duplicates, the handler's observed duration against its lease expiry, and whether the duplicate rows share a natural key that a unique constraint could have caught.
Teach the shape rather than the answer. A lease cannot distinguish a dead worker from a slow one, so a handler that outruns its lease is running twice by design, and deploys deliver the other half by killing handlers mid-run on every rollout. Both of their candidate theories produce identical duplicate rows, which is why the evidence has to come from timings rather than from argument.
Hand over a checklist they execute: a natural key on every write the handler performs so the second copy collides rather than appends, the record of intent written before any external effect, a lease heartbeat while running, and the metric that shows it working.
Keep ownership with them deliberately. Pair on the first write, then step back; if you finish it yourself you have closed one ticket and left the same person stuck on the next redelivery.
Close on the systemic gap that let two days pass, which is usually a missing dashboard for attempt counts or an undocumented at-least-once contract, and fix that rather than only the bug.

Follow-up

How would you distinguish a genuine double-delivery from a lease expiry using only the data already stored?
Their handler calls an external endpoint before recording that it did. What do you tell them to change first?
What do you do the third time the same person brings you the same class of bug?