Interview guides · Crary Industries
Crary Industries Software Engineer Interview Questions & Guide 2026
Take-home style practice for Crary Industries software engineer candidates: a small service, a fixed budget, a written tradeoff.
The 60-second brief
As a Software Engineer at Crary Industries, you are positioned at the intersection of traditional manufacturing excellence and modern technological innovation. You support the software infrastructure behind Crary Industries' operational efficiency and product quality. Your work gives the company's engineering teams the reliable tools and systems they need to stay competitive in the agricultural and industrial sectors.
For a design discussion, a catalogue of architectures is worth less than the ability to turn a vague requirement into a data model and an API contract. A box diagram with no schema under it collapses at the first follow-up question.
Focus areas
- Reserve inventory without overselling under concurrent allocation
- Keep a site operating during a network partition
- Fold late, out-of-order scan events by event time
Overview
This role is both challenging and rewarding because it requires a bridge-building mindset. You will often collaborate with cross-functional teams, including mechanical and manufacturing engineers, to solve complex problems that go beyond pure code. Success in this position means you are not just writing software; you are actively contributing to the physical output and long-term success of Crary Industries.
While the interview process is often described as straightforward, you should prepare to clearly articulate how your technical skills translate into practical solutions for a manufacturing environment.
Where to focus your preparation
Treat the sections below as preparation areas and confirm the format with your recruiter.
Preparation focus
No round sequence has been reported for this company, so work the categories below and confirm the format with your recruiter.
What to demonstrate
- Breadth across SQL, experimentation and product reasoning
- Ability to state assumptions before choosing a method
How to prepare
- Drill the practice exercises below and time yourself
- Prepare three quantified stories about decisions you drove
Where candidates lose points
Retrying an irreversible external call without an owned idempotency key
Timeouts and connection resets are ambiguous by construction - the partner may have committed before the response was lost - so a generic retry policy around an HTTP client is, in this domain, a machine for buying two labels and dispatching two trucks. Relying on the partner's deduplication is not a substitute, because their window is usually short, their key is often derived from fields you may legitimately change on retry, and many older integrations have no such concept at all. The workable pattern is to generate the key yourself, persist it with an explicit unknown status before the call, and resolve ambiguity by querying the partner for that key rather than re-issuing; the sweeper that does this is the component that has to be correct, not the call site. Teams get this right for payments and then forget that a carrier tender, a warehouse work release and an EDI shipping notice have the same shape.
Treating the database as the truth about physical stock
Every system in this domain drifts from reality, through miscounts, unrecorded damage, mis-scans and theft, so a design with no adjustment path forces the drift somewhere worse. The usual shape is a CHECK constraint or application guard that clamps a position at zero: the negative that would have exposed a missing receipt is silently swallowed, and the first visible symptom is a picker standing at an empty location for an order the system promised. The correct posture is to let the invariant fail loudly - record the cycle-count adjustment as a first-class movement with its own reason code, surface the variance, and let planning see that this location is unreliable - because an adjustment that is indistinguishable from a normal pick destroys the only signal that would have found the root cause. A related version of the same mistake is deleting or updating a movement row to fix a mistake, which leaves the position correct and the history a lie.
Issuing one query per row of a result set
Fetch related rows in a single batched query keyed by the ids you already hold, or join them into the original query. A per-row round trip multiplies network latency by the row count, and it looks perfectly fine against the ten rows in your development database.
Designing for a scale nobody asked for
Ask for request rate, data size, read-to-write ratio and expected growth, then size the simplest option first; one relational instance on current hardware covers a large share of real workloads. Reaching for shards, queues and a cache tier before any number has been quoted reads as pattern-matching rather than judgement.
Questions & practice
Split an order across the fewest nodes that can fill it
Hard · set cover · bitmask dp · approximation
A fulfilment order has L lines, L up to 25. After filtering by capability, service time and stock, N candidate nodes remain; N is usually under 14 but can reach 200. Each node covers a known subset of the lines in full. Return a minimum-size set of nodes covering every line, tie-broken by total distance. Give an exact algorithm for the small case with its exact cost in operations and bytes, say precisely why enumerating node subsets stops working, and give what you ship when N is 200.
Approach
- Name it: this is minimum-cardinality set cover, which is NP-hard, so the useful move is choosing which parameter you are exponential in rather than hunting for a polynomial exact algorithm. Enumerating subsets of nodes is 2^N and correct — at N = 14 that is 16,384 subsets and finishes instantly, which is exactly why it survives review and then fails in production at N = 200, where 2^200 is about 1.6e60.
- Be exponential in L instead, because L is the parameter you bound. Index the DP by the set of lines still uncovered:
dp[rem]is the cheapest way to cover the lines inrem, withdp[0] = (0, 0)and the value a(node_count, total_distance)pair compared lexicographically. Letbbe the lowest set bit ofrem; any cover must contain some node that stocks lineb, so the transition ranges only over those nodes:dp[rem] = min over v with b in cover(v) of dp[rem & ~cover(v)] + (1, dist(v)). Worst case O(2^L * N) — at L = 25 and N = 14 that is 33,554,432 states times 14, about 4.7e8 transitions — and the lowest-bit restriction cuts the real branching factor to the nodes stocking one specific line. - Quote the memory for what you actually store, not for a count array alone. Over 2^25 masks you need three dense arrays:
uint8node count (33.5 MB),uint32total distance (134.2 MB, integer metres so the tie-break compares exactly and the array stays fixed-width), anduint8chosen node (33.5 MB, since N <= 200 fits a byte) — about 201 MB, roughly six times the 33.5 MB that a bare count array would suggest. The same layout at L = 20 is 6.3 MB, which is why this is comfortable up to about 20 lines and needs a deliberate decision at 25. - Reconstruction is free in this formulation and needs no parent-mask array: because every transition eliminates the lowest uncovered line, the predecessor of
remunder the stored choicevis exactlyrem & ~cover(v), so walkremfrom the full mask down to 0 emittingchoice[rem]. If the reachable state count is far below 2^L — coverage sets are usually highly correlated — a top-down memoized recursion over a hash map trades the 201 MB for roughly 40-60 bytes per visited state; measure the visited count first, because a hash map that ends up touching most masks is strictly worse than the dense arrays. Add a branch-and-bound cut ofceil(popcount(rem) / max_v popcount(cover(v) & rem)), which prunes long before the theoretical bound bites. - When exact is out of reach, ship greedy with a stated guarantee: repeatedly take the node covering the most currently-uncovered lines, using bitset popcounts, O(N * ceil(L/64)) word operations per pick and at most min(N, L) picks. Greedy is within H(L) <= ln L + 1 of optimal (Chvatal), and no polynomial algorithm achieves (1 - o(1)) ln L unless P = NP (Dinur and Steurer, 2014) — so effort belongs in the pre-filter and the tie-break, not in a cleverer heuristic. At L = 25 the bound is H(25), about 3.82x worst case, while the realistic gap on correlated coverage sets is zero or one node.
- Close on the objective, because fewest nodes is not cheapest. Two shipments from distant nodes routinely cost more than three from near ones, and a split also costs a second box, a second carrier pickup and a worse customer experience. If cost is the real objective it is weighted set cover — greedy by cost divided by newly-covered lines, same logarithmic bound — and if a node can cover a line only partially, the line mask is no longer sufficient state, since you must track remaining quantity per line; that is the point at which you stop hand-rolling and either restrict splits or hand an MILP to a solver with a time budget.
Follow-up
- The pre-filter is what keeps N small. What is in it, and what happens to your exact path the day someone loosens it?
- One node can cover a line only partially. Show where the bitmask formulation breaks and what state replaces it.
- You have 40 milliseconds inside a checkout call. Which of these runs there, and what runs asynchronously afterwards?
Find the busiest ten-minute window per reader after deduplication
Medium · sliding window · event time · deduplication
One day of observation_event from fixed readers: source_id (reader), subject_id (container), occurred_at, received_at, dedupe_key. 300 million rows, partner and device replays resend whole windows, and some events arrive six hours after they occurred. For each reader, return the 10-minute event-time window containing the most distinct containers, with the window's start. Target linear work per reader after sorting. State the window semantics you chose, and what the job does with an event that lands after its window was already reported.
Approach
- Deduplicate before windowing, on
dedupe_key, and keep the two concerns separate. A replayed batch is a no-op only if the duplicate is removed first; deduplicating inside the window logic counts a resend as traffic and reports a congestion peak that never happened. Dedupe is a hash set per reader-day or a sort-unique on the key — the same sort you need anyway. - Partition by
source_idand sort each partition byoccurred_at, never byreceived_at. Arrival order is an artefact of device connectivity and partner batch cadence; a reader that went offline for six hours would otherwise show its entire backlog as one impossible burst. Sorting dominates: O(p log p) per reader of size p, O(n log n) overall. - Two pointers over the sorted partition with a hash multiset
container -> countand a separatedistinctinteger. Advance the right pointer, incrementing the count and bumpingdistinctonly on a 0-to-1 transition; advance the left pointer whileoccurred_at[left] <= occurred_at[right] - 600s, decrementing and loweringdistinctonly on a 1-to-0 transition, and erasing the key from the map so the map's size anddistinctnever diverge. Each element enters and leaves once: O(p) amortised after the sort, O(window occupancy) space. - Justify the candidate set rather than scanning every possible start. With half-open windows
[t, t+600), sliding a window right until its left edge coincides with an event can only add events and never removes one, so the maximum is attained at some event'soccurred_at; checking the p windows anchored at events is sufficient and exhaustive. State the half-open convention explicitly — an event exactly att+600belongs to the next window, and flip-flopping on that produces off-by-one disagreements between this job and whatever reads its output. - Handle lateness by choosing a policy, not by hoping. Either run over a closed event-time batch with a stated lateness allowance — the day plus six hours, so the six-hour tail is inside the batch and the day-late tail is not — or emit under a watermark and restate the affected windows when a late event lands. Both are defensible; what is not defensible is silently dropping the late event, because that is how the busiest window quietly becomes the one with the best connectivity.
Follow-up
- Two events for one container arrive from two readers a second apart. Is that congestion, a misread, or one container passing two portals?
- You need this continuously rather than as a daily batch. What changes, and where does the memory go?
- A device's clock is three hours fast. What does that do to this result, and how would you detect it from
clock_offset_ms?
Size dock doors from overlapping appointment windows
Medium · sweep line · intervals · time zones
For one node and one local calendar day you have up to 200,000 planned dock appointments from shipment_leg: leg_id, planned_arrive_at, planned_depart_at (both TIMESTAMPTZ), plus the node's IANA zone. A door is occupied over [arrive, depart). Return the minimum number of doors that lets every appointment start on time, and the maximal interval over which that peak is sustained. Target O(n log n). Say how you derive the day's boundaries and how you break ties between an arrival and a departure at the same instant.
Approach
- Name the result before computing it: with one interchangeable door type, the minimum door count equals the maximum number of simultaneously occupied intervals. That equality is not a heuristic — interval graphs are perfect, so their chromatic number equals their clique number, and greedy left-to-right assignment achieves it. Say this, because it is what licenses solving a scheduling question with a counter.
- Emit 2n endpoints,
(t, +1)at each arrival and(t, -1)at each departure, sort byt, and at equaltorder the-1before the+1. Half-open occupancy means a trailer leaving at 10:00 frees the door for one arriving at 10:00; ordering arrivals first inflates the answer by exactly the number of back-to-back handoffs, which on a well-packed schedule is most of them. - Scan once with a running counter, tracking the maximum and the endpoint index where it was first reached. The peak is attained on a half-open interval between two consecutive endpoints,
[t_i, t_{i+1}), not at an instant — report it that way or the operations team cannot act on it. Extend the interval while the counter stays at the maximum. - Complexity: O(n log n) dominated by the sort, O(n) space. If rows already arrive ordered by
planned_arrive_atfrom an index, use the min-heap variant instead — push each departure, pop all departures at or before the current arrival, and the heap size is the current occupancy. Same time bound, O(peak) space rather than O(n), which matters when peak occupancy is 40 and n is 200,000. - Derive the day's boundaries from the zone, not from arithmetic on instants. Convert local midnight and the next local midnight to instants in the node's IANA zone; on a transition day that span is 23 or 25 hours, so adding 86,400 seconds silently drops or duplicates an hour of appointments. Decide explicitly what a zero-length dwell means — under half-open semantics it occupies nothing — and state it.
Worked solution (25 min)
- Write the endpoint tuple with the tie-break baked into the sort key — for example
(t, kind)where departure sorts as 0 and arrival as 1 — rather than handling ties in the scan body. - Implement the scan, returning both the maximum and the
[t_i, t_{i+1})span at which it holds. - Build a fixture of six appointments where three are strictly back-to-back at the same minute, and hand-compute the answer before running.
- Add an appointment spanning the local DST transition and derive the day window from the zone, asserting the appointment count inside the window.
- Re-run with the input shuffled to confirm the result depends only on the set of intervals.
Expected result: The back-to-back fixture needs one door, not three; the reported peak is a half-open interval between two endpoints; on the transition day the local window is 23 or 25 hours and no appointment is dropped or double-counted.
Verify it yourself
- Flipping the tie-break to arrivals-first changes the answer on the back-to-back fixture — if it does not, the fixture is not actually testing the tie.
- The maximum equals the largest number of intervals containing any single probe instant, checked by brute force on a small fixture.
- The result is identical under any permutation of the input.
Follow-up
- Doors are typed: some break pallets, some are parcel-only. Does max-overlap still equal the door count?
- Appointments have a tolerance — a trailer may start up to 20 minutes late without penalty. How does that change the objective, and is it still solvable by a sweep?
- Half the appointments are actuals and half are plans. Which do you sweep for tomorrow's staffing, and which for last week's utilisation report?
Write the reconciliation query that finds ledger and position variance
Easy · aggregation · outer joins · reconciliation
inventory_movement(movement_id, item_id, node_id, lot_id, state, delta_qty) is the ledger; inventory_position(item_id, node_id, lot_id, state, qty, version, last_movement_id, reconciled_through_movement_id) is its running sum, written in the same transaction as each movement. Write one query that reports every key whose stored qty disagrees with the sum of its movements, including keys present in only one of the two tables, bounded by a watermark movement_id you pass in. State what the job does with each variance, and how you avoid re-summing four billion rows every night.
Approach
- Aggregate once and join once: a CTE doing SELECT item_id, node_id, lot_id, state, SUM(delta_qty) AS ledger_qty FROM inventory_movement WHERE movement_id <= :w GROUP BY 1,2,3,4, then FULL OUTER JOIN inventory_position USING those four columns. One pass plus one hash aggregate, O(n) in movements scanned; a correlated subquery per position row instead costs one index scan per key and is orders of magnitude slower at 40 million keys.
- Use FULL OUTER JOIN, not LEFT. A key with movements and no position row is the signature of a lost position update, and a position row with no movements is the signature of a write that bypassed the ledger. Both disappear under a LEFT JOIN from either side.
- Compare with IS DISTINCT FROM and COALESCE the join keys out of both sides, so a missing row reads as a variance rather than as NULL = NULL evaluating to unknown and being filtered away.
- Run both sides in one snapshot: either a single statement, or a REPEATABLE READ transaction. Reading positions in one transaction and movements in another manufactures variances for every key written in between.
- Make it incremental with reconciled_through_movement_id: sum only movements in (reconciled_through, :w] and add that delta to the previously verified total. The catch is that sequence values are handed out before commit, so a movement whose id is below :w can become visible after the scan and would then never be re-summed. Set :w to a max(movement_id) observed a few minutes earlier, beyond the longest write transaction, and full-scan one partition per night on a rotation as a backstop.
- A variance is reported, not repaired. Rewriting qty to match the ledger hides the double-applied retry that caused it; only a physical count may produce a cycle_count_adjust movement, and the location should be flagged unreliable to planning until it is counted.
Worked solution (25 min)
- Write the aggregate CTE with the watermark predicate, then the FULL OUTER JOIN and the IS DISTINCT FROM filter, projecting COALESCE(m.item_id, p.item_id) and both quantities.
- Seed a fixture with four cases: agreeing key, disagreeing key, movements with no position row, position row with no movements.
- Run it and confirm exactly three rows come back, with the agreeing key absent.
- Rewrite the same comparison as an incremental sum from reconciled_through_movement_id and confirm it reports the same three keys on a second run after new movements land.
Expected result: A result set of (item_id, node_id, lot_id, state, ledger_qty, stored_qty, ledger_qty - stored_qty) containing one row per disagreeing key and both orphan cases, and zero rows on a consistent fixture.
Verify it yourself
- Swap the FULL OUTER JOIN for a LEFT JOIN and confirm the movements-only orphan vanishes from the output.
- Replace IS DISTINCT FROM with <> and confirm the rows with a NULL on one side stop being reported.
- Insert a movement while the query runs and confirm the result does not change, proving both sides came from one snapshot.
- Compare the incremental result against a full re-sum of the same partition and require an exact match.
Follow-up
- The job finds 14 variances, all on one node, all negative. What do you look at first, and what would make you suspect the reconciliation job itself rather than the data?
- How do you run this without the nightly scan competing with the ATP read path for buffer cache and I/O?
- What does the query return for a key whose only movements are a receipt and its exact reversal, and is that row worth reporting?
Model the movement ledger so a retried write is a no-op
Easy · append-only ledger · idempotency · audit history
inventory_movement is append-only: movement_id, item_id, node_id, lot_id, state, delta_qty (signed BIGINT), uom_code, reason_code, ref_type, ref_id, idempotency_key, occurred_at, recorded_at, reversed_by. inventory_position is keyed (item_id, node_id, lot_id, state) with qty, version, last_movement_id. Give the DDL constraints and the exact statements a pick confirmation runs so that a client which times out after the commit and retries changes nothing. Then say how you correct a pick recorded against the wrong lot two days ago, and why the position carries CHECK (qty >= 0) rather than clamping at zero.
Approach
- Put the guarantee in the schema, not the handler: UNIQUE (idempotency_key) on inventory_movement. A SELECT-then-INSERT cannot work, because two concurrent retries both read nothing before either commits and both then insert.
- Claim with INSERT ... ON CONFLICT (idempotency_key) DO NOTHING RETURNING movement_id, and branch on rowcount. Zero rows means the first attempt already committed, so the handler returns success without touching the position. This branch is the whole point: applying the position update on the conflict path is what makes the ledger and the shelf disagree by one pick forever.
- Keep the insert and the position update in one transaction: UPDATE inventory_position SET qty = qty + :delta, version = version + 1, last_movement_id = :id WHERE item_id = :i AND node_id = :n AND lot_id = :l AND state = :s AND version = :v, then test the rowcount and retry the transaction on zero.
- Use lot_id BIGINT NOT NULL DEFAULT 0 rather than NULL. In a default unique index NULLs are distinct from each other, so a NULL lot would let two position rows exist for the same untracked item and split the running sum in half.
- Correct the mis-keyed pick with two compensating movements, never an UPDATE or a deleted_at flag: one reversal restoring the wrong lot, one fresh pick against the right lot, both with reason_code = 'reversal' or 'pick' and ref pointing at the original movement_id. The original row keeps its values; reversed_by is a back-pointer written in the same transaction so a reader does not need a reverse scan to see that the row was undone.
- CHECK (qty >= 0) aborts the transaction and surfaces a real defect. GREATEST(qty + :delta, 0) writes a plausible number instead, so the missing receipt that caused the negative is never investigated and the first symptom is a picker at an empty location.
Follow-up
- Two retries of the same pick arrive concurrently on different application instances. Walk through both transactions statement by statement and say which one commits a position change.
- occurred_at from a handheld is five minutes ahead of recorded_at because the device clock drifted. Which of your queries break, and which column should each of them actually be using?
- How long do you keep idempotency_key unique for, given the table grows by tens of millions of rows a month?
Would you be able to come in for an interview this Thursday?
Medium
Approach
- Say what you would check first and why it is the highest-information step.
- Work from the requirement backwards to the design.
- Clarify what is being asked and what a complete answer contains.
Follow-up
- How would you know your answer was wrong?
- What assumption would you test first?
Available-to-promise reads for a cart across candidate nodes
Easy · caching · read fan-out · invalidation · degradation
The checkout page asks for availability of up to 20 lines against up to 30 candidate nodes, 3,000 times per second at peak, inside a 25 ms p99 budget. Availability at a node is that node's on_hand position minus the outstanding held and committed allocations against it. Design the read path: what is cached, keyed how, invalidated by what, and what the service returns when the inventory primary is unreachable. State which direction the error must go and why.
Approach
- Size the fan-out before choosing anything: 20 lines by 30 nodes is 600 (item, node) pairs per call and 1.8 million pair reads per second. Per-pair round trips are ruled out by tail amplification alone. If each lookup exceeds 20 ms one time in a hundred, the chance at least one of 600 does is 1 - 0.99^600, about 99.8%, so the call's p99 becomes the slowest of 600. One batched multi-get, or one query with a composite IN list, per call.
- Fix the cache entry: key (item_id, node_id), value {on_hand_qty, outstanding_alloc_qty, position_version}. Availability is computed at read time from those two numbers, so a movement and an allocation status change invalidate the same key rather than two different ones.
- Invalidate by version, not by TTL alone. The Availability Service publishes (key, version) on every committed write; the cache applies it as a compare-and-set that discards a version lower than the one held. That makes out-of-order invalidations safe, and a lost invalidation message costs staleness bounded by a 1-2 second backstop TTL instead of permanent divergence.
- State the contract out loud: this path is advisory. The authoritative check is the conditional allocation write against the position row. Cache staleness therefore sets the rate of promise-then-retry, not the rate of overselling, and that is the whole reason a cache is permissible in front of inventory at all.
- Degrade deliberately. With the primary unreachable, serve last-known values minus a safety buffer and collapse the response to a coarse bucket (in stock / low / out) so no surface can quote an exact count from stale data. Err toward under-promising: the cost is a lost sale, against a short ship, a re-source and a second shipment's freight on the other side.
Worked solution (25 min)
- Compute the per-call fan-out and the resulting pair-read rate, then decide batched versus per-pair from the tail-amplification arithmetic rather than from taste.
- Write the cache entry layout and key, and name the exactly two events that must invalidate it: any movement on that position, and any allocation entering or leaving held/committed.
- Write the invalidation message shape and the compare-and-set rule that makes an out-of-order invalidation harmless.
- Write the degraded response shape, the safety-buffer rule, and one sentence stating the direction of error and its cost on each side.
Expected result: One batched lookup per ATP call over roughly 600 keys; a per-(item, node) cache entry carrying position_version and outstanding allocation; version-stamped invalidation with a 1-2 second backstop TTL; an explicitly advisory contract where the allocation write re-checks under a lock; and a degraded mode that buckets availability and under-reports rather than quoting an exact stale count.
Verify it yourself
- The design names the fan-out number and justifies the batching decision, rather than saying 'add a cache'.
- An invalidation arriving out of order cannot resurrect an older quantity.
- A dropped invalidation message degrades to bounded staleness, not to permanent divergence.
- No path in the read design can oversell on its own; the only write that changes a promise is the conditional allocation.
Follow-up
- A promoted item invalidates its 30 node keys every 50 ms. What stops the invalidation stream from evicting entries faster than they can be refilled, and what does a read do on a miss for that key?
- The candidate node set spans two regions. Does the read cross the region boundary, and what does that do to the 25 ms budget?
- How do you measure whether the safety buffer is set correctly, rather than guessing it?
Two committed allocations against a position holding one unit
Hard · concurrency · compare-and-set · isolation levels
Six oversells in a week, all on the twenty hottest item-node pairs, all inside promotion minutes. In one case inventory_position.qty was 1 and two allocation rows exist with qty 1 and status 'committed', created 4 ms apart; both transactions committed cleanly. The code reads availability, then issues UPDATE inventory_position SET qty = qty - :n, version = version + 1 WHERE item_id = :i AND node_id = :d AND lot_id = :l AND state = 'on_hand' AND version = :v, then inserts the allocation. A 200-thread load test across 50,000 SKUs never reproduces it. Find the defect and prove it.
Approach
- Reconstruct from the data before reading the code. Pull inventory_movement and allocation for that key ordered by identifier with the version values involved. Two commits that both proceeded from version v prove the compare-and-set did not gate anything; a single row correctly stepping v then v+1 with an oversell anyway would instead send you to the read path.
- Name the defect: the affected-row count of the conditional UPDATE is never tested. When version has moved the UPDATE matches nothing and affects zero rows, but control flow continues into the INSERT and the transaction commits, so the WHERE version = :v clause is decoration. A compare-and-set only protects you if the zero-row outcome changes what happens next.
- Explain why the load test is silent. With 200 threads spread over 50,000 SKUs, the chance that two allocators land on one row inside the UPDATE's execution window is negligible, while production traffic concentrates on twenty rows. Pin the same test to a single SKU with a position of 1 and N threads, assert that exactly one allocation survives, and it fails on the first run. That test is the proof, and it belongs in the suite.
- Rule out the decoy rather than ignoring it. If availability is read from a replica, lag peaks exactly when write volume does, so the read is often stale. But a stale read only mis-sizes the attempt: with a working compare-and-set the write still fails and retries, so the replica is a latency and retry-rate problem, not the oversell. Say that explicitly instead of fixing two things and learning nothing.
- Fix and then choose the mechanism deliberately. Test the rowcount; on zero rows re-read the position from the primary and retry with bounded attempts and jittered backoff, and never reuse the version captured before the failed attempt. Compare-and-set is lock-free but burns retries under contention on a single hot row; SELECT ... FOR UPDATE inside a short transaction queues writers predictably and is usually calmer for twenty hot keys; above that, funnel allocations for one item-node key through a single writer and accept the added latency.
- State what the isolation level does and does not do, since identical code is not equally safe everywhere. On PostgreSQL READ COMMITTED, which is the default and what this deployment runs, the second UPDATE blocks on the row lock, re-evaluates its predicate against the newly committed row, matches nothing and affects zero rows; control flow falls into the INSERT and the transaction commits, and that is the oversell. InnoDB's REPEATABLE READ ends the same way for this statement, because an UPDATE takes a current read rather than the snapshot read: it blocks, applies against the latest row, affects zero rows and does not abort. PostgreSQL REPEATABLE READ is the one that does not oversell, and not because the code is right: it raises SQLSTATE 40001 on a row a concurrent transaction updated after the snapshot, which aborts the whole transaction, so the unchecked INSERT can never commit. That is failing closed on an exception no handler anticipated, on one of the three configurations, at the cost of an error the caller has to catch and retry. Two of three commit the oversell, so the rowcount check is not optional, and the invariant belongs in a continuous monitor as well as in a test: assert per key that the sum of allocation.qty in status 'held' or 'committed' does not exceed the position quantity, and let the reconciliation job re-sum movements into positions and raise a variance when they disagree.
Follow-up
- An allocation must span two positions atomically. What changes in your mechanism, and what if the two live in different shards?
- Under the retry design, what stops a hot key from livelocking when fifty allocators all retry together?
- The six oversells are already promised to customers. What is the compensating path, and what does it write to inventory_movement?
Suggested practice checklist
For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.
Day 1 — Numbers before diagrams
- Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
- Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
- Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.
Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.
Day 2 — One system, from requirements to schema
- Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
- Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
- Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.
Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.
Day 3 — The consistency you are actually buying
- Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
- Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
- Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.
Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.
Day 4 — Failure is the design
- For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
- Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
- Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
- Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.
Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.
Day 5 — Scaling the hot path
- Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
- Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
- Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
- Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.
Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.
Day 6 — Keep the coding hand in, at the bar that applies to you
- Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
- Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
- Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.
Deliverable: One problem hardened to review standard, with its test list and one written review comment.
Day 7 — Defend it while being interrupted
- Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
- Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
- Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.
Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.
The failure that was yours
Keep one story where the bad call was yours rather than a dependency's or a manager's. Name the check that would have caught it, whether you added that check afterwards, and whether it has fired since. Answers that route blame outward end the conversation early; answers that end in a guardrail someone still relies on tend to open it up.
Where do you see yourself in 5 years?
Medium · behavioural and engineering judgement
Approach
- Close with what you would do differently, concretely.
- Name the disagreement and how you resolved it with evidence.
- Pick a story where you made the decision, not one where you watched it.
Follow-up
- What would you do differently if you ran that again?
- How did you know your change caused the improvement?
Can you walk us through the projects you highlighted on your resume?
Medium · behavioural and engineering judgement
Approach
- Name the disagreement and how you resolved it with evidence.
- Pick a story where you made the decision, not one where you watched it.
- Give the blast radius: what could have broken, and what you measured.
Follow-up
- How did you know your change caused the improvement?
- What did you decide not to do, and why?
What specific opportunities or experiences from your background make you a good fit for this department?
Medium · behavioural and engineering judgement
Approach
- State the situation in two sentences and spend the rest on the reasoning.
- Name the disagreement and how you resolved it with evidence.
- Give the blast radius: what could have broken, and what you measured.
Follow-up
- What would you do differently if you ran that again?
- What did you decide not to do, and why?