Interview guides · Alexander Dennis
Alexander Dennis Software Engineer Interview Questions & Guide 2026
Caching practice for Alexander Dennis software engineer prep: settle the staleness budget before choosing an invalidation strategy.
The 60-second brief
A Software Engineer at Alexander Dennis (ADL) occupies a pivotal role at the intersection of traditional automotive engineering and modern digital systems. As a world-leading manufacturer of buses and coaches, Alexander Dennis relies on its engineering teams to integrate complex electrical and mechanical systems that power the next generation of public transportation. Your work directly impacts the reliability, efficiency, and performance of the vehicles that millions of people use daily.
Treat capacity estimation as a conversion skill rather than a table to memorise: turn a user count and an action rate into requests per second and bytes per day, then name the component that number breaks first. The figure only matters if it changes the design.
Focus areas
- Reserve inventory without overselling under concurrent allocation
- Make every external side effect safe to retry
- Fold late, out-of-order scan events by event time
Overview
This role is not merely about writing code; it is about ensuring that software-driven components are seamlessly integrated into the physical build of the vehicle. Whether you are working in a Build Support capacity focusing on electrical or mechanical systems, you will be responsible for bridging the gap between design intent and factory-floor execution. Success requires a blend of deep technical curiosity, a pragmatic approach to problem-solving, and the ability to thrive in a fast-paced, industrial manufacturing environment.
The Alexander Dennis interview process
Alexander Dennis candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.
Online Application
What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.
What to demonstrate
- Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
- Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
- Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly
How to prepare
- Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
- Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
- Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live
Screening Phase
You cannot drill a format you do not know, so put the preparation into material that travels. Three pieces of your own work, each rehearsed until you can take a follow-up you did not anticipate, will carry a conversation or a code walkthrough equally well. Specificity is what separates that from filler. A number needs its definition before it means anything: a p99 is over some window and measured at some hop, and a server-side figure excludes the queueing and network time a client would see. The number you cannot qualify is the one to leave out.
What to demonstrate
- Whether your examples carry detail only someone who did the work would hold, such as what the binding constraint actually was, which alternative you rejected and why it was worse, and what you measured on each side of the change
- Whether a number survives one follow-up, meaning you can say what it was measured over and whether it moved because of your change or merely alongside it
- Whether a failure is described with the specific change that followed it, rather than a lesson stated in general terms
- Whether your part in a team effort is stated accurately, including what other people did
How to prepare
- Write a page on each of three projects covering the constraint, the option you rejected, the measurement before and after, and what went wrong. Cut any line you cannot take a follow-up on, since you are writing the parts you will be pressed on rather than a summary.
- Recover the real figures while you still have access: request volume, data size, latency with its percentile and window, team size, timeline. Note where each came from, whether a dashboard, a design document or memory, and mark the estimates so you can say which they are out loud.
- Take your weakest project story to someone who works in a different area and have them ask why four times in succession. The point where you run out of answer is the part to go and re-read before the round.
Video Interview
When a round has no standard shape, it is often there because something is still open: an area no earlier conversation reached, a round where the signal came out mixed, or a decision someone is not ready to make alone. Work out which by going back over what each earlier round actually covered rather than how it felt, and arrive able to give evidence on that point without being asked twice. Weak answers replay the loop's earlier material at the same depth. Strong ones go a level deeper and stay consistent with what you already said.
What to demonstrate
- Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
- Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
- Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
- Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened
How to prepare
- Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
- Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
- Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure
Where candidates lose points
Judging delivery windows and cut-offs in UTC
A delivery appointment, a receiving cut-off and a same-day promise are all statements about local wall-clock time at a specific stop, so evaluating them in UTC misclassifies on-time performance for any leg that crosses a zone, and misses cut-offs by an hour on the two days a year a region changes offset. The subtlety that catches experienced engineers is that converting a future local time to a UTC instant at write time bakes in the offset rules as they were known then: when a jurisdiction changes its rules, an appointment stored that way silently moves, which is why a future window is better stored as local wall time plus an IANA zone identifier and resolved at read time, while a past event is correctly stored as an instant. Duration arithmetic needs the same care, because a transit that spans a DST boundary is genuinely 23 or 25 hours long in local terms even though the elapsed instant count is unchanged. Storing only a date with no zone - common on partner feeds - is the version of this that cannot be fixed downstream, so the zone has to be captured at ingestion.
Treating the database as the truth about physical stock
Every system in this domain drifts from reality, through miscounts, unrecorded damage, mis-scans and theft, so a design with no adjustment path forces the drift somewhere worse. The usual shape is a CHECK constraint or application guard that clamps a position at zero: the negative that would have exposed a missing receipt is silently swallowed, and the first visible symptom is a picker standing at an empty location for an order the system promised. The correct posture is to let the invariant fail loudly - record the cycle-count adjustment as a first-class movement with its own reason code, surface the variance, and let planning see that this location is unreliable - because an adjustment that is indistinguishable from a normal pick destroys the only signal that would have found the root cause. A related version of the same mistake is deleting or updating a movement row to fix a mistake, which leaves the position correct and the history a lie.
Naming no test cases at all
State what you would test before being asked: empty input, a single element, all elements equal, the maximum permitted size, and the input that exercises the branch you just wrote. It costs thirty seconds and is much of what separates someone who has shipped code from someone who has only solved puzzles.
Hardcoding to the sample inputs
Solve the stated problem rather than the two examples; special-casing a literal to make a sample pass is obvious immediately and reads as either a misunderstanding or an attempt to fake progress. If you genuinely cannot generalise yet, say which part is a stub and what would replace it.
Questions & practice
Trace a suspect lot forward to every shipped order line
Medium · graph traversal · genealogy · indexing
lot_edge(parent_lot_id, child_lot_id) has 400 million rows with btree indexes on both (parent_lot_id, child_lot_id) and (child_lot_id, parent_lot_id). lot_shipment(lot_id, order_line_id) maps lots to shipped lines. Given one suspect received lot, return every downstream order_line_id. The reachable subgraph can reach 2 million lots and 12 levels; the full edge table does not fit in memory. Give the traversal, its complexity in terms of the reachable subgraph, and what you do if the data contains a cycle.
Approach
- Genealogy is a directed acyclic graph, not a tree: a production run consumes several lots and one lot splits across many shipments, so a node is reached by several paths and a tree traversal revisits it combinatorially. Breadth-first from the suspect lot over the
(parent_lot_id, ...)index, with a hash set of visited lot ids, is O(V' + E') where V' and E' count only the reachable subgraph — 2 million 8-byte ids is 16 MB of payload and a few tens of MB with open-addressing overhead, which is why the visited set is affordable and the edge table's 400 million rows never enter memory. - Batch each level into one indexed read. Issue
WHERE parent_lot_id = ANY($1)with a few thousand parents per call instead of a query per node: at 12 levels and 2 million nodes, per-node round trips dominate the wall clock entirely, and the recall's deadline is a real-world response time, not a query-plan aesthetic. The batch size is the tuning knob between round-trip count and per-statement planning cost. - Keep the visited set even though the schema says DAG. Rework and repack loops — material returned into an earlier lot — do occur, and an unguarded traversal on a cyclic edge set does not terminate; a set membership test is O(1) and turns a hang into a correct answer plus a data-quality alert. Also cap the traversal by level and by node count, and fail loudly on the cap rather than returning a partial set silently.
- Join to
lot_shipmentonce at the end over every visited lot id, not over the leaves and not per node. A lot is routinely both shipped and consumed: half a lot goes out on an order line while the other half is repacked into a child lot, so it has alot_shipmentrow and outgoing edges at the same time and is not a leaf. Restricting the join to nodes with no outgoing edges drops exactly those order lines, which is under-recall — the one failure mode this trace is not allowed to have. One bulk join against an index onlot_idover the whole visited set is a single pass; per-node lookups multiply the round-trip problem by the number of visited lots. - State the semantics honestly: reachability answers which shipped lines could contain material from the suspect lot, and without propagating edge quantities it is a superset of physical containment. For a recall that is the correct direction of error — over-recall, never under-recall — and saying so is the point, because a trace that returns quickly and incompletely will be believed. The upstream question, which received lots are in a suspect shipped unit, is the same traversal over the reverse index, which is why both indexes exist.
Worked solution (30 min)
- Draw a six-lot fixture containing a merge (two parents into one child), a split (one parent into three children), and one interior lot that is partly shipped on an order line and partly consumed into a child lot, then write down the expected reachable set and the expected order lines by hand.
- Implement level-batched BFS with an explicit frontier list, a visited hash set, and a per-level batch read.
- Add a cycle to the fixture and confirm the traversal terminates and flags it rather than hanging.
- Bulk-join
lot_shipmentonce over every visited lot id — the whole visited set, not the leaves — and count the queries issued, confirming it is one per level plus one. - Re-run with the frontier order reversed to confirm the output set is unchanged.
Expected result: Every lot downstream of the suspect appears exactly once in the visited set; the merge node is visited once, not twice; the partly-shipped interior lot contributes its order line even though it has children; the cyclic fixture terminates with the same reachable set plus a flagged edge; the query count is levels + 1, not proportional to node count.
Verify it yourself
- The reachable set is a superset of the set you computed by hand, and equal to it on the acyclic fixture.
- Restricting the final join to lots with no outgoing edges drops the partly-shipped interior lot's order line — the failing test that pins the over-recall rule to code.
- Running the same traversal over the reverse index from a leaf returns the suspect lot.
- Removing the visited set makes the diamond fixture return duplicates — a failing test that documents why the set is there.
Follow-up
- A recursive CTE would express this in ten lines. What is its cost here if the index on the traversal column is missing, and how would you notice?
- You need the quantity of suspect material in each downstream line, not just reachability. What has to be on the edge, and what does that do to the traversal?
- The traversal must run while a production run is writing new edges. What does a consistent answer mean here, and what do you cut it at?
Parse partner quantity and date segments without floating point
Easy · parsing · unit-of-measure · time zones
A partner sends a delimited flat file whose segments look like QTY+17:4.500:CS and DTM+232:20260317. Quantities carry up to three decimal places in a named unit; each item has an integer pack factor to eaches that is versioned by effective date. Produce observation_event rows with an integer quantity in eaches, the unit recorded, and occurred_at resolved, rejecting any line whose conversion is not exact. Files reach 4 GB. Single pass, no floating point. State your bounds and what you do with the zoneless date.
Approach
- Scan with indices, not a regex. Find the segment terminator, then split the payload on
:by scanning forward and slicing — O(bytes) with no backtracking and no per-line allocation beyond the output row. A regex with nested quantifiers over 4 GB is where this job silently becomes CPU-bound. - Parse the decimal as an integer mantissa plus a scale.
4.500becomes(mantissa = 4500, scale = 3): accumulate digits into an int64, count digits after the point, and never call the runtime's float parser. Reject a mantissa that would overflow int64 when multiplied by the pack factor, checking before the multiply rather than after. - Convert by integer arithmetic and demand exactness:
numerator = mantissa * factor; emitnumerator / 10^scaleonly whennumerator % 10^scale == 0, otherwise reject the line with a reason code. Worked:4.500 CSat 12 eaches per case gives 54000/1000 = 54, exact.0.125 PLat 36 eaches per pallet gives 4500/1000 = 4.5, which is half a sealed unit and must fail at the boundary, not round three services away. - Select the pack factor by the file's effective date, not by the current item master. Factors are properties of a point in time — a supplier moving a case from 12 to 10 must not reinterpret last quarter's file — so the lookup is a range query on the factor's validity interval.
20260317carries no zone and no time. Attach the IANA zone of the node the segment refers to, captured at ingestion, because it cannot be recovered downstream; then pick and document one convention for a date-only field (start of local day is the usual choice) and store the raw local string beside the resolved instant so the assumption stays visible. Complexity: O(total bytes) time, O(longest line) working space, one pass.
Follow-up
- The same file is re-sent in full tomorrow with three lines changed. What makes the second ingest a no-op for the unchanged lines?
- A line names a unit that is not in the ladder for that item. Reject the line, reject the file, or quarantine it — and who finds out?
- The partner starts sending an offset like
+0100on some dates. Does that change your storage decision for a future appointment as well as a past event?
Re-sum a movement ledger and report positions that disagree
Easy · aggregation · reconciliation · append-only ledger
inventory_movement holds 3 billion immutable rows: item_id, node_id, lot_id (NULL when the item is lot-untracked), state, delta_qty (signed int64 in the item's smallest transacting unit), uom_code, and reversed_by. inventory_position holds one row per (item_id, node_id, lot_id, state) with qty, version and last_movement_id, using lot_id = 0 as the sentinel for untracked. Up to 80 million distinct keys. Report every key whose summed movements disagree with the stored qty, while writes continue. State your time and space bounds and how you bound the comparison.
Approach
- Fix the key first.
inventory_movement.lot_idis NULL for untracked items andinventory_position.lot_idis 0, so the group key is(item_id, node_id, COALESCE(lot_id, 0), state). Getting this wrong does not error — it silently produces two groups that each look like a variance, and the report becomes noise nobody reads. - Cut the ledger at a watermark, but do not take
W = max(movement_id)at the start of the scan. A sequence hands out an id before the inserting transaction commits, so at the instant you read that maximum there are ids below it still in flight and invisible to your snapshot. Summingmovement_id <= Wmisses them on this run, and advancingreconciled_through_movement_idtoWmakes the miss permanent: every later incremental run starts above those ids and they are never summed again. That is a hole in the ledger's own re-derivation, not a transient skew. - Take a watermark that is provably settled instead. Bound write transactions with a statement or transaction timeout so 'the longest write' is a number T, record
(observed_at, max_movement_id)samples periodically, and use asW_safethe largest sampled id whoseobserved_atis older than T — every id at or below it has committed or rolled back. Sum rows withmovement_id <= W_safe, compare only against position rows withlast_movement_id <= W_safe, treat anything newer as a write that raced you rather than a variance, and advancereconciled_through_movement_idonly toW_safeso the next run starts there instead of re-reading 3 billion rows. If the ledger carries a commit timestamp, cutting on that is the same guarantee without the sampling table. - Sum
delta_qtyas int64, and include reversal rows.reversed_byis a back-pointer for audit, not an exclusion filter: an original of +10 and its reversal of -10 must both be summed to reach 0. Excluding the original while keeping the reversal produces -10 and a false variance on every corrected key. - Guard the denomination rather than trusting it.
uom_codeis stored per row because pack factors change over time, so reject — do not sum — any row whoseuom_codeis not the item's smallest transacting unit, and report those keys separately. A mixed-denomination sum is arithmetically meaningless and looks exactly like a real variance. - Hash aggregation is O(n) time and O(distinct keys) space: 80 million entries at roughly 40-56 bytes each in a typical runtime is 3-5 GB, so quote the number. The bounded-memory alternative is an external sort-merge on the group key — O(n log n) comparisons, resident memory bounded by the merge fan-in rather than by key count, and it streams — or hash-partition by
hash(item_id) % Pand run P independent passes for 1/P of the peak.
Follow-up
- A key shows a variance of exactly one pick, repeatedly, at one node. What do you look at first, and what would distinguish a duplicate movement from a missed one?
- The job takes six hours and the variance report is stale by the time anyone reads it. How would you make it incremental without losing the guarantee that it re-derives from the ledger?
- Who writes the correcting movement, and what reason code does it carry?
Decide whether available-to-promise is derived or stored
Hard · denormalisation · read path · cache staleness
Available to promise at one node equals inventory_position.qty in state on_hand minus the sum of allocation.qty in status held or committed for that key. The read sits inside checkout: 4,000 reads per second at peak, p99 budget 20 ms. inventory_position holds 40 million rows; during a promotion a single hot item-node pair carries about 2,000 open allocations. Decide whether to compute this per read or to maintain available_qty on the position row. Give both paths, the failure mode of each, and what the browse read may do that the allocate read may not.
Approach
- Cost the derived read in tuples, not in plans. One index probe into the position is trivial; the aggregate over allocation touches one tuple per open claim on that key. Even if only a tenth of peak lands on the hot pair, 400 reads per second times 2,000 rows is 800,000 index tuples per second for one item at one node, and it gets worse exactly as the key gets hotter. A partial index on status IN ('held','committed') does not help, because the row count is the cost, not the selectivity.
- Maintaining available_qty on the position row is nearly free on the contention axis, which is the point most candidates miss. The allocator already takes a row lock or a compare-and-set on that same position row, so the extra column costs no lock that was not already held, and the read collapses to a single primary-key probe well inside 20 ms.
- It is not free on the correctness axis. available_qty becomes a second derived value that can drift, so it joins qty under the same reconciliation job with its own re-sum against the allocation table, and a drift alert on that comparison is the operational price of the read speed.
- Split the two reads rather than arguing about one. The browse read may serve from a replica or a short-TTL cache and must err low, showing less than is there. The allocate read must be on the primary and inside the same transaction as the conditional update, because replication lag peaks during the write burst that made the key hot. Conflating them is what makes a replica read look safe in a benchmark.
- Pick the TTL against the drain rate, with a number. If a promotion burns 2,000 units in a minute, a 30-second cache is a thousand units stale, so the displayed figure should be a banded signal such as in stock or low stock rather than an exact count that the allocate step will then contradict.
- Do not denormalise availability across nodes into a single network number. A yes-somewhere answer cannot be allocated against, so sourcing will disagree with the page, and the customer sees a promise retracted after checkout rather than an honest out-of-stock before it.
Follow-up
- Roll this out on a live system. How do both paths run side by side, what do you compare, and what result gates the cut-over?
- The reconciliation job finds available_qty drifted by 3 on one key. What do you do, and what must you not do?
- Lot-tracked items multiply the position rows per item-node. Does your answer change, and where does the read now spend its time?
Stop two allocators promising the same unit of stock
Medium · isolation levels · lock contention · compare-and-set
inventory_position is keyed (item_id, node_id, lot_id, state) and holds qty BIGINT, which is on-hand stock and is moved only by the movement ledger; available_qty BIGINT, maintained on the same row; and version. The declared invariant is available_qty = qty - SUM(allocation.qty) over that key in status held or committed, enforced by CHECK (available_qty >= 0) and CHECK (available_qty <= qty). allocation holds allocation_id, order_line_id, item_id, node_id, lot_id, qty, tier, status (held, committed, consumed, released, expired) and expires_at. Twelve allocator threads try to claim the last eight units of one item at one node inside the same 50 ms. Give the exact statements that let at most eight units be claimed, then say what changes if identical code runs under PostgreSQL REPEATABLE READ versus InnoDB REPEATABLE READ, and what a retried allocate for the same order line must do.
Approach
- Collapse the check and the write into one statement whose WHERE clause is the business rule: UPDATE inventory_position SET available_qty = available_qty - :q, version = version + 1 WHERE item_id = :i AND node_id = :n AND lot_id = :l AND state = 'on_hand' AND available_qty >= :q. qty is deliberately untouched: a hold is a promise, not a physical move, so only a movement may change on-hand and the ledger-equals-position invariant survives. Rowcount zero is a refusal, not an error, and the allocation row is inserted in the same transaction only when the rowcount is one.
- If the deployment carries no available_qty column and availability must be derived, the alternative is SELECT ... FOR UPDATE on the position row followed by the SUM over held and committed allocations inside the same transaction. That is correct but holds the row lock for two statements instead of one, which matters on a key taking hundreds of writes a second.
- Isolation is not a substitute for the atomic step, and it changes the error contract. Under PostgreSQL READ COMMITTED the losing UPDATE blocks, then re-evaluates its predicate against the newly committed row version and returns rowcount 0, which the handler already handles. Under PostgreSQL REPEATABLE READ the same statement instead raises SQLSTATE 40001 and the whole transaction must be retried, so a handler that only tests rowcount fails closed on an exception it never expected. InnoDB REPEATABLE READ does neither: a locking read or UPDATE performs a current read of the latest committed row, so the loser blocks and then sees rowcount 0, with a deadlock rather than a serialization failure as its exceptional case.
- Make the retry a no-op with a partial unique index: UNIQUE (order_line_id, node_id, lot_id) WHERE status IN ('held','committed'). Then a retried allocate conflicts instead of double-claiming, and the handler reads the existing allocation and returns it.
- Never source the availability number from a replica or a cache for this path. Replication lag is largest during the write burst that made the key hot, so the guarantee fails precisely when it is load-bearing.
- For the hottest item-node pairs, shed load without overselling: route allocations for that key through a single writer with a bounded queue and refuse with a conservative not-available when the queue is full. Splitting the position into N sub-rows cuts contention N-fold but strands the remainder across sub-rows and produces false out-of-stock when the last units are spread thin.
Worked solution (30 min)
- Write the conditional UPDATE, the rowcount branch and the allocation INSERT, plus the partial unique index DDL and the two CHECK constraints the invariant needs.
- Set a position to qty = 8 and available_qty = 8, launch 12 threads each claiming 1, and assert exactly 8 succeed, available_qty lands at 0 and qty is still 8.
- Re-run the same race under PostgreSQL REPEATABLE READ and record how many transactions fail with 40001 rather than returning rowcount 0.
- Send one allocate twice for the same order line and node, and assert one allocation row and the same allocation_id returned both times.
- Remove the available_qty >= :q predicate and confirm CHECK (available_qty >= 0) aborts the losing transactions, proving the test exercises the race rather than serialising by luck.
Expected result: Exactly 8 allocation rows in status held, available_qty = 0 with qty unchanged at 8, 4 refusals returned as a clean not-available response, and a duplicate allocate that returns the existing allocation without creating a second row.
Verify it yourself
- Repeat the 12-way race 50 times and require 8 successes on every run, not on average.
- Sample mid-run and require available_qty = qty - SUM(allocation.qty) over held and committed at every sample, with that sum never exceeding 8.
- Confirm the REPEATABLE READ run produces 40001 errors and that the handler retries rather than surfacing them.
- Drop the partial unique index and confirm the duplicate allocate now creates two rows.
Follow-up
- One order line needs two units from two different nodes, atomically. What do you do when those positions live in different shards and no distributed transaction is available?
- Soft-tier allocations expire. Show the sweeper statement, and say what stops it from releasing an allocation that is being committed at that instant.
- Under sustained contention your CAS retries. What is the retry budget, and what does the caller see when it is exhausted?
How would you manage the process of integrating software systems into a physical assembly?
Medium
Approach
- State your assumptions explicitly before working the problem.
- Clarify what is being asked and what a complete answer contains.
- Work from the requirement backwards to the design.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
Describe the process of understanding development requirements and executing the DV (Design Verification) test program.
Medium
Approach
- Work from the requirement backwards to the design.
- Clarify what is being asked and what a complete answer contains.
- Say what you would check first and why it is the highest-information step.
Follow-up
- How would you know your answer was wrong?
- What assumption would you test first?
What is your approach to troubleshooting issues that arise during the build phase?
Medium
Approach
- Say what you would check first and why it is the highest-information step.
- Clarify what is being asked and what a complete answer contains.
- Work from the requirement backwards to the design.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
Fan shipment status out to five thousand customer webhooks
Medium · fan-out · queues · backpressure · at-least-once
Customers subscribe to shipment status by webhook: 5,000 endpoints, some with an 8 second p99, some down for a day. The upstream is folded leg status, not raw observations. Requirements are per-shipment ordering at each subscriber, at-least-once delivery with a stable dedupe id, and no subscriber able to slow another. Assume 3 million legs a day, about six status transitions each, and a mean of three subscribers per shipment. Design the fan-out and quantify what folding first saves you.
Approach
- Fan out transitions, not observations, and do the arithmetic before designing delivery. Raw ingest averages 9,000 observations a second, about 780 million a day. Three million legs at six transitions each is 18 million notification sources a day, 208 per second, and three subscribers apiece is 54 million deliveries a day, 625 per second. Folding first is worth a factor of about 43 - 208 sources per second instead of 9,000, and 625 deliveries per second instead of roughly 27,000 - and that factor dominates every subsequent optimisation, so settle it before tuning delivery.
- Keep delivery state as a durable per-subscriber cursor over a per-subscriber queue rather than a row per attempt. At 54 million deliveries a day a row per attempt is roughly 19.7 billion rows a year before a single retry is counted, and retries only add to it; a cursor plus a bounded in-flight window gives retry and progress without any of them. Per-event rows earn their cost only where you must later prove a specific delivery in a dispute, so scope them to the subscribers who contract for that rather than to all 5,000.
- Preserve ordering deliberately: a bounded in-flight window greater than one can reorder. Give each subscriber's queue a small number of lanes keyed by hash(shipment_id), so order is strict within a lane and parallelism comes from lanes. Ordering is then a property of the key, not of a hopeful single-threaded consumer.
- Isolate subscribers with a per-subscriber concurrency cap drawn against a shared worker pool. By Little's law an 8 second endpoint receiving 20 notifications per second wants 160 workers in flight; without a cap it takes them from the pool and starves the other 4,999. The cap converts a slow subscriber into its own growing backlog instead of a shared outage.
- Handle the dead endpoint as a bounded liability: circuit-break after consecutive failures, stop attempting, let the cursor lag, and retain the queue to a fixed horizon such as 72 hours. At the horizon, advance the cursor, record a gap, and tell the subscriber to re-sync from the pull API. Unbounded retention for one dead customer is how the fan-out store becomes the next incident.
- Make at-least-once an explicit contract rather than an accident: every payload carries a stable event id and the leg's status_rank, so a customer can discard a duplicate or a stale retry on their side, and that requirement is documented rather than discovered.
Worked solution (30 min)
- Derive the notification rate from legs per day, transitions per leg and subscribers per shipment, set it beside the raw observation rate, and state the reduction factor that justifies folding first.
- Choose between per-event delivery rows and a per-subscriber cursor, and show how the chosen shape still preserves per-shipment ordering under retry.
- Set a per-subscriber concurrency cap and compute an 8 second endpoint's in-flight occupancy at its share of traffic.
- Write the circuit-breaker states, the retention horizon and the re-sync contract for a subscriber that returns after a day offline.
Expected result: Notifications sourced from folded transitions - about 208 per second and 625 deliveries per second, against 9,000 observations per second raw, a factor of about 43 - carried on per-subscriber queues with lanes keyed by shipment so ordering holds while lanes run in parallel; a per-subscriber concurrency cap bounding a slow endpoint's occupancy; a circuit breaker with a bounded retention horizon and a re-sync path; and an explicit at-least-once contract with a stable dedupe id.
Verify it yourself
- A subscriber that stops responding for 24 hours changes nothing for the other 4,999.
- Two notifications for one shipment cannot arrive at a subscriber out of order, including after a retry.
- Retention is bounded and there is a defined, documented behaviour when the horizon is exceeded.
- The volume reduction from folding first is quantified and the ratio matches the stated rates - about 43 times, 9,000 observations per second down to 208 transitions - rather than asserted.
Follow-up
- A customer demands exactly-once. What can you actually offer, and what must they implement to get the effect they want?
- One subscriber covers 40 percent of all shipments. What breaks first in a per-subscriber queue design, and what do you shard on instead?
- How do you replay a day of notifications for one subscriber after a bug on your side, without re-sending to the other 4,999?
Two committed allocations against a position holding one unit
Hard · concurrency · compare-and-set · isolation levels
Six oversells in a week, all on the twenty hottest item-node pairs, all inside promotion minutes. In one case inventory_position.qty was 1 and two allocation rows exist with qty 1 and status 'committed', created 4 ms apart; both transactions committed cleanly. The code reads availability, then issues UPDATE inventory_position SET qty = qty - :n, version = version + 1 WHERE item_id = :i AND node_id = :d AND lot_id = :l AND state = 'on_hand' AND version = :v, then inserts the allocation. A 200-thread load test across 50,000 SKUs never reproduces it. Find the defect and prove it.
Approach
- Reconstruct from the data before reading the code. Pull inventory_movement and allocation for that key ordered by identifier with the version values involved. Two commits that both proceeded from version v prove the compare-and-set did not gate anything; a single row correctly stepping v then v+1 with an oversell anyway would instead send you to the read path.
- Name the defect: the affected-row count of the conditional UPDATE is never tested. When version has moved the UPDATE matches nothing and affects zero rows, but control flow continues into the INSERT and the transaction commits, so the WHERE version = :v clause is decoration. A compare-and-set only protects you if the zero-row outcome changes what happens next.
- Explain why the load test is silent. With 200 threads spread over 50,000 SKUs, the chance that two allocators land on one row inside the UPDATE's execution window is negligible, while production traffic concentrates on twenty rows. Pin the same test to a single SKU with a position of 1 and N threads, assert that exactly one allocation survives, and it fails on the first run. That test is the proof, and it belongs in the suite.
- Rule out the decoy rather than ignoring it. If availability is read from a replica, lag peaks exactly when write volume does, so the read is often stale. But a stale read only mis-sizes the attempt: with a working compare-and-set the write still fails and retries, so the replica is a latency and retry-rate problem, not the oversell. Say that explicitly instead of fixing two things and learning nothing.
- Fix and then choose the mechanism deliberately. Test the rowcount; on zero rows re-read the position from the primary and retry with bounded attempts and jittered backoff, and never reuse the version captured before the failed attempt. Compare-and-set is lock-free but burns retries under contention on a single hot row; SELECT ... FOR UPDATE inside a short transaction queues writers predictably and is usually calmer for twenty hot keys; above that, funnel allocations for one item-node key through a single writer and accept the added latency.
- State what the isolation level does and does not do, since identical code is not equally safe everywhere. On PostgreSQL READ COMMITTED, which is the default and what this deployment runs, the second UPDATE blocks on the row lock, re-evaluates its predicate against the newly committed row, matches nothing and affects zero rows; control flow falls into the INSERT and the transaction commits, and that is the oversell. InnoDB's REPEATABLE READ ends the same way for this statement, because an UPDATE takes a current read rather than the snapshot read: it blocks, applies against the latest row, affects zero rows and does not abort. PostgreSQL REPEATABLE READ is the one that does not oversell, and not because the code is right: it raises SQLSTATE 40001 on a row a concurrent transaction updated after the snapshot, which aborts the whole transaction, so the unchecked INSERT can never commit. That is failing closed on an exception no handler anticipated, on one of the three configurations, at the cost of an error the caller has to catch and retry. Two of three commit the oversell, so the rowcount check is not optional, and the invariant belongs in a continuous monitor as well as in a test: assert per key that the sum of allocation.qty in status 'held' or 'committed' does not exceed the position quantity, and let the reconciliation job re-sum movements into positions and raise a variance when they disagree.
Follow-up
- An allocation must span two positions atomically. What changes in your mechanism, and what if the two live in different shards?
- Under the retry design, what stops a hot key from livelocking when fifty allocators all retry together?
- The six oversells are already promised to customers. What is the compensating path, and what does it write to inventory_movement?
Suggested practice checklist
For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.
Day 1 — Measure before reasoning
- Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
- Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
- Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.
Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.
Day 2 — References, copies, and the bugs they produce
- Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
- Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
- Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.
Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.
Day 3 — Types, once, in a language that checks them
- Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
- Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
- Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.
Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.
Day 4 — Concurrency, starting with what actually runs at the same time
- Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
- Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
- Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.
Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.
Day 5 — Debugging as a procedure rather than an instinct
- Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
- Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
- Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.
Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.
Day 6 — Tests that catch the bug you are about to write
- Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
- Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
- Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.
Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.
Day 7 — Debug something broken, out loud
- Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
- Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
- Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.
Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.
What the behavioural round is actually measuring
Every story you tell gets read for blast radius and judgement: what could have broken, who else it touched, what you knew at the moment you decided. Nobody can audit your code in an hour, so they audit your reasoning instead. Pick work where the call was genuinely yours and the consequences were real enough to remember.
Cite an example of team development you have been responsible for and how you measured its effectiveness.
Medium · behavioural and engineering judgement
Approach
- Close with what you would do differently, concretely.
- Name the disagreement and how you resolved it with evidence.
- Give the blast radius: what could have broken, and what you measured.
Follow-up
- What would you do differently if you ran that again?
- How did you know your change caused the improvement?
What specifically attracted you to this role within our organization?
Medium · behavioural and engineering judgement
Approach
- State the situation in two sentences and spend the rest on the reasoning.
- Pick a story where you made the decision, not one where you watched it.
- Give the blast radius: what could have broken, and what you measured.
Follow-up
- What would you do differently if you ran that again?
- How did you know your change caused the improvement?
Why do you want to work for Alexander Dennis?
Medium · behavioural and engineering judgement
Approach
- Give the blast radius: what could have broken, and what you measured.
- State the situation in two sentences and spend the rest on the reasoning.
- Pick a story where you made the decision, not one where you watched it.
Follow-up
- What did you decide not to do, and why?
- How did you know your change caused the improvement?