Interview guides · AuST Manufacturing

AuST Manufacturing Software Engineer Interview Questions & Guide 2026

Testing practice for AuST Manufacturing software engineer candidates: the boundaries worth covering, and why flakiness is a real defect.

The 60-second brief

As a Software Engineer at AuST Manufacturing, you sit at the critical intersection of high-level software architecture and physical production environments. Your work goes beyond writing code: it builds the digital systems that let the company's manufacturing processes run with precision, efficiency, and reliability. You will develop the control systems and software solutions that directly affect AuST Manufacturing's production output and operations.

Treat capacity estimation as a conversion skill rather than a table to memorise: turn a user count and an action rate into requests per second and bytes per day, then name the component that number breaks first. The figure only matters if it changes the design.

Focus areas

Make every external side effect safe to retry
Reserve inventory without overselling under concurrent allocation
Reconcile a movement ledger against derived positions

Overview

This role is essential because it bridges the gap between abstract software concepts and tangible manufacturing outcomes. You will work within a fast-paced environment where your solutions must be robust enough to handle the rigors of a production plant while remaining agile enough to adapt to new product requirements. If you enjoy solving complex problems where your code directly influences physical hardware and large-scale manufacturing workflows, this position offers a unique opportunity for high-impact engineering.

The AuST Manufacturing interview process

AuST Manufacturing candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Onsite Interview

Nobody in the room with you decides this. Interviewers typically write their rounds up separately, often before seeing anyone else's, and the outcome is settled later from those write-ups. A split panel gets resolved by whichever note carries specific evidence, so what you want out of each room is one concrete thing that person could write down: a bug you caught yourself, a trade-off you named, a decision you owned. The rest is arithmetic. The project you describe in a behavioural conversation is often the same system you sketched an hour earlier, and the two accounts have to agree.

What to demonstrate

Whether the scale, team size and timeline you attach to a project hold steady when that project resurfaces in a different round
Whether each interviewer leaves with a specific thing to cite rather than a general impression of competence
Whether a trade-off you defended in one round survives a challenge in another, instead of being quietly swapped for the answer the new interviewer seemed to want
Whether a question you have already answered earlier in the day gets the same answer at the same depth, without visible impatience

How to prepare

Write a one-page sheet per project fixing the figures you will quote — request volume, data size, team size, elapsed time, what broke — and say them aloud from the sheet until they come out identical every time
For each round on the schedule, decide in advance the one sentence you want in that person's notes, then check in a mock that you said it outright instead of leaving it to be inferred
Have someone ask you the same project question twice, an hour apart, and diff the two answers for numbers that moved or a trade-off that reversed

Team-Based Evaluations

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Where candidates lose points

Holding a database transaction open across a physical or third-party operation

It is natural to open a transaction, lock the position, call the rating or tendering API, and commit on the response, and it works perfectly until the partner's p99 goes from 200 milliseconds to thirty seconds. At that point every request holding a lock on a hot item-node pair queues behind it, the connection pool fills with transactions that are waiting on the network rather than on the database, and an unrelated service sharing the pool fails at the same moment. In this domain the effect is amplified because demand concentrates on a few hot rows during a promotion or a seasonal peak, exactly when partner latency is also degraded. The structural fix is to keep transactions short and local - commit the state change together with an outbox row, let a relay perform the external call, and reconcile asynchronously - accepting at-least-once delivery and making the effect idempotent rather than trying to stretch a transaction over something the database cannot roll back.

Ordering events by arrival rather than by event time

Offline handhelds, batch partner feeds and store-and-forward gateways deliver observations out of order as a matter of course, so a pipeline that folds whatever arrived last will flap a delivered shipment back to in transit and recompute on-time performance from the wrong facts. Message-broker ordering guarantees do not rescue this: per-partition ordering only holds within a partition, so unless the producer keys by the entity being tracked, two events for one leg can land on different partitions and be processed concurrently. The defence has two halves that are often confused - deduplicate on a stable key, then fold with a monotonic status lattice ordered by occurred_at - and both are needed, because deduplication alone still lets a stale event win. Note also that occurred_at is device-reported and therefore sometimes wrong, which is why storing received_at and a measured clock offset beside it is what makes event-time logic auditable instead of merely plausible.

Designing for a scale nobody asked for

Ask for request rate, data size, read-to-write ratio and expected growth, then size the simplest option first; one relational instance on current hardware covers a large share of real workloads. Reaching for shards, queues and a cache tier before any number has been quoted reads as pattern-matching rather than judgement.

Going silent while thinking

Narrate the candidates and why you are discarding them, even in fragments: sorting first would make this a two-pointer scan, but it destroys the original indices, which the output needs. From the other side of the table, a candidate thinking hard and a candidate stuck are indistinguishable until one of them speaks.

Questions & practice

Parse partner quantity and date segments without floating point

Easy · parsing · unit-of-measure · time zones

A partner sends a delimited flat file whose segments look like QTY+17:4.500:CS and DTM+232:20260317. Quantities carry up to three decimal places in a named unit; each item has an integer pack factor to eaches that is versioned by effective date. Produce observation_event rows with an integer quantity in eaches, the unit recorded, and occurred_at resolved, rejecting any line whose conversion is not exact. Files reach 4 GB. Single pass, no floating point. State your bounds and what you do with the zoneless date.

Approach

Scan with indices, not a regex. Find the segment terminator, then split the payload on : by scanning forward and slicing — O(bytes) with no backtracking and no per-line allocation beyond the output row. A regex with nested quantifiers over 4 GB is where this job silently becomes CPU-bound.
Parse the decimal as an integer mantissa plus a scale. 4.500 becomes (mantissa = 4500, scale = 3): accumulate digits into an int64, count digits after the point, and never call the runtime's float parser. Reject a mantissa that would overflow int64 when multiplied by the pack factor, checking before the multiply rather than after.
Convert by integer arithmetic and demand exactness: numerator = mantissa * factor; emit numerator / 10^scale only when numerator % 10^scale == 0, otherwise reject the line with a reason code. Worked: 4.500 CS at 12 eaches per case gives 54000/1000 = 54, exact. 0.125 PL at 36 eaches per pallet gives 4500/1000 = 4.5, which is half a sealed unit and must fail at the boundary, not round three services away.
Select the pack factor by the file's effective date, not by the current item master. Factors are properties of a point in time — a supplier moving a case from 12 to 10 must not reinterpret last quarter's file — so the lookup is a range query on the factor's validity interval.
20260317 carries no zone and no time. Attach the IANA zone of the node the segment refers to, captured at ingestion, because it cannot be recovered downstream; then pick and document one convention for a date-only field (start of local day is the usual choice) and store the raw local string beside the resolved instant so the assumption stays visible. Complexity: O(total bytes) time, O(longest line) working space, one pass.

Follow-up

The same file is re-sent in full tomorrow with three lines changed. What makes the second ingest a no-op for the unchanged lines?
A line names a unit that is not in the ladder for that item. Reject the line, reject the file, or quarantine it — and who finds out?
The partner starts sending an offset like +0100 on some dates. Does that change your storage decision for a future appointment as well as a past event?

Trace a suspect lot forward to every shipped order line

Medium · graph traversal · genealogy · indexing

lot_edge(parent_lot_id, child_lot_id) has 400 million rows with btree indexes on both (parent_lot_id, child_lot_id) and (child_lot_id, parent_lot_id). lot_shipment(lot_id, order_line_id) maps lots to shipped lines. Given one suspect received lot, return every downstream order_line_id. The reachable subgraph can reach 2 million lots and 12 levels; the full edge table does not fit in memory. Give the traversal, its complexity in terms of the reachable subgraph, and what you do if the data contains a cycle.

Approach

Genealogy is a directed acyclic graph, not a tree: a production run consumes several lots and one lot splits across many shipments, so a node is reached by several paths and a tree traversal revisits it combinatorially. Breadth-first from the suspect lot over the (parent_lot_id, ...) index, with a hash set of visited lot ids, is O(V' + E') where V' and E' count only the reachable subgraph — 2 million 8-byte ids is 16 MB of payload and a few tens of MB with open-addressing overhead, which is why the visited set is affordable and the edge table's 400 million rows never enter memory.
Batch each level into one indexed read. Issue WHERE parent_lot_id = ANY($1) with a few thousand parents per call instead of a query per node: at 12 levels and 2 million nodes, per-node round trips dominate the wall clock entirely, and the recall's deadline is a real-world response time, not a query-plan aesthetic. The batch size is the tuning knob between round-trip count and per-statement planning cost.
Keep the visited set even though the schema says DAG. Rework and repack loops — material returned into an earlier lot — do occur, and an unguarded traversal on a cyclic edge set does not terminate; a set membership test is O(1) and turns a hang into a correct answer plus a data-quality alert. Also cap the traversal by level and by node count, and fail loudly on the cap rather than returning a partial set silently.
Join to lot_shipment once at the end over every visited lot id, not over the leaves and not per node. A lot is routinely both shipped and consumed: half a lot goes out on an order line while the other half is repacked into a child lot, so it has a lot_shipment row and outgoing edges at the same time and is not a leaf. Restricting the join to nodes with no outgoing edges drops exactly those order lines, which is under-recall — the one failure mode this trace is not allowed to have. One bulk join against an index on lot_id over the whole visited set is a single pass; per-node lookups multiply the round-trip problem by the number of visited lots.
State the semantics honestly: reachability answers which shipped lines could contain material from the suspect lot, and without propagating edge quantities it is a superset of physical containment. For a recall that is the correct direction of error — over-recall, never under-recall — and saying so is the point, because a trace that returns quickly and incompletely will be believed. The upstream question, which received lots are in a suspect shipped unit, is the same traversal over the reverse index, which is why both indexes exist.

Follow-up

A recursive CTE would express this in ten lines. What is its cost here if the index on the traversal column is missing, and how would you notice?
You need the quantity of suspect material in each downstream line, not just reachability. What has to be on the edge, and what does that do to the traversal?
The traversal must run while a production run is writing new edges. What does a consistent answer mean here, and what do you cut it at?

Find the busiest ten-minute window per reader after deduplication

Medium · sliding window · event time · deduplication

One day of observation_event from fixed readers: source_id (reader), subject_id (container), occurred_at, received_at, dedupe_key. 300 million rows, partner and device replays resend whole windows, and some events arrive six hours after they occurred. For each reader, return the 10-minute event-time window containing the most distinct containers, with the window's start. Target linear work per reader after sorting. State the window semantics you chose, and what the job does with an event that lands after its window was already reported.

Approach

Deduplicate before windowing, on dedupe_key, and keep the two concerns separate. A replayed batch is a no-op only if the duplicate is removed first; deduplicating inside the window logic counts a resend as traffic and reports a congestion peak that never happened. Dedupe is a hash set per reader-day or a sort-unique on the key — the same sort you need anyway.
Partition by source_id and sort each partition by occurred_at, never by received_at. Arrival order is an artefact of device connectivity and partner batch cadence; a reader that went offline for six hours would otherwise show its entire backlog as one impossible burst. Sorting dominates: O(p log p) per reader of size p, O(n log n) overall.
Two pointers over the sorted partition with a hash multiset container -> count and a separate distinct integer. Advance the right pointer, incrementing the count and bumping distinct only on a 0-to-1 transition; advance the left pointer while occurred_at[left] <= occurred_at[right] - 600s, decrementing and lowering distinct only on a 1-to-0 transition, and erasing the key from the map so the map's size and distinct never diverge. Each element enters and leaves once: O(p) amortised after the sort, O(window occupancy) space.
Justify the candidate set rather than scanning every possible start. With half-open windows [t, t+600), sliding a window right until its left edge coincides with an event can only add events and never removes one, so the maximum is attained at some event's occurred_at; checking the p windows anchored at events is sufficient and exhaustive. State the half-open convention explicitly — an event exactly at t+600 belongs to the next window, and flip-flopping on that produces off-by-one disagreements between this job and whatever reads its output.
Handle lateness by choosing a policy, not by hoping. Either run over a closed event-time batch with a stated lateness allowance — the day plus six hours, so the six-hour tail is inside the batch and the day-late tail is not — or emit under a watermark and restate the affected windows when a late event lands. Both are defensible; what is not defensible is silently dropping the late event, because that is how the busiest window quietly becomes the one with the best connectivity.

Worked solution (30 min)

Write the window semantics as one sentence — half-open [t, t+600), anchored at event timestamps — at the top of the function, and make the comparisons match it.
Implement dedupe, sort, then the two-pointer scan with the multiset and the explicit distinct counter.
Build a fixture for one reader: twelve events, one container seen three times inside a window, one duplicate dedupe_key, and one event at exactly t+600.
Hand-compute the expected maximum and window start, then run.
Re-run with the fixture shuffled and with the duplicate resent five more times, confirming the answer is unchanged.

Expected result: The container seen three times contributes 1 to the distinct count; the resent duplicates contribute 0; the event at exactly t+600 falls in the next window; the reported start is an event timestamp, not a rounded clock boundary.

Verify it yourself

A brute-force O(p^2) scan over all event-anchored windows agrees on a 1,000-event fixture.
The distinct counter equals the multiset's key count at every step — assert it in the loop while testing.
Sorting by received_at instead of occurred_at changes the answer on a fixture with a backlogged device, which is the test that proves event time is doing work.

Follow-up

Two events for one container arrive from two readers a second apart. Is that congestion, a misread, or one container passing two portals?
You need this continuously rather than as a daily batch. What changes, and where does the memory go?
A device's clock is three hours fast. What does that do to this result, and how would you detect it from clock_offset_ms?

Attribute multi-stop freight cost without fanning it out

Medium · many-to-many joins · window functions · integer money

shipment_leg(leg_id, shipment_id, sequence_no, linehaul_cost_cents, accessorial_cost_cents, actual_arrive_at) joins to fulfilment_order_line through leg_order_line(leg_id, order_line_id, allocated_weight_g). One multi-stop truckload leg carries 40 order lines. The current report SUMs linehaul_cost_cents after joining all three tables and returns roughly forty times the real freight spend. Write the query that returns cost per order for one month, explain the fan-out in terms of the join's row count, and make the per-order amounts sum exactly to the leg totals in integer cents.

Approach

Name the cardinality: joining shipment_leg to leg_order_line produces one row per (leg, line) pair, so the result has SUM over legs of lines-per-leg rows rather than one row per leg. SUM(linehaul_cost_cents) over that result counts each leg's cost once per line it carries, which is why 40 lines gives 40 times the spend. The bug is not the SUM, it is aggregating a parent measure across a child-expanded row set.
Compute a share before aggregating anything. In one pass: leg.linehaul_cost_cents * b.allocated_weight_g / SUM(b.allocated_weight_g) OVER (PARTITION BY b.leg_id), which keeps the denominator local to the leg and needs no self-join.
Do the division in integer cents and settle the remainder deliberately. Floor each share, then distribute the leftover cents one each by descending remainder, ordered by a stable tiebreak such as order_line_id. Floating-point shares leave a residue that makes the freight accrual miss by a cent per leg, which costs more to investigate than the freight.
Aggregate to the order only after the share exists: GROUP BY order_id over the per-line allocated cents. Restrict the month on shipment_leg.actual_arrive_at, not on the order date, so a leg is attributed in the period it actually ran.
Index for the shape of the query: leg_order_line needs (leg_id) for the window partition and (order_line_id) for the reverse lookup, and shipment_leg needs an index on actual_arrive_at to bound the month without a full scan.
Apply the same treatment to accessorial_cost_cents separately. Accessorials such as a detention charge often belong to one stop rather than to the whole leg, so weight-sharing them across every line on the load misattributes cost even though the total ties out.

Follow-up

A line is cancelled after the leg ran. Does its share of the freight disappear, move to the remaining lines, or stay as a cost with no revenue?
Weight is the wrong basis for a light bulky pallet. What would you use instead, and does the reconciliation check still hold?
The carrier invoice arrives three weeks later and differs from the rated cost. Where does the correction land, and does it re-run this allocation?

Stop two allocators promising the same unit of stock

Medium · isolation levels · lock contention · compare-and-set

inventory_position is keyed (item_id, node_id, lot_id, state) and holds qty BIGINT, which is on-hand stock and is moved only by the movement ledger; available_qty BIGINT, maintained on the same row; and version. The declared invariant is available_qty = qty - SUM(allocation.qty) over that key in status held or committed, enforced by CHECK (available_qty >= 0) and CHECK (available_qty <= qty). allocation holds allocation_id, order_line_id, item_id, node_id, lot_id, qty, tier, status (held, committed, consumed, released, expired) and expires_at. Twelve allocator threads try to claim the last eight units of one item at one node inside the same 50 ms. Give the exact statements that let at most eight units be claimed, then say what changes if identical code runs under PostgreSQL REPEATABLE READ versus InnoDB REPEATABLE READ, and what a retried allocate for the same order line must do.

Approach

Collapse the check and the write into one statement whose WHERE clause is the business rule: UPDATE inventory_position SET available_qty = available_qty - :q, version = version + 1 WHERE item_id = :i AND node_id = :n AND lot_id = :l AND state = 'on_hand' AND available_qty >= :q. qty is deliberately untouched: a hold is a promise, not a physical move, so only a movement may change on-hand and the ledger-equals-position invariant survives. Rowcount zero is a refusal, not an error, and the allocation row is inserted in the same transaction only when the rowcount is one.
If the deployment carries no available_qty column and availability must be derived, the alternative is SELECT ... FOR UPDATE on the position row followed by the SUM over held and committed allocations inside the same transaction. That is correct but holds the row lock for two statements instead of one, which matters on a key taking hundreds of writes a second.
Isolation is not a substitute for the atomic step, and it changes the error contract. Under PostgreSQL READ COMMITTED the losing UPDATE blocks, then re-evaluates its predicate against the newly committed row version and returns rowcount 0, which the handler already handles. Under PostgreSQL REPEATABLE READ the same statement instead raises SQLSTATE 40001 and the whole transaction must be retried, so a handler that only tests rowcount fails closed on an exception it never expected. InnoDB REPEATABLE READ does neither: a locking read or UPDATE performs a current read of the latest committed row, so the loser blocks and then sees rowcount 0, with a deadlock rather than a serialization failure as its exceptional case.
Make the retry a no-op with a partial unique index: UNIQUE (order_line_id, node_id, lot_id) WHERE status IN ('held','committed'). Then a retried allocate conflicts instead of double-claiming, and the handler reads the existing allocation and returns it.
Never source the availability number from a replica or a cache for this path. Replication lag is largest during the write burst that made the key hot, so the guarantee fails precisely when it is load-bearing.
For the hottest item-node pairs, shed load without overselling: route allocations for that key through a single writer with a bounded queue and refuse with a conservative not-available when the queue is full. Splitting the position into N sub-rows cuts contention N-fold but strands the remainder across sub-rows and produces false out-of-stock when the last units are spread thin.

Worked solution (30 min)

Write the conditional UPDATE, the rowcount branch and the allocation INSERT, plus the partial unique index DDL and the two CHECK constraints the invariant needs.
Set a position to qty = 8 and available_qty = 8, launch 12 threads each claiming 1, and assert exactly 8 succeed, available_qty lands at 0 and qty is still 8.
Re-run the same race under PostgreSQL REPEATABLE READ and record how many transactions fail with 40001 rather than returning rowcount 0.
Send one allocate twice for the same order line and node, and assert one allocation row and the same allocation_id returned both times.
Remove the available_qty >= :q predicate and confirm CHECK (available_qty >= 0) aborts the losing transactions, proving the test exercises the race rather than serialising by luck.

Expected result: Exactly 8 allocation rows in status held, available_qty = 0 with qty unchanged at 8, 4 refusals returned as a clean not-available response, and a duplicate allocate that returns the existing allocation without creating a second row.

Verify it yourself

Repeat the 12-way race 50 times and require 8 successes on every run, not on average.
Sample mid-run and require available_qty = qty - SUM(allocation.qty) over held and committed at every sample, with that sum never exceeding 8.
Confirm the REPEATABLE READ run produces 40001 errors and that the handler retries rather than surfacing them.
Drop the partial unique index and confirm the duplicate allocate now creates two rows.

Follow-up

One order line needs two units from two different nodes, atomically. What do you do when those positions live in different shards and no distributed transaction is available?
Soft-tier allocations expire. Show the sweeper statement, and say what stops it from releasing an allocation that is being committed at that instant.
Under sustained contention your CAS retries. What is the retry budget, and what does the caller see when it is exhausted?

Allocate through a regional failover with unknown replication state

Hard · failover · replication lag · consistency · durability

The inventory primary sits in one region with an asynchronous standby in another, typically 2 seconds behind. The region becomes unreachable at peak, with 3,000 allocations and 3,000 ATP reads per second in flight. Decide whether to promote, what happens to allocations that may have committed without replicating, and what ATP answers meanwhile. Then quantify the cost of making the allocation write synchronous instead, given today's 4 ms transaction and about 42 allocations per second on the hottest item-node pair.

Approach

Split the decision by operation class before touching the failover mechanics. ATP reads can move to the standby at once with a stated staleness, because a 2-second-old read only mis-advertises and the allocation write re-checks anyway. Allocations cannot: promoting an asynchronous standby discards up to 2 seconds of committed allocations, several thousand at this rate, and each one is a promise already made to a customer.
Recover those from the requester's durable intent, not from the database that just vanished. If the orchestrator wrote its saga step and an outbox row in one local transaction before calling, every lost allocation can be re-driven after promotion under its original idempotency key, and allocation's partial unique index makes a re-drive that did replicate fail closed instead of double-claiming. Without that intent record the loss is simply unrecoverable, and the honest design refuses allocations until the primary returns rather than pretending otherwise.
Price synchronous replication in the currency that actually binds, which is hot-row throughput rather than average latency. A cross-region synchronous commit adds tens of milliseconds inside the transaction, turning a 4 ms hold into roughly 65 ms and dropping one row from about 250 to about 15 allocations per second, below the 42 demanded - so the promotion tail becomes an outage every promotion. A synchronous standby in a second zone of the same region costs 1-2 ms, leaves the hold near 6 ms and the ceiling near 170 per second, still four times demand.
So the configuration is: synchronous in-region for zero data loss on a node or zone failure, asynchronous cross-region for the regional case, and promotion of the cross-region standby as a deliberate human decision with a written procedure. Automatic promotion of an asynchronous standby is how a regional blip becomes two primaries and a permanently split ledger that reconciliation cannot repair, because both halves are internally consistent.
Define the degraded contracts explicitly for the gap. ATP serves the conservative bucketed answer from the standby with staleness stated; allocate returns a typed unavailability so the orchestrator holds lines in sourcing rather than cancelling or retrying them into a storm. After promotion, replay the outbox, then run the reconciliation re-sum before any node's availability is trusted again, because the re-summed positions are the only evidence of what actually survived.

Follow-up

The old primary returns holding writes the standby never saw. What do you do with them, and what does the reconciliation job report about them?
How do you rehearse this? Describe a game day that exercises the promotion path without risking an oversell in production.
Which of the two regions' clocks do you trust for occurred_at ordering during the switch, and does it matter here?

Redesign a tender endpoint whose retry books two trucks

Medium · idempotency · irreversible effects · retries

POST /shipments/{shipment_id}/tender asks the Carrier Integration Gateway to tender a load and returns the partner booking reference. The HTTP client has a 10 s timeout and a three-attempt retry decorator; partner p99 is 4 s with occasional 40 s stalls. shipment_leg already carries booking_idempotency_key TEXT NOT NULL UNIQUE. Last week two trailers arrived at one dock for one load. Redesign the contract so the physical effect happens at most once: give the resource shape, the status model including the ambiguous outcome, who performs retries, and what the caller does on a timeout.

Approach

Diagnose the shape, not the bug. A POST to an action with server-minted identity gives the caller no way to name the effect it wants, so every retry is a new request by construction. Replace it with a caller-named sub-resource - PUT /legs/{leg_id}/booking/{booking_idempotency_key} - where the key is in the URI and a second identical PUT converges on the same resource instead of creating another.
Persist before you call. Write the booking row with status requested and the key committed locally before the gateway touches the partner. A crash between that commit and the call is recoverable by a sweeper; a call with no local record is not recoverable at all, because you cannot ask about a key you never stored.
Make unknown a first-class status: requested, unknown, confirmed, rejected. A timeout or a connection reset moves the row to unknown and the API answers 202 with the resource and a Location, never a 5xx - a 5xx is an invitation to the decorator to re-issue an irreversible act.
Move retries off the request path entirely. Delete the decorator for this call and let a sweeper read rows in requested or unknown and resolve each by querying the partner for that key, or by their list-by-reference endpoint when they have no key concept. The sweeper is now the component that must be correct, and it is testable in a way an in-process retry that dies with the pod is not.
Keep the database transaction off the network call. Commit the state row together with an outbox entry and let a relay make the partner call; a transaction held open across a 40 s stall parks a pooled connection, and at tender peak the pool is what fails first, taking unrelated traffic with it.
State the caller's rule in one line: a timeout or 5xx from this endpoint means unknown, never not booked, and the only permitted follow-up is to read the booking resource.

Worked solution (35 min)

Draw the current timeline that produces two trailers: request sent, partner commits, response lost at 10 s, decorator re-issues, partner commits again.
Rewrite the endpoint as a key-in-URI PUT and write the request and response bodies for first call, duplicate call and ambiguous outcome.
Write the status table with allowed transitions and mark every transition that touches the partner.
Write the sweeper's query and its resolution rule for a row stuck in unknown, including the decision it makes when the partner reports nothing for that key.
Re-examine the connection lifecycle and confirm no transaction is open while the partner call is in flight.

Expected result: A PUT whose URI carries the caller-owned key, a persisted requested state written before any partner contact, an explicit unknown state returned as 202, retries owned by a sweeper reading persisted rows, and no database transaction spanning the external call.

Verify it yourself

Replaying the same PUT ten times yields one booking reference and one partner-side booking.
Killing the process immediately after the partner call leaves a row a sweeper can resolve without guessing.
No code path both holds a row lock and awaits the gateway.

Follow-up

The partner supports no idempotency key and no query-by-reference. What do you do, and what residual risk do you accept?
Two sweeper instances pick up the same unknown row at once. What stops the resolution itself from producing a second booking?
How do you cancel a booking whose outcome is still unknown, and what does the leg's status machine look like during that?

Availability p99 spikes in a sawtooth locked to the cache TTL

Medium · cache stampede · single-flight · load shedding

Available-to-promise reads sit inside checkout with a 12 ms p99 behind a 60-second per-key cache. During a promotion, p99 spikes to 900 ms in a sawtooth exactly on 60-second boundaries and is normal between spikes. The inventory database shows bursts of thousands of identical single-row position reads inside 50 ms across about 40 item-node keys, and the connection pool saturates during each burst. Request throughput is unchanged. Diagnose, fix, and state which callers may read a cached availability and which may not.

Approach

Separate a stampede from saturation by the shape. Latency that rises smoothly with request rate is capacity; a sawtooth whose period equals the TTL, with normal service between peaks, is expiry-driven. Confirm by overlaying spike timestamps on key expiry times and watching the hit ratio fall to near zero for a few milliseconds each cycle.
Size the herd with Little's law rather than adjectives. Concurrency equals arrival rate times service time, so a key taking 1,200 requests per second with a 30 ms origin read admits about 36 simultaneous origin reads at each expiry, and 40 keys that expire together produce roughly 1,440 against a 100-connection pool. That fourteen-fold oversubscription is the 900 ms, and the arithmetic matching is what confirms the diagnosis.
Explain why the 40 keys expire in lockstep: they were all populated in the same second when promotion traffic arrived, so their TTLs are synchronised and stay synchronised. Jitter of plus or minus twenty percent on the TTL desynchronises them; it does not reduce the per-key herd.
Bound the per-key herd with single-flight. One in-flight origin read per cache key, with concurrent callers awaiting the same result, takes origin load to one read per key per TTL regardless of request rate. Add serve-stale-while-revalidate so the waiters get the previous value immediately and staleness is bounded and measured rather than incidental.
Draw the correctness boundary explicitly, because it is not a performance question. Browse and product pages may read a cached or stale availability. The allocation path may not: the check and the write must be one atomic step against the primary, so a cached number can inform a display but must never authorise a reservation. Checkout sits in between and should either read through with a short TTL or accept that its number is advisory and can fail at allocation.
Decide the degradation direction in advance. If the origin is saturated, the read path should shed to a conservative answer, showing limited availability rather than a precise count and never rounding upward, because an optimistic stale read converts a latency incident into oversells and short-ships.

Follow-up

What does your single-flight do when the origin read times out: fail the waiters, or serve them the stale value, and what bounds the staleness?
One key gets hot enough that even one origin read per TTL per replica is too much across 200 application instances. What changes?
How would you invalidate a key on a movement write without opening a second stampede on the invalidation itself?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

Know where your numbers came from

Bring the two or three numbers the story rests on and know how they were collected. A p99 whose timer starts inside your handler excludes the time a request spent queued, so it can sit flat while users wait longer. Give the window, the percentile and what the measurement left out, or drop the number.

Describe a time when you had to prioritize speed without compromising the integrity of your code.

Medium · behavioural and engineering judgement

Approach

Pick a story where you made the decision, not one where you watched it.
Give the blast radius: what could have broken, and what you measured.
Close with what you would do differently, concretely.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?

What is your approach when a system failure occurs during active manufacturing?

Medium · behavioural and engineering judgement

Approach

State the situation in two sentences and spend the rest on the reasoning.
Name the disagreement and how you resolved it with evidence.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?

How do you communicate technical constraints to non-technical stakeholders on the plant floor?

Medium · behavioural and engineering judgement

Approach

Name the disagreement and how you resolved it with evidence.
Close with what you would do differently, concretely.
State the situation in two sentences and spend the rest on the reasoning.

Follow-up

What did you decide not to do, and why?
How did you know your change caused the improvement?

How do you work under pressure?

Medium · behavioural and engineering judgement

Approach

Name the disagreement and how you resolved it with evidence.
State the situation in two sentences and spend the rest on the reasoning.
Close with what you would do differently, concretely.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?