cswork / libraryOpen practice workspace

Interview guides · ALUMA-FORM

ALUMA-FORM Software Engineer Interview Questions & Guide 2026

Concurrency drills for ALUMA-FORM software engineer prep: races, lock ordering, and what a check-then-act sequence assumes.

The 60-second brief

At ALUMA-FORM, the Software Engineer role serves as a critical bridge between advanced manufacturing processes and digital infrastructure. You are not just writing code; you are building the systems that ensure precision, efficiency, and safety in ALUMA-FORM's production environments. Your work directly affects how the company designs, manufactures, and distributes components that power critical electrical utility infrastructure.

Ask how many rounds there are and what each one is before you plan your weeks, because the composition is what your hours should follow. A loop with three coding rounds and one short design conversation deserves a different split from the reverse, and whoever schedules it will usually say plainly which it is.

Focus areas

  • Reconcile a movement ledger against derived positions
  • Keep a site operating during a network partition
  • Make every external side effect safe to retry

Overview

This position is inherently multidisciplinary. You will collaborate with teams in Plastics Processing, Manufacturing Engineering, and Product Design to translate real-world engineering challenges into scalable software solutions. Whether you are optimizing data flows for shop-floor machinery or developing internal tools that streamline design cycles, your contributions are essential to maintaining the high quality and operational excellence that define ALUMA-FORM.

The ALUMA-FORM interview process

ALUMA-FORM candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Half of this call is the part candidates treat as small talk: start date, notice period, work authorisation and its timing, location and time zone, on-call, and the number. Those are what kill offers late, after several engineers have each spent a day. Surfacing a hard constraint now costs you nothing and occasionally buys you something, since a loop compressed to fit a competing deadline can usually only be arranged if it is asked for early. The common failure is deflecting the compensation question twice, then discovering at offer stage that the band never reached your number.

What to demonstrate

  • Whether your hard constraints are compatible with the role before a loop gets booked: earliest start, notice period, what authorisation you hold and when it needs action, days on site, willingness to carry a pager
  • Whether you give a compensation range with something behind it, such as current total compensation or a competing timeline, rather than leaving the band untested
  • Whether your stated timeline is real, since a competing deadline raised now is something scheduling can sometimes work around and the same deadline raised at offer stage usually is not

How to prepare

  • Write each constraint down in one line before the call and state them as facts rather than negotiating them live under a question you were not expecting
  • Set your range from two or three current data points for that level and location, and name the structure you are quoting in, so the number is comparable to the one they are holding
  • If another process is running, say where it stands and by when, and ask directly whether this loop can be scheduled inside that window

Technical Assessments

Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.

What to demonstrate

  • Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
  • Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
  • Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
  • Whether a failing case is isolated and explained before any edit is made to the code

How to prepare

  • From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
  • Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
  • Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.

Behavioral Stories

What you say here is written down by each interviewer and compared afterwards, so the unit of evaluation is a claim someone else could check, not a well-told narrative. Two things make a story checkable: detail only a participant would hold, and a clean line around which part was yours. Vague ownership is the usual failure and it is usually accidental, because engineers say we about the team's work and we about their own, so the thing they personally built disappears into the plural. Name the part you wrote, and name who did the rest.

What to demonstrate

  • Whether your details are ones a participant would hold and an observer would not: the constraint that ruled out the obvious approach, the first attempt that failed, the person who objected and on what grounds
  • Whether ownership survives a direct question, since a follow-up to we decided is routinely who decided, and an answer that stays plural at that point is read as the work belonging to someone else
  • Whether the numbers you quote are ones you would say identically to a former colleague with the dashboard open

How to prepare

  • Go through each story replacing every we with either I or a named role (the on-call engineer, the reviewer, the other team) and check the story still holds together. Wherever it stops making sense you have found a part you cannot actually speak to
  • Open the artefacts for two of your stories, the pull request, the design doc, the incident notes, and read them for dates and figures you have been rounding in the retelling. Correct your version to match
  • For each story write the single sentence you would least want repeated to a former teammate, then either make it accurate or take it out

Final Decision

Where the day includes a partner from product, design or data, that conversation is weighted like the technical ones and prepared for least. They are deciding one thing: whether having you in the room makes their decisions cheaper. That means options with costs attached, not implementation detail and not "it depends". An estimate someone can plan against — a range, the assumption that would push it to the high end, and what you would drop to hit the low one — is worth more than a confident single number, which everyone present already knows is wrong.

What to demonstrate

  • Whether an estimate comes as a range with the assumption most likely to break it, and states what a specific scope cut would actually buy
  • Whether a technical constraint is handed over as a choice with consequences on their side, rather than as a verdict they have no standing to argue with
  • Whether you establish what decision is on the table before proposing anything
  • Whether risk is raised while it can still change the plan, with the trigger that would confirm it, instead of reported afterwards as a slip

How to prepare

  • Take a project that shipped late and write the two-sentence warning you could have given three weeks earlier, naming what you would have needed decided at that point
  • Rehearse one estimate out loud until it arrives in three parts: the range, the single assumption that would blow it, and the smallest thing you would cut to protect the date
  • Rewrite an objection you have actually made — the "we can't do that" version — as two options with their costs, so the choice ends up with the person who owns it

Where candidates lose points

Treating the database as the truth about physical stock

Every system in this domain drifts from reality, through miscounts, unrecorded damage, mis-scans and theft, so a design with no adjustment path forces the drift somewhere worse. The usual shape is a CHECK constraint or application guard that clamps a position at zero: the negative that would have exposed a missing receipt is silently swallowed, and the first visible symptom is a picker standing at an empty location for an order the system promised. The correct posture is to let the invariant fail loudly - record the cycle-count adjustment as a first-class movement with its own reason code, surface the variance, and let planning see that this location is unreliable - because an adjustment that is indistinguishable from a normal pick destroys the only signal that would have found the root cause. A related version of the same mistake is deleting or updating a movement row to fix a mistake, which leaves the position correct and the history a lie.

Retrying an irreversible external call without an owned idempotency key

Timeouts and connection resets are ambiguous by construction - the partner may have committed before the response was lost - so a generic retry policy around an HTTP client is, in this domain, a machine for buying two labels and dispatching two trucks. Relying on the partner's deduplication is not a substitute, because their window is usually short, their key is often derived from fields you may legitimately change on retry, and many older integrations have no such concept at all. The workable pattern is to generate the key yourself, persist it with an explicit unknown status before the call, and resolve ambiguity by querying the partner for that key rather than re-issuing; the sweeper that does this is the component that has to be correct, not the call site. Teams get this right for payments and then forget that a carrier tender, a warehouse work release and an EDI shipping notice have the same shape.

Naming no test cases at all

State what you would test before being asked: empty input, a single element, all elements equal, the maximum permitted size, and the input that exercises the branch you just wrote. It costs thirty seconds and is much of what separates someone who has shipped code from someone who has only solved puzzles.

Assuming the bug is in the framework

Suspect your own code first: read the stack trace top to bottom, check which versions are actually installed rather than which ones you believe are, and reproduce in isolation before blaming a library that thousands of people run daily. When the fault really is upstream, you need that minimal reproduction to say so credibly anyway.

Questions & practice

Split an order across the fewest nodes that can fill it

Hard · set cover · bitmask dp · approximation

A fulfilment order has L lines, L up to 25. After filtering by capability, service time and stock, N candidate nodes remain; N is usually under 14 but can reach 200. Each node covers a known subset of the lines in full. Return a minimum-size set of nodes covering every line, tie-broken by total distance. Give an exact algorithm for the small case with its exact cost in operations and bytes, say precisely why enumerating node subsets stops working, and give what you ship when N is 200.

Approach

  1. Name it: this is minimum-cardinality set cover, which is NP-hard, so the useful move is choosing which parameter you are exponential in rather than hunting for a polynomial exact algorithm. Enumerating subsets of nodes is 2^N and correct — at N = 14 that is 16,384 subsets and finishes instantly, which is exactly why it survives review and then fails in production at N = 200, where 2^200 is about 1.6e60.
  2. Be exponential in L instead, because L is the parameter you bound. Index the DP by the set of lines still uncovered: dp[rem] is the cheapest way to cover the lines in rem, with dp[0] = (0, 0) and the value a (node_count, total_distance) pair compared lexicographically. Let b be the lowest set bit of rem; any cover must contain some node that stocks line b, so the transition ranges only over those nodes: dp[rem] = min over v with b in cover(v) of dp[rem & ~cover(v)] + (1, dist(v)). Worst case O(2^L * N) — at L = 25 and N = 14 that is 33,554,432 states times 14, about 4.7e8 transitions — and the lowest-bit restriction cuts the real branching factor to the nodes stocking one specific line.
  3. Quote the memory for what you actually store, not for a count array alone. Over 2^25 masks you need three dense arrays: uint8 node count (33.5 MB), uint32 total distance (134.2 MB, integer metres so the tie-break compares exactly and the array stays fixed-width), and uint8 chosen node (33.5 MB, since N <= 200 fits a byte) — about 201 MB, roughly six times the 33.5 MB that a bare count array would suggest. The same layout at L = 20 is 6.3 MB, which is why this is comfortable up to about 20 lines and needs a deliberate decision at 25.
  4. Reconstruction is free in this formulation and needs no parent-mask array: because every transition eliminates the lowest uncovered line, the predecessor of rem under the stored choice v is exactly rem & ~cover(v), so walk rem from the full mask down to 0 emitting choice[rem]. If the reachable state count is far below 2^L — coverage sets are usually highly correlated — a top-down memoized recursion over a hash map trades the 201 MB for roughly 40-60 bytes per visited state; measure the visited count first, because a hash map that ends up touching most masks is strictly worse than the dense arrays. Add a branch-and-bound cut of ceil(popcount(rem) / max_v popcount(cover(v) & rem)), which prunes long before the theoretical bound bites.
  5. When exact is out of reach, ship greedy with a stated guarantee: repeatedly take the node covering the most currently-uncovered lines, using bitset popcounts, O(N * ceil(L/64)) word operations per pick and at most min(N, L) picks. Greedy is within H(L) <= ln L + 1 of optimal (Chvatal), and no polynomial algorithm achieves (1 - o(1)) ln L unless P = NP (Dinur and Steurer, 2014) — so effort belongs in the pre-filter and the tie-break, not in a cleverer heuristic. At L = 25 the bound is H(25), about 3.82x worst case, while the realistic gap on correlated coverage sets is zero or one node.
  6. Close on the objective, because fewest nodes is not cheapest. Two shipments from distant nodes routinely cost more than three from near ones, and a split also costs a second box, a second carrier pickup and a worse customer experience. If cost is the real objective it is weighted set cover — greedy by cost divided by newly-covered lines, same logarithmic bound — and if a node can cover a line only partially, the line mask is no longer sufficient state, since you must track remaining quantity per line; that is the point at which you stop hand-rolling and either restrict splits or hand an MILP to a solver with a time budget.

Worked solution (40 min)

  1. Write cover(v) as an L-bit mask per node and confirm that the union of all masks equals the full mask, otherwise the order is unfillable and the answer is a backorder, not a cover.
  2. Implement the DP over uncovered masks with the lowest-uncovered-line transition, the (node_count, total_distance) value and the choice array, then measure resident memory and wall time at L = 20 and L = 25 against the quoted 6.3 MB and 201 MB.
  3. Implement greedy separately and run both on 10,000 random instances with L = 12, N = 10, comparing sizes.
  4. Construct the textbook instance where greedy is strictly worse than optimal — nested sets of geometrically decreasing size plus two disjoint halves — and confirm your greedy reproduces the gap.
  5. Add the distance tie-break to both and confirm the exact path returns the cheapest minimum-size cover, not merely a minimum-size one.

Expected result: The DP and greedy agree on the large majority of random instances; on the adversarial instance greedy returns more nodes than the DP, by a margin consistent with the H(L) bound; the DP at L = 25 holds about 201 MB across its three arrays and runs in seconds rather than milliseconds, with cache misses over that working set dominating the 4.7e8 transitions.

Verify it yourself

  • Measured resident memory at L = 25 is near the quoted 201 MB — if it comes out near 33 MB you have dropped either the distance component or the reconstruction array, and the reported figure is describing a program you did not write.
  • Every returned set actually covers the full mask — assert it on the result, not just on the transitions.
  • Reconstruction from choice alone reproduces a set whose node count and total distance equal dp[full], which is what proves the predecessor identity rem & ~cover(v) holds.
  • The DP's answer never exceeds greedy's on any instance.
  • Doubling N multiplies DP runtime by about two, while subset enumeration would square its work — measure it rather than asserting it.

Follow-up

  • The pre-filter is what keeps N small. What is in it, and what happens to your exact path the day someone loosens it?
  • One node can cover a line only partially. Show where the bitmask formulation breaks and what state replaces it.
  • You have 40 milliseconds inside a checkout call. Which of these runs there, and what runs asynchronously afterwards?

Trace a suspect lot forward to every shipped order line

Medium · graph traversal · genealogy · indexing

lot_edge(parent_lot_id, child_lot_id) has 400 million rows with btree indexes on both (parent_lot_id, child_lot_id) and (child_lot_id, parent_lot_id). lot_shipment(lot_id, order_line_id) maps lots to shipped lines. Given one suspect received lot, return every downstream order_line_id. The reachable subgraph can reach 2 million lots and 12 levels; the full edge table does not fit in memory. Give the traversal, its complexity in terms of the reachable subgraph, and what you do if the data contains a cycle.

Approach

  1. Genealogy is a directed acyclic graph, not a tree: a production run consumes several lots and one lot splits across many shipments, so a node is reached by several paths and a tree traversal revisits it combinatorially. Breadth-first from the suspect lot over the (parent_lot_id, ...) index, with a hash set of visited lot ids, is O(V' + E') where V' and E' count only the reachable subgraph — 2 million 8-byte ids is 16 MB of payload and a few tens of MB with open-addressing overhead, which is why the visited set is affordable and the edge table's 400 million rows never enter memory.
  2. Batch each level into one indexed read. Issue WHERE parent_lot_id = ANY($1) with a few thousand parents per call instead of a query per node: at 12 levels and 2 million nodes, per-node round trips dominate the wall clock entirely, and the recall's deadline is a real-world response time, not a query-plan aesthetic. The batch size is the tuning knob between round-trip count and per-statement planning cost.
  3. Keep the visited set even though the schema says DAG. Rework and repack loops — material returned into an earlier lot — do occur, and an unguarded traversal on a cyclic edge set does not terminate; a set membership test is O(1) and turns a hang into a correct answer plus a data-quality alert. Also cap the traversal by level and by node count, and fail loudly on the cap rather than returning a partial set silently.
  4. Join to lot_shipment once at the end over every visited lot id, not over the leaves and not per node. A lot is routinely both shipped and consumed: half a lot goes out on an order line while the other half is repacked into a child lot, so it has a lot_shipment row and outgoing edges at the same time and is not a leaf. Restricting the join to nodes with no outgoing edges drops exactly those order lines, which is under-recall — the one failure mode this trace is not allowed to have. One bulk join against an index on lot_id over the whole visited set is a single pass; per-node lookups multiply the round-trip problem by the number of visited lots.
  5. State the semantics honestly: reachability answers which shipped lines could contain material from the suspect lot, and without propagating edge quantities it is a superset of physical containment. For a recall that is the correct direction of error — over-recall, never under-recall — and saying so is the point, because a trace that returns quickly and incompletely will be believed. The upstream question, which received lots are in a suspect shipped unit, is the same traversal over the reverse index, which is why both indexes exist.

Follow-up

  • A recursive CTE would express this in ten lines. What is its cost here if the index on the traversal column is missing, and how would you notice?
  • You need the quantity of suspect material in each downstream line, not just reachability. What has to be on the edge, and what does that do to the traversal?
  • The traversal must run while a production run is writing new edges. What does a consistent answer mean here, and what do you cut it at?

Re-sum a movement ledger and report positions that disagree

Easy · aggregation · reconciliation · append-only ledger

inventory_movement holds 3 billion immutable rows: item_id, node_id, lot_id (NULL when the item is lot-untracked), state, delta_qty (signed int64 in the item's smallest transacting unit), uom_code, and reversed_by. inventory_position holds one row per (item_id, node_id, lot_id, state) with qty, version and last_movement_id, using lot_id = 0 as the sentinel for untracked. Up to 80 million distinct keys. Report every key whose summed movements disagree with the stored qty, while writes continue. State your time and space bounds and how you bound the comparison.

Approach

  1. Fix the key first. inventory_movement.lot_id is NULL for untracked items and inventory_position.lot_id is 0, so the group key is (item_id, node_id, COALESCE(lot_id, 0), state). Getting this wrong does not error — it silently produces two groups that each look like a variance, and the report becomes noise nobody reads.
  2. Cut the ledger at a watermark, but do not take W = max(movement_id) at the start of the scan. A sequence hands out an id before the inserting transaction commits, so at the instant you read that maximum there are ids below it still in flight and invisible to your snapshot. Summing movement_id <= W misses them on this run, and advancing reconciled_through_movement_id to W makes the miss permanent: every later incremental run starts above those ids and they are never summed again. That is a hole in the ledger's own re-derivation, not a transient skew.
  3. Take a watermark that is provably settled instead. Bound write transactions with a statement or transaction timeout so 'the longest write' is a number T, record (observed_at, max_movement_id) samples periodically, and use as W_safe the largest sampled id whose observed_at is older than T — every id at or below it has committed or rolled back. Sum rows with movement_id <= W_safe, compare only against position rows with last_movement_id <= W_safe, treat anything newer as a write that raced you rather than a variance, and advance reconciled_through_movement_id only to W_safe so the next run starts there instead of re-reading 3 billion rows. If the ledger carries a commit timestamp, cutting on that is the same guarantee without the sampling table.
  4. Sum delta_qty as int64, and include reversal rows. reversed_by is a back-pointer for audit, not an exclusion filter: an original of +10 and its reversal of -10 must both be summed to reach 0. Excluding the original while keeping the reversal produces -10 and a false variance on every corrected key.
  5. Guard the denomination rather than trusting it. uom_code is stored per row because pack factors change over time, so reject — do not sum — any row whose uom_code is not the item's smallest transacting unit, and report those keys separately. A mixed-denomination sum is arithmetically meaningless and looks exactly like a real variance.
  6. Hash aggregation is O(n) time and O(distinct keys) space: 80 million entries at roughly 40-56 bytes each in a typical runtime is 3-5 GB, so quote the number. The bounded-memory alternative is an external sort-merge on the group key — O(n log n) comparisons, resident memory bounded by the merge fan-in rather than by key count, and it streams — or hash-partition by hash(item_id) % P and run P independent passes for 1/P of the peak.

Follow-up

  • A key shows a variance of exactly one pick, repeatedly, at one node. What do you look at first, and what would distinguish a duplicate movement from a missed one?
  • The job takes six hours and the variance report is stale by the time anyone reads it. How would you make it incremental without losing the guarantee that it re-derives from the ledger?
  • Who writes the correcting movement, and what reason code does it carry?

Stop two allocators promising the same unit of stock

Medium · isolation levels · lock contention · compare-and-set

inventory_position is keyed (item_id, node_id, lot_id, state) and holds qty BIGINT, which is on-hand stock and is moved only by the movement ledger; available_qty BIGINT, maintained on the same row; and version. The declared invariant is available_qty = qty - SUM(allocation.qty) over that key in status held or committed, enforced by CHECK (available_qty >= 0) and CHECK (available_qty <= qty). allocation holds allocation_id, order_line_id, item_id, node_id, lot_id, qty, tier, status (held, committed, consumed, released, expired) and expires_at. Twelve allocator threads try to claim the last eight units of one item at one node inside the same 50 ms. Give the exact statements that let at most eight units be claimed, then say what changes if identical code runs under PostgreSQL REPEATABLE READ versus InnoDB REPEATABLE READ, and what a retried allocate for the same order line must do.

Approach

  1. Collapse the check and the write into one statement whose WHERE clause is the business rule: UPDATE inventory_position SET available_qty = available_qty - :q, version = version + 1 WHERE item_id = :i AND node_id = :n AND lot_id = :l AND state = 'on_hand' AND available_qty >= :q. qty is deliberately untouched: a hold is a promise, not a physical move, so only a movement may change on-hand and the ledger-equals-position invariant survives. Rowcount zero is a refusal, not an error, and the allocation row is inserted in the same transaction only when the rowcount is one.
  2. If the deployment carries no available_qty column and availability must be derived, the alternative is SELECT ... FOR UPDATE on the position row followed by the SUM over held and committed allocations inside the same transaction. That is correct but holds the row lock for two statements instead of one, which matters on a key taking hundreds of writes a second.
  3. Isolation is not a substitute for the atomic step, and it changes the error contract. Under PostgreSQL READ COMMITTED the losing UPDATE blocks, then re-evaluates its predicate against the newly committed row version and returns rowcount 0, which the handler already handles. Under PostgreSQL REPEATABLE READ the same statement instead raises SQLSTATE 40001 and the whole transaction must be retried, so a handler that only tests rowcount fails closed on an exception it never expected. InnoDB REPEATABLE READ does neither: a locking read or UPDATE performs a current read of the latest committed row, so the loser blocks and then sees rowcount 0, with a deadlock rather than a serialization failure as its exceptional case.
  4. Make the retry a no-op with a partial unique index: UNIQUE (order_line_id, node_id, lot_id) WHERE status IN ('held','committed'). Then a retried allocate conflicts instead of double-claiming, and the handler reads the existing allocation and returns it.
  5. Never source the availability number from a replica or a cache for this path. Replication lag is largest during the write burst that made the key hot, so the guarantee fails precisely when it is load-bearing.
  6. For the hottest item-node pairs, shed load without overselling: route allocations for that key through a single writer with a bounded queue and refuse with a conservative not-available when the queue is full. Splitting the position into N sub-rows cuts contention N-fold but strands the remainder across sub-rows and produces false out-of-stock when the last units are spread thin.

Worked solution (30 min)

  1. Write the conditional UPDATE, the rowcount branch and the allocation INSERT, plus the partial unique index DDL and the two CHECK constraints the invariant needs.
  2. Set a position to qty = 8 and available_qty = 8, launch 12 threads each claiming 1, and assert exactly 8 succeed, available_qty lands at 0 and qty is still 8.
  3. Re-run the same race under PostgreSQL REPEATABLE READ and record how many transactions fail with 40001 rather than returning rowcount 0.
  4. Send one allocate twice for the same order line and node, and assert one allocation row and the same allocation_id returned both times.
  5. Remove the available_qty >= :q predicate and confirm CHECK (available_qty >= 0) aborts the losing transactions, proving the test exercises the race rather than serialising by luck.

Expected result: Exactly 8 allocation rows in status held, available_qty = 0 with qty unchanged at 8, 4 refusals returned as a clean not-available response, and a duplicate allocate that returns the existing allocation without creating a second row.

Verify it yourself

  • Repeat the 12-way race 50 times and require 8 successes on every run, not on average.
  • Sample mid-run and require available_qty = qty - SUM(allocation.qty) over held and committed at every sample, with that sum never exceeding 8.
  • Confirm the REPEATABLE READ run produces 40001 errors and that the handler retries rather than surfacing them.
  • Drop the partial unique index and confirm the duplicate allocate now creates two rows.

Follow-up

  • One order line needs two units from two different nodes, atomically. What do you do when those positions live in different shards and no distributed transaction is available?
  • Soft-tier allocations expire. Show the sweeper statement, and say what stops it from releasing an allocation that is being committed at that instant.
  • Under sustained contention your CAS retries. What is the retry budget, and what does the caller see when it is exhausted?

Two scan-event queries that stopped using their indexes

Medium · indexing · partition pruning · brin correlation

observation_event is range-partitioned monthly on occurred_at, with a btree on (subject_type, subject_id, occurred_at), a BRIN on occurred_at and a GiST on geo. Rows are inserted in received_at order and events are routinely hours late. Two queries regressed: (A) WHERE occurred_at::date = current_date AND observed_status = 'delivered'; (B) a backfill scanning WHERE received_at >= now() - interval '1 day'. For each, say why no index is used and no partition is pruned, and give the fix.

Approach

  1. Query A wraps the column in a cast, so the predicate is over occurred_at::date and not over occurred_at. A btree or BRIN on the bare column cannot be probed, and the planner has no range to prune partitions with, so it plans every monthly partition. Fix it by rewriting to a half-open range: occurred_at >= :start AND occurred_at < :start + interval '1 day', with :start computed by the caller from the relevant IANA zone.
  2. Do not reach for an expression index as the first fix here. The timestamptz-to-date cast is STABLE rather than IMMUTABLE because it depends on the session TimeZone, so PostgreSQL refuses to index it; indexing ((occurred_at AT TIME ZONE 'UTC')::date) is accepted but silently pins the day boundary to UTC, which is the wrong day for any stop outside it.
  3. Query B filters a column that is neither indexed nor the partition key, so it is a full scan of every partition by construction. The subtler half is why the BRIN cannot rescue it: BRIN stores a min and max per block range and only pays when physical order correlates with the column. Rows arrive in received_at order while the indexed column is occurred_at, and late events interleave hours-old timestamps with current ones, so each range's min and max span days and almost every range qualifies.
  4. Measure the correlation rather than assuming it: SELECT attname, correlation FROM pg_stats WHERE tablename = 'observation_event'. Near 1 means BRIN earns its keep, near 0 means it is a full scan with extra steps. Expect received_at near 1 and occurred_at well below it on this insert pattern.
  5. Fix B with a btree on received_at, which stays compact because it is correlated with insert order, or better, drive the backfill from a monotonic cursor it already has such as event_id, which needs no new index on a table taking tens of thousands of inserts per second.
  6. Confirm with EXPLAIN (ANALYZE, BUFFERS) and read the plan, not the clock: check how many partitions appear, whether rows removed by filter dwarfs rows returned, and whether shared buffers read matches the size of one partition or of the table.

Follow-up

  • Where would you put the partition boundary given events arrive up to a week late, and what happens to a row whose occurred_at falls in a partition that has already been detached?
  • The delivered-today query is asked per node in local time. Does that change your partitioning, your index, or only the caller?
  • Adding a btree to this table costs write amplification. How would you decide whether the backfill is worth it?

How do you stay current with evolving software standards while working in a specialized industry?

Medium

Approach

  1. Work from the requirement backwards to the design.
  2. State your assumptions explicitly before working the problem.
  3. Say what you would check first and why it is the highest-information step.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Walk me through a challenging design project you led from concept to completion.

Medium

Approach

  1. State your assumptions explicitly before working the problem.
  2. Work from the requirement backwards to the design.
  3. Clarify what is being asked and what a complete answer contains.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Fan shipment status out to five thousand customer webhooks

Medium · fan-out · queues · backpressure · at-least-once

Customers subscribe to shipment status by webhook: 5,000 endpoints, some with an 8 second p99, some down for a day. The upstream is folded leg status, not raw observations. Requirements are per-shipment ordering at each subscriber, at-least-once delivery with a stable dedupe id, and no subscriber able to slow another. Assume 3 million legs a day, about six status transitions each, and a mean of three subscribers per shipment. Design the fan-out and quantify what folding first saves you.

Approach

  1. Fan out transitions, not observations, and do the arithmetic before designing delivery. Raw ingest averages 9,000 observations a second, about 780 million a day. Three million legs at six transitions each is 18 million notification sources a day, 208 per second, and three subscribers apiece is 54 million deliveries a day, 625 per second. Folding first is worth a factor of about 43 - 208 sources per second instead of 9,000, and 625 deliveries per second instead of roughly 27,000 - and that factor dominates every subsequent optimisation, so settle it before tuning delivery.
  2. Keep delivery state as a durable per-subscriber cursor over a per-subscriber queue rather than a row per attempt. At 54 million deliveries a day a row per attempt is roughly 19.7 billion rows a year before a single retry is counted, and retries only add to it; a cursor plus a bounded in-flight window gives retry and progress without any of them. Per-event rows earn their cost only where you must later prove a specific delivery in a dispute, so scope them to the subscribers who contract for that rather than to all 5,000.
  3. Preserve ordering deliberately: a bounded in-flight window greater than one can reorder. Give each subscriber's queue a small number of lanes keyed by hash(shipment_id), so order is strict within a lane and parallelism comes from lanes. Ordering is then a property of the key, not of a hopeful single-threaded consumer.
  4. Isolate subscribers with a per-subscriber concurrency cap drawn against a shared worker pool. By Little's law an 8 second endpoint receiving 20 notifications per second wants 160 workers in flight; without a cap it takes them from the pool and starves the other 4,999. The cap converts a slow subscriber into its own growing backlog instead of a shared outage.
  5. Handle the dead endpoint as a bounded liability: circuit-break after consecutive failures, stop attempting, let the cursor lag, and retain the queue to a fixed horizon such as 72 hours. At the horizon, advance the cursor, record a gap, and tell the subscriber to re-sync from the pull API. Unbounded retention for one dead customer is how the fan-out store becomes the next incident.
  6. Make at-least-once an explicit contract rather than an accident: every payload carries a stable event id and the leg's status_rank, so a customer can discard a duplicate or a stale retry on their side, and that requirement is documented rather than discovered.

Worked solution (30 min)

  1. Derive the notification rate from legs per day, transitions per leg and subscribers per shipment, set it beside the raw observation rate, and state the reduction factor that justifies folding first.
  2. Choose between per-event delivery rows and a per-subscriber cursor, and show how the chosen shape still preserves per-shipment ordering under retry.
  3. Set a per-subscriber concurrency cap and compute an 8 second endpoint's in-flight occupancy at its share of traffic.
  4. Write the circuit-breaker states, the retention horizon and the re-sync contract for a subscriber that returns after a day offline.

Expected result: Notifications sourced from folded transitions - about 208 per second and 625 deliveries per second, against 9,000 observations per second raw, a factor of about 43 - carried on per-subscriber queues with lanes keyed by shipment so ordering holds while lanes run in parallel; a per-subscriber concurrency cap bounding a slow endpoint's occupancy; a circuit breaker with a bounded retention horizon and a re-sync path; and an explicit at-least-once contract with a stable dedupe id.

Verify it yourself

  • A subscriber that stops responding for 24 hours changes nothing for the other 4,999.
  • Two notifications for one shipment cannot arrive at a subscriber out of order, including after a retry.
  • Retention is bounded and there is a defined, documented behaviour when the horizon is exceeded.
  • The volume reduction from folding first is quantified and the ratio matches the stated rates - about 43 times, 9,000 observations per second down to 208 transitions - rather than asserted.

Follow-up

  • A customer demands exactly-once. What can you actually offer, and what must they implement to get the effect they want?
  • One subscriber covers 40 percent of all shipments. What breaks first in a per-subscriber queue design, and what do you shard on instead?
  • How do you replay a day of notifications for one subscriber after a bug on your side, without re-sending to the other 4,999?

Orchestrator memory climbs a gigabyte a day at flat throughput

Medium · memory leak · timers · heap analysis

The fulfilment orchestrator's resident memory rises about 900 MB a day and old-generation occupancy after each full collection rises monotonically; a restart resets both. Transition throughput and latency are unchanged, and saga state itself lives in the database. The process keeps an in-heap timer wheel with one timeout entry per forward step of each order line, and lines legitimately sit in one state for days. Diagnose in order, prove the hypothesis with arithmetic before changing code, and give the fix.

Approach

  1. Distinguish a leak from a large working set first, because the remedies are opposite. Plot occupancy immediately after each full collection, not resident memory: a monotone rise across days is retention, while a rise that plateaus is a cache doing its job. Live sagas legitimately number in the millions here, so the working set is genuinely large and the question is only whether it is bounded.
  2. Diff two heap dumps taken two hours apart by retained size per dominator rather than guessing at the suspect. The growth will show as timer entries that transitively pin whole saga objects; the reference path from the GC root through the wheel's bucket array to the entry to the saga is what names the owner, and without that path you are guessing.
  3. Prove the hypothesis numerically before touching code. 900 MB a day divided by roughly 1.5 KB retained per entry is about 600,000 entries a day, near seven a second. Compare that to entries created per second minus entries removed per second. If the rates do not match within an order of magnitude, the dominator is something else and the timer wheel is a bystander.
  4. Name the defect precisely: the entry is registered when a step starts and is only released when its timeout fires, so a step that completes early leaves an orphan that survives until its deadline, and a step whose line is cancelled or re-sourced leaves one that may never be cleaned at all. Age cannot be used to identify orphans, because a line sitting in one state for days is normal.
  5. Fix by cancelling on transition, keyed by (order_line_id, step), so the entry is removed by the same code path that advances the saga. Then move the timer out of the heap entirely: persist due_at on the saga row and let a sweeper poll WHERE due_at < now() AND status IN (open states) behind an index on due_at partial to those states. That bounds memory by the poll batch instead of by open sagas, and fixes the separate bug that an in-heap wheel silently loses every pending timeout on restart or deploy.
  6. Add a bound so the next version of this is loud. Cap the in-flight timer structure, emit a metric on entries created versus removed, and alert on the difference rather than on memory, which only moves after the damage is done.

Follow-up

  • The sweeper now polls millions of open sagas. What does the index look like and how do you keep the poll from scanning the whole table?
  • Two orchestrator instances poll the same due sagas. How do you stop both from firing the same timeout?
  • How would you have detected the lost-timeouts-on-restart bug, given that it produced no error and no memory symptom?

Suggested practice checklist

Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.

Day 1 — Diagnostic, scored before you study anything

  • Sit a 110-minute diagnostic in four blocks: forty-five minutes on two coding problems, twenty-five on one design prompt taken to interface and data model, twenty of short-answer fundamentals, and twenty delivering two behavioural answers aloud.
  • Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct and 0 is stuck, grading the artifact rather than how the attempt felt.
  • Allocate days two to five in proportion to 3 minus each block's score, write the allocation down, and commit to leaving it alone.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

  • Split the weakest area into named sub-skills and rate each separately. For coding those are restating the problem, choosing the structure, stating the invariant, turning the invariant into loop bounds, handling empty and single-element input, and accounting for complexity out loud.
  • Attempt three items positioned just above where the rating drops off, and for each write the first move you failed to make.
  • Re-attempt one of them from blank four hours later with nothing open.

Deliverable: A sub-skill map with the two blocking sub-skills circled.

Day 3 — Drill the blocking sub-skill by repeating the shape

  • Do eight short repetitions of the same shape rather than eight different problems, so what gets practised is the pattern and not the puzzle.
  • State the rule you now hold in one sentence, then test it against a case built to break it, a sliding window over an array containing negative values, or a cache-aside read path whose invalidation message is dropped.
  • Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on the strongest area

  • Run the same sub-skill decomposition on the second-largest gap in half the time.
  • Spend twenty-five timed minutes on the block you scored highest, choosing the hardest item you can still finish rather than a warm-up.
  • Write whether each area fails you on recall, on setup, or on execution, and set the fix accordingly: repetition for recall, a written checklist for setup, timed work for execution.

Deliverable: A second sub-skill map plus a one-line failure diagnosis for each area.

Day 5 — The gap that is not a skill

  • Record one technical and one behavioural answer, then count two things in the playback: seconds before your first clarifying question, and sentences you began without knowing where they would end.
  • Practise saying that you do not know, followed by how you would find out, without letting it soften into a guess, and practise stating a complexity or an estimate before being asked for it.
  • Redeliver one answer under a hard ninety-second cap, which forces structure ahead of detail.

Deliverable: Two recordings with a counted reduction in time-to-first-question.

Day 6 — Retest under day-one conditions

  • Sit the same 110-minute structure with new prompts of comparable difficulty and score it on the identical rubric.
  • For any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
  • Write down which single block you would still lose the offer on.

Deliverable: A second scored rubric placed beside the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

  • Run a sixty-minute mock over the two blocks that moved least, with an interviewer briefed to interrupt and change direction mid-answer.
  • Write the recovery script for going blank: restate the question, state your assumption, name the first thing you would check.
  • Say every rule from the week aloud without reading it, and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive an interruption.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

Risky changes and the reversal path

For anything that touched live traffic, be ready to say how you would have undone it: a flag, a staged rollout, dual writes with the old path still authoritative. Once the old column is dropped or the source rows are overwritten there is no reverse, so name what you kept a copy of and for how long.

Give an example of how you have mentored a colleague or contributed to a more collaborative team culture.

Medium · behavioural and engineering judgement

Approach

  1. State the situation in two sentences and spend the rest on the reasoning.
  2. Pick a story where you made the decision, not one where you watched it.
  3. Give the blast radius: what could have broken, and what you measured.

Follow-up

  • How did you know your change caused the improvement?
  • What would you do differently if you ran that again?

Can you describe a time you had to troubleshoot a complex technical issue in a production environment?

Medium · behavioural and engineering judgement

Approach

  1. Give the blast radius: what could have broken, and what you measured.
  2. Close with what you would do differently, concretely.
  3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

  • What would you do differently if you ran that again?
  • How did you know your change caused the improvement?

Tell me about a time you had to explain a highly technical concept to a non-technical stakeholder.

Medium · behavioural and engineering judgement

Approach

  1. State the situation in two sentences and spend the rest on the reasoning.
  2. Pick a story where you made the decision, not one where you watched it.
  3. Give the blast radius: what could have broken, and what you measured.

Follow-up

  • How did you know your change caused the improvement?
  • What would you do differently if you ran that again?

How do you handle disagreements regarding technical direction or project design?

Medium · behavioural and engineering judgement

Approach

  1. Name the disagreement and how you resolved it with evidence.
  2. Pick a story where you made the decision, not one where you watched it.
  3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

  • What would you do differently if you ran that again?
  • How did you know your change caused the improvement?