cswork / libraryOpen practice workspace

Interview guides · Amcor

Amcor Software Engineer Interview Questions & Guide 2026

Amcor software engineer prep on operations: decide what you would need to see to know a release is going wrong.

The 60-second brief

A Software Engineer at Amcor plays a pivotal role in bridging the gap between cutting-edge digital solutions and large-scale industrial manufacturing. As a global leader in responsible packaging solutions, Amcor relies on its engineering talent to optimize production lines, enhance operational excellence, and drive sustainability through innovative technology. You are not just writing code; you are building the systems that allow a complex, global supply chain to function with precision and efficiency.

Browser-facing seats are not covered by algorithm practice. Scope in state ownership, what the page does on a slow or failed request, and how you would diagnose something that renders correctly but feels slow.

Focus areas

  • Make every external side effect safe to retry
  • Reconcile a movement ledger against derived positions
  • Model units, lots and state machines precisely

Overview

In this role, you will often find yourself working at the intersection of software development and physical production. Whether you are automating quality control processes, integrating data systems for supply chain management, or designing tools to support sustainable, recyclable packaging initiatives, your work has a direct, measurable impact on the business. The environment is fast-paced and highly collaborative, requiring you to communicate complex technical requirements to stakeholders ranging from plant managers to corporate leadership.

Success at Amcor requires a blend of technical proficiency and a pragmatic, solution-oriented mindset. You should be prepared to navigate the challenges of a traditional manufacturing environment while advocating for modern digital transformation. This role is ideal for engineers who enjoy seeing the tangible impact of their software in a real-world, high-stakes industrial context.

The Amcor interview process

Amcor candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

  • Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
  • Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
  • Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

  • Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
  • Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
  • Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Hiring Manager Interviews

This round is a resourcing decision in the shape of a conversation: how large a piece of work can be handed to you with a one-line brief and no check-in for two weeks. The manager is listening for the seams in your story, the places where you settled something yourself and the places you went back for a ruling. Most candidates describe the system and skip the decisions, which reads as having been present rather than responsible. Say who wanted the work, which option you rejected and why, and where you would have stopped and escalated.

What to demonstrate

  • Whether you can point at a design decision that was yours rather than the team's, and name the alternative you turned down and the constraint that killed it
  • What you treat as yours to settle against what you take to someone else, and how long you sit on a blocker before raising it
  • Whether the scope you claim survives a follow-up into the unglamorous part of it: the data migration, the backfill, the rollout to users who were already on the old path
  • Whether you asked what problem the request was solving before building what was literally asked for, and what changed in the design once you had the answer

How to prepare

  • Write out the brief for your last two projects exactly as it reached you, usually one sentence in a ticket, then list every question you had to answer yourself before code could be written. That list is most of what this round is asking for.
  • For one decision in each project, write down the rejected option, what you were trading against, and the piece of evidence that would have flipped you. A tradeoff you cannot argue in reverse was not really a decision.
  • Have an escalation ready: a blocker you took to your manager, what you had already tried, and the specific thing you were asking them to decide. If every example is something you handled alone, that reads as someone who does not ask.

Onsite Visit

Nobody in the room with you decides this. Interviewers typically write their rounds up separately, often before seeing anyone else's, and the outcome is settled later from those write-ups. A split panel gets resolved by whichever note carries specific evidence, so what you want out of each room is one concrete thing that person could write down: a bug you caught yourself, a trade-off you named, a decision you owned. The rest is arithmetic. The project you describe in a behavioural conversation is often the same system you sketched an hour earlier, and the two accounts have to agree.

What to demonstrate

  • Whether the scale, team size and timeline you attach to a project hold steady when that project resurfaces in a different round
  • Whether each interviewer leaves with a specific thing to cite rather than a general impression of competence
  • Whether a trade-off you defended in one round survives a challenge in another, instead of being quietly swapped for the answer the new interviewer seemed to want
  • Whether a question you have already answered earlier in the day gets the same answer at the same depth, without visible impatience

How to prepare

  • Write a one-page sheet per project fixing the figures you will quote — request volume, data size, team size, elapsed time, what broke — and say them aloud from the sheet until they come out identical every time
  • For each round on the schedule, decide in advance the one sentence you want in that person's notes, then check in a mock that you said it outright instead of leaving it to be inferred
  • Have someone ask you the same project question twice, an hour apart, and diff the two answers for numbers that moved or a trade-off that reversed

Panel Interviews

A day like this is several different games in a row, and the expensive mistake is carrying the previous one into the next room. Coding rewards narrow precision and finishing inside a timer. Design rewards breadth, stated assumptions and naming what you are deliberately not building. Behavioural rewards specificity about people and decisions. Candidates who over-engineer a coding problem they were supposed to finish, or who start sketching class hierarchies before anyone has agreed what the system has to do, are usually still playing the last round. Between rooms, name out loud which game the next one is.

What to demonstrate

  • Whether the coding round ends with something that runs and has been traced against a degenerate input, rather than an extensible design that was never finished
  • Whether a design discussion opens by agreeing on traffic shape, read-to-write ratio and what is allowed to be stale, instead of proceeding from an architecture you arrived with
  • Whether a behavioural answer names a person, a disagreement and what you did about it, rather than describing the system the story happened inside
  • Whether the opening habits still appear late in the day: restating the problem, asking for constraints, saying the plan before typing

How to prepare

  • Book three mocks of different types back to back on one afternoon and ask each interviewer afterwards which round you answered in the wrong mode
  • Write a three-line opening script per round type — coding: restate, name the approach and its cost, then type; design: ask for scale, read-write mix and what must not break; behavioural: name the person, the stakes and the decision — and run it off a card so the switch is mechanical rather than remembered
  • Practise coding with a timer you do not extend, stopping when it stops, so the trained reflex is to finish a correct solution rather than to keep improving one

Where candidates lose points

Floating-point quantities and implicit unit-of-measure conversions

Binary floating point cannot represent most decimal fractions exactly, so a chain of pallet-to-case-to-each conversions accumulates a residue that a later rounding turns into a unit that was created or destroyed, breaking conservation with no concurrency involved at all. The bug is slow and non-local: it surfaces as a cycle-count variance weeks later, at a node nobody changed, and it is untraceable because the arithmetic looked correct at every individual step. The fix is to hold quantities as integers in the smallest transacting unit, carry the UoM explicitly on every row that carries a number, store conversion factors as integers that are versioned in time, and reject a conversion that does not divide exactly instead of rounding it. The same discipline applies to money on the freight side, where an accessorial in floating-point dollars produces invoice reconciliation differences of a cent that cost more to investigate than the freight.

Retrying an irreversible external call without an owned idempotency key

Timeouts and connection resets are ambiguous by construction - the partner may have committed before the response was lost - so a generic retry policy around an HTTP client is, in this domain, a machine for buying two labels and dispatching two trucks. Relying on the partner's deduplication is not a substitute, because their window is usually short, their key is often derived from fields you may legitimately change on retry, and many older integrations have no such concept at all. The workable pattern is to generate the key yourself, persist it with an explicit unknown status before the call, and resolve ambiguity by querying the partner for that key rather than re-issuing; the sweeper that does this is the component that has to be correct, not the call site. Teams get this right for payments and then forget that a carrier tender, a warehouse work release and an EDI shipping notice have the same shape.

Sorting when the problem never required a total order

Match the algorithm to the guarantee actually needed: the top k comes from a size-k heap in O(n log k) time and O(k) space, distinctness needs a set rather than an ordering, and a small bounded integer key range admits a linear counting pass. A full O(n log n) sort is the right default only when you genuinely need everything in order.

Treating a network call as though it were a local function call

A remote call can be slow, fail, or return after you stopped waiting, so name the timeout, the retry policy, and what the caller sees while the dependency is down. A call with no timeout turns one slow dependency into an exhausted thread or connection pool in every service upstream of it.

Questions & practice

Trace a suspect lot forward to every shipped order line

Medium · graph traversal · genealogy · indexing

lot_edge(parent_lot_id, child_lot_id) has 400 million rows with btree indexes on both (parent_lot_id, child_lot_id) and (child_lot_id, parent_lot_id). lot_shipment(lot_id, order_line_id) maps lots to shipped lines. Given one suspect received lot, return every downstream order_line_id. The reachable subgraph can reach 2 million lots and 12 levels; the full edge table does not fit in memory. Give the traversal, its complexity in terms of the reachable subgraph, and what you do if the data contains a cycle.

Approach

  1. Genealogy is a directed acyclic graph, not a tree: a production run consumes several lots and one lot splits across many shipments, so a node is reached by several paths and a tree traversal revisits it combinatorially. Breadth-first from the suspect lot over the (parent_lot_id, ...) index, with a hash set of visited lot ids, is O(V' + E') where V' and E' count only the reachable subgraph — 2 million 8-byte ids is 16 MB of payload and a few tens of MB with open-addressing overhead, which is why the visited set is affordable and the edge table's 400 million rows never enter memory.
  2. Batch each level into one indexed read. Issue WHERE parent_lot_id = ANY($1) with a few thousand parents per call instead of a query per node: at 12 levels and 2 million nodes, per-node round trips dominate the wall clock entirely, and the recall's deadline is a real-world response time, not a query-plan aesthetic. The batch size is the tuning knob between round-trip count and per-statement planning cost.
  3. Keep the visited set even though the schema says DAG. Rework and repack loops — material returned into an earlier lot — do occur, and an unguarded traversal on a cyclic edge set does not terminate; a set membership test is O(1) and turns a hang into a correct answer plus a data-quality alert. Also cap the traversal by level and by node count, and fail loudly on the cap rather than returning a partial set silently.
  4. Join to lot_shipment once at the end over every visited lot id, not over the leaves and not per node. A lot is routinely both shipped and consumed: half a lot goes out on an order line while the other half is repacked into a child lot, so it has a lot_shipment row and outgoing edges at the same time and is not a leaf. Restricting the join to nodes with no outgoing edges drops exactly those order lines, which is under-recall — the one failure mode this trace is not allowed to have. One bulk join against an index on lot_id over the whole visited set is a single pass; per-node lookups multiply the round-trip problem by the number of visited lots.
  5. State the semantics honestly: reachability answers which shipped lines could contain material from the suspect lot, and without propagating edge quantities it is a superset of physical containment. For a recall that is the correct direction of error — over-recall, never under-recall — and saying so is the point, because a trace that returns quickly and incompletely will be believed. The upstream question, which received lots are in a suspect shipped unit, is the same traversal over the reverse index, which is why both indexes exist.

Follow-up

  • A recursive CTE would express this in ten lines. What is its cost here if the index on the traversal column is missing, and how would you notice?
  • You need the quantity of suspect material in each downstream line, not just reachability. What has to be on the edge, and what does that do to the traversal?
  • The traversal must run while a production run is writing new edges. What does a consistent answer mean here, and what do you cut it at?

Size dock doors from overlapping appointment windows

Medium · sweep line · intervals · time zones

For one node and one local calendar day you have up to 200,000 planned dock appointments from shipment_leg: leg_id, planned_arrive_at, planned_depart_at (both TIMESTAMPTZ), plus the node's IANA zone. A door is occupied over [arrive, depart). Return the minimum number of doors that lets every appointment start on time, and the maximal interval over which that peak is sustained. Target O(n log n). Say how you derive the day's boundaries and how you break ties between an arrival and a departure at the same instant.

Approach

  1. Name the result before computing it: with one interchangeable door type, the minimum door count equals the maximum number of simultaneously occupied intervals. That equality is not a heuristic — interval graphs are perfect, so their chromatic number equals their clique number, and greedy left-to-right assignment achieves it. Say this, because it is what licenses solving a scheduling question with a counter.
  2. Emit 2n endpoints, (t, +1) at each arrival and (t, -1) at each departure, sort by t, and at equal t order the -1 before the +1. Half-open occupancy means a trailer leaving at 10:00 frees the door for one arriving at 10:00; ordering arrivals first inflates the answer by exactly the number of back-to-back handoffs, which on a well-packed schedule is most of them.
  3. Scan once with a running counter, tracking the maximum and the endpoint index where it was first reached. The peak is attained on a half-open interval between two consecutive endpoints, [t_i, t_{i+1}), not at an instant — report it that way or the operations team cannot act on it. Extend the interval while the counter stays at the maximum.
  4. Complexity: O(n log n) dominated by the sort, O(n) space. If rows already arrive ordered by planned_arrive_at from an index, use the min-heap variant instead — push each departure, pop all departures at or before the current arrival, and the heap size is the current occupancy. Same time bound, O(peak) space rather than O(n), which matters when peak occupancy is 40 and n is 200,000.
  5. Derive the day's boundaries from the zone, not from arithmetic on instants. Convert local midnight and the next local midnight to instants in the node's IANA zone; on a transition day that span is 23 or 25 hours, so adding 86,400 seconds silently drops or duplicates an hour of appointments. Decide explicitly what a zero-length dwell means — under half-open semantics it occupies nothing — and state it.

Follow-up

  • Doors are typed: some break pallets, some are parcel-only. Does max-overlap still equal the door count?
  • Appointments have a tolerance — a trailer may start up to 20 minutes late without penalty. How does that change the objective, and is it still solvable by a sweep?
  • Half the appointments are actuals and half are plans. Which do you sweep for tomorrow's staffing, and which for last week's utilisation report?

Find the hottest write keys under a fixed memory budget

Medium · top-k · heap · streaming sketches

One day of writes to inventory_movement is 200 million rows; each carries (item_id, node_id). Distinct pairs reach 60 million. Return the 200 pairs with the most writes so they can be moved behind a single writer before the next peak. First give the exact answer assuming you may hold 60 million counters, then give an answer under a 200 MB budget, and state the error guarantee that second answer carries. Both must be one pass over the day's rows unless you argue for a second.

Approach

  1. Exact version: one hash pass to build (item_id, node_id) -> int64, then a single scan of the map maintaining a min-heap of size K = 200, pushing and popping when the incoming count exceeds the heap root. O(n + d log K) with n = 2e8 and d = 6e7, versus O(d log d) if you sort all 60 million entries — a factor of roughly log d / log K, about 3.4x, for no benefit. The heap must be a min-heap: the root is the weakest survivor, so eviction is O(log K).
  2. Cost the map honestly. 60 million entries at a 16-byte key, an int64 count and per-entry overhead is 2-3 GB in most runtimes, which is the whole reason for the second version rather than a theoretical aside.
  3. Bounded version: Misra-Gries with m counters. Keep at most m keys with counts; on a key already present increment it; on a new key when fewer than m are held, insert at 1; otherwise decrement every held counter and drop those reaching zero. With m = 1,000,000 at roughly 40 bytes per counter, that is about 40 MB, comfortably inside 200 MB alongside the heap.
  4. State the guarantee as a number, not as 'approximate'. Misra-Gries returns an estimate satisfying f - n/(m+1) <= f_hat <= f, so with n = 2e8 and m = 1e6 every count is an undercount by at most 200. If the 200th hottest pair has hundreds of thousands of writes, a 200-count error cannot reorder the top of the list — and if the counts near rank 200 are within 200 of each other, the ranking there was never meaningful for the decision being made. The sketch also merges: run one per shard, combine, prune back to m, and the bound holds with n the combined length.
  5. If exactness is genuinely required, argue for two passes rather than more memory: the Misra-Gries pass yields a candidate superset of size m, and a second pass counts only those candidates exactly in O(m) memory. That is the standard way to get an exact top-K under a memory bound, and the cost is one extra read of the day's rows.

Worked solution (25 min)

  1. Implement the exact path first with a hash map and a size-200 min-heap, and record its peak resident memory on a scaled-down input.
  2. Implement Misra-Gries with an explicit decrement step, and assert after every insert that the held-counter count never exceeds m.
  3. Generate a synthetic stream with a known Zipf-ish head: 200 planted heavy keys plus a long tail, so the true top-K is known.
  4. Run both, diff the two top-200 lists, and measure the largest per-key undercount against the n/(m+1) bound.
  5. Shard the stream into eight parts, run eight sketches, merge and prune, and confirm the merged result still respects the bound.

Expected result: Both paths return the same 200 keys on the planted input; every Misra-Gries estimate is at or below the true count and never short by more than n/(m+1); the merged eight-shard sketch matches the single-stream sketch's guarantee.

Verify it yourself

  • No estimate ever exceeds the true count — an overestimate means the decrement step is wrong.
  • Peak memory of the bounded path is flat as the number of distinct keys grows, while the exact path's grows linearly.
  • Shuffling the stream changes individual estimates but never violates the bound.

Follow-up

  • The decision downstream is which keys get a single writer. Does a rank error at position 190 change that decision, and what does the answer tell you about how much accuracy to buy?
  • Write volume on one key is bursty — a promotion concentrates it into twenty minutes. Does a daily top-K find it, and what window would?
  • How do you keep the sketch across a worker restart without replaying the day?

Two scan-event queries that stopped using their indexes

Medium · indexing · partition pruning · brin correlation

observation_event is range-partitioned monthly on occurred_at, with a btree on (subject_type, subject_id, occurred_at), a BRIN on occurred_at and a GiST on geo. Rows are inserted in received_at order and events are routinely hours late. Two queries regressed: (A) WHERE occurred_at::date = current_date AND observed_status = 'delivered'; (B) a backfill scanning WHERE received_at >= now() - interval '1 day'. For each, say why no index is used and no partition is pruned, and give the fix.

Approach

  1. Query A wraps the column in a cast, so the predicate is over occurred_at::date and not over occurred_at. A btree or BRIN on the bare column cannot be probed, and the planner has no range to prune partitions with, so it plans every monthly partition. Fix it by rewriting to a half-open range: occurred_at >= :start AND occurred_at < :start + interval '1 day', with :start computed by the caller from the relevant IANA zone.
  2. Do not reach for an expression index as the first fix here. The timestamptz-to-date cast is STABLE rather than IMMUTABLE because it depends on the session TimeZone, so PostgreSQL refuses to index it; indexing ((occurred_at AT TIME ZONE 'UTC')::date) is accepted but silently pins the day boundary to UTC, which is the wrong day for any stop outside it.
  3. Query B filters a column that is neither indexed nor the partition key, so it is a full scan of every partition by construction. The subtler half is why the BRIN cannot rescue it: BRIN stores a min and max per block range and only pays when physical order correlates with the column. Rows arrive in received_at order while the indexed column is occurred_at, and late events interleave hours-old timestamps with current ones, so each range's min and max span days and almost every range qualifies.
  4. Measure the correlation rather than assuming it: SELECT attname, correlation FROM pg_stats WHERE tablename = 'observation_event'. Near 1 means BRIN earns its keep, near 0 means it is a full scan with extra steps. Expect received_at near 1 and occurred_at well below it on this insert pattern.
  5. Fix B with a btree on received_at, which stays compact because it is correlated with insert order, or better, drive the backfill from a monotonic cursor it already has such as event_id, which needs no new index on a table taking tens of thousands of inserts per second.
  6. Confirm with EXPLAIN (ANALYZE, BUFFERS) and read the plan, not the clock: check how many partitions appear, whether rows removed by filter dwarfs rows returned, and whether shared buffers read matches the size of one partition or of the table.

Follow-up

  • Where would you put the partition boundary given events arrive up to a week late, and what happens to a row whose occurred_at falls in a partition that has already been detached?
  • The delivered-today query is asked per node in local time. Does that change your partitioning, your index, or only the caller?
  • Adding a btree to this table costs write amplification. How would you decide whether the backfill is worth it?

Model the movement ledger so a retried write is a no-op

Easy · append-only ledger · idempotency · audit history

inventory_movement is append-only: movement_id, item_id, node_id, lot_id, state, delta_qty (signed BIGINT), uom_code, reason_code, ref_type, ref_id, idempotency_key, occurred_at, recorded_at, reversed_by. inventory_position is keyed (item_id, node_id, lot_id, state) with qty, version, last_movement_id. Give the DDL constraints and the exact statements a pick confirmation runs so that a client which times out after the commit and retries changes nothing. Then say how you correct a pick recorded against the wrong lot two days ago, and why the position carries CHECK (qty >= 0) rather than clamping at zero.

Approach

  1. Put the guarantee in the schema, not the handler: UNIQUE (idempotency_key) on inventory_movement. A SELECT-then-INSERT cannot work, because two concurrent retries both read nothing before either commits and both then insert.
  2. Claim with INSERT ... ON CONFLICT (idempotency_key) DO NOTHING RETURNING movement_id, and branch on rowcount. Zero rows means the first attempt already committed, so the handler returns success without touching the position. This branch is the whole point: applying the position update on the conflict path is what makes the ledger and the shelf disagree by one pick forever.
  3. Keep the insert and the position update in one transaction: UPDATE inventory_position SET qty = qty + :delta, version = version + 1, last_movement_id = :id WHERE item_id = :i AND node_id = :n AND lot_id = :l AND state = :s AND version = :v, then test the rowcount and retry the transaction on zero.
  4. Use lot_id BIGINT NOT NULL DEFAULT 0 rather than NULL. In a default unique index NULLs are distinct from each other, so a NULL lot would let two position rows exist for the same untracked item and split the running sum in half.
  5. Correct the mis-keyed pick with two compensating movements, never an UPDATE or a deleted_at flag: one reversal restoring the wrong lot, one fresh pick against the right lot, both with reason_code = 'reversal' or 'pick' and ref pointing at the original movement_id. The original row keeps its values; reversed_by is a back-pointer written in the same transaction so a reader does not need a reverse scan to see that the row was undone.
  6. CHECK (qty >= 0) aborts the transaction and surfaces a real defect. GREATEST(qty + :delta, 0) writes a plausible number instead, so the missing receipt that caused the negative is never investigated and the first symptom is a picker at an empty location.

Worked solution (20 min)

  1. Write the two CREATE TABLE statements including UNIQUE (idempotency_key), the composite primary key on the position, CHECK (qty >= 0) and lot_id NOT NULL DEFAULT 0.
  2. Write the handler as one transaction: the ON CONFLICT DO NOTHING RETURNING insert, the rowcount branch, then the versioned conditional UPDATE with its own rowcount test.
  3. Send the same pick twice with one idempotency key and assert one movement row and a position that moved by delta once, not twice.
  4. Apply the lot correction as two new movements and show that summing delta_qty per lot now matches the shelf while both the wrong and the right lot retain a legible history.

Expected result: One movement row per idempotency key, a position equal to the signed sum of its movements, and a corrected pick represented as three rows (original, reversal, re-pick) with no row ever updated except reversed_by and the position aggregate.

Verify it yourself

  • SUM(delta_qty) per (item_id, node_id, lot_id, state) equals inventory_position.qty after the double-send and after the correction.
  • Drop the UNIQUE constraint and confirm the double-send test now fails; if it still passes, the retry is not actually racing.
  • Force a negative by replaying a pick against an empty position and confirm the transaction aborts rather than storing zero.
  • Query the history of the mis-keyed pick and confirm the original quantity and its reason code are still readable.

Follow-up

  • Two retries of the same pick arrive concurrently on different application instances. Walk through both transactions statement by statement and say which one commits a position change.
  • occurred_at from a handheld is five minutes ahead of recorded_at because the device clock drifted. Which of your queries break, and which column should each of them actually be using?
  • How long do you keep idempotency_key unique for, given the table grows by tens of millions of rows a month?

What do you know about validation protocols or quality standards?

Medium

Approach

  1. Say what you would check first and why it is the highest-information step.
  2. Work from the requirement backwards to the design.
  3. State your assumptions explicitly before working the problem.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

What program or technology do you feel the most comfortable with?

Medium

Approach

  1. Say what you would check first and why it is the highest-information step.
  2. State your assumptions explicitly before working the problem.
  3. Work from the requirement backwards to the design.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

Allocate through a regional failover with unknown replication state

Hard · failover · replication lag · consistency · durability

The inventory primary sits in one region with an asynchronous standby in another, typically 2 seconds behind. The region becomes unreachable at peak, with 3,000 allocations and 3,000 ATP reads per second in flight. Decide whether to promote, what happens to allocations that may have committed without replicating, and what ATP answers meanwhile. Then quantify the cost of making the allocation write synchronous instead, given today's 4 ms transaction and about 42 allocations per second on the hottest item-node pair.

Approach

  1. Split the decision by operation class before touching the failover mechanics. ATP reads can move to the standby at once with a stated staleness, because a 2-second-old read only mis-advertises and the allocation write re-checks anyway. Allocations cannot: promoting an asynchronous standby discards up to 2 seconds of committed allocations, several thousand at this rate, and each one is a promise already made to a customer.
  2. Recover those from the requester's durable intent, not from the database that just vanished. If the orchestrator wrote its saga step and an outbox row in one local transaction before calling, every lost allocation can be re-driven after promotion under its original idempotency key, and allocation's partial unique index makes a re-drive that did replicate fail closed instead of double-claiming. Without that intent record the loss is simply unrecoverable, and the honest design refuses allocations until the primary returns rather than pretending otherwise.
  3. Price synchronous replication in the currency that actually binds, which is hot-row throughput rather than average latency. A cross-region synchronous commit adds tens of milliseconds inside the transaction, turning a 4 ms hold into roughly 65 ms and dropping one row from about 250 to about 15 allocations per second, below the 42 demanded - so the promotion tail becomes an outage every promotion. A synchronous standby in a second zone of the same region costs 1-2 ms, leaves the hold near 6 ms and the ceiling near 170 per second, still four times demand.
  4. So the configuration is: synchronous in-region for zero data loss on a node or zone failure, asynchronous cross-region for the regional case, and promotion of the cross-region standby as a deliberate human decision with a written procedure. Automatic promotion of an asynchronous standby is how a regional blip becomes two primaries and a permanently split ledger that reconciliation cannot repair, because both halves are internally consistent.
  5. Define the degraded contracts explicitly for the gap. ATP serves the conservative bucketed answer from the standby with staleness stated; allocate returns a typed unavailability so the orchestrator holds lines in sourcing rather than cancelling or retrying them into a storm. After promotion, replay the outbox, then run the reconciliation re-sum before any node's availability is trusted again, because the re-summed positions are the only evidence of what actually survived.

Worked solution (40 min)

  1. Classify every operation as safe or unsafe on a 2-second-stale replica and write that classification down before choosing a failover posture.
  2. Trace one allocation that committed a second before the outage through promotion, and name the exact durable record that lets you re-drive it.
  3. Recompute the hot-row ceiling under in-region and cross-region synchronous commit and set both against the 42 per second demand.
  4. Write the degraded contracts: what ATP returns, what allocate returns, and what the orchestrator does with each.
  5. Write the post-promotion sequence - outbox replay, reconciliation re-sum, then re-enable - and say what is not trusted until it completes.

Expected result: Reads promoted immediately with stated staleness and writes not; lost allocations recovered by re-driving from the orchestrator's durable intent under their original idempotency keys; synchronous commit placed in a second in-region zone (hold about 6 ms, ceiling near 170 per second against 42 demanded) rather than cross-region (about 65 ms, ceiling near 15); and promotion as a deliberate decision followed by outbox replay and a reconciliation re-sum before availability is trusted.

Verify it yourself

  • Every allocation a customer was told about is either present after promotion or re-drivable from a record held outside the failed database.
  • The cost of synchronous replication is expressed as a change in the hot-row ceiling, not only as added average latency.
  • Nothing in the design permits the old primary and the promoted standby to accept allocations at the same time.
  • The degraded ATP answer errs toward under-promising and says so explicitly.

Follow-up

  • The old primary returns holding writes the standby never saw. What do you do with them, and what does the reconciliation job report about them?
  • How do you rehearse this? Describe a game day that exercises the promotion path without risking an oversell in production.
  • Which of the two regions' clocks do you trust for occurred_at ordering during the switch, and does it matter here?

One partner's leg statuses freeze while every other feed advances

Easy · poison message · head-of-line blocking · event ingest

Since 06:40 UTC no leg fed by one partner has changed status; every other feed is current. Ingest is healthy and CPU is flat, and observation_event rows from that source are still landing with current received_at and processing_status 'pending'. Consumer lag is growing on exactly one stream partition, whose head row has over 900 attempts and a byte-identical last_error on each one. Give the ordered diagnosis and the change that stops one unapplicable observation from stalling a partition again.

Approach

  1. Separate the partner not sending from us not applying, because both look identical to a customer. Rows landing with current received_at and processing_status 'pending' prove ingestion is fine and the fold is stuck; if received_at had gone stale instead, this would be a partner or transport incident and nothing in the consumer would be at fault.
  2. Read the blocked partition at its head, ordered by offset, and classify the error rather than counting it. An identical error string across 900 attempts is deterministic and implicates the payload; a varying set of connection resets, timeouts and 5xx implicates the downstream store. Only the deterministic case is a poison message.
  3. Reproduce outside the consumer by running the fold over that one event. The usual cause here is an observed_status the partner added that has no rank in the progress lattice, or a subject_id whose leg does not exist yet, so the fold throws before it can write.
  4. Decide the exit against the ordering contract, which is per subject and not per partition. Many legs hash into one partition, so blocking on one of them is a bug and not a guarantee. Park the row as processing_status 'rejected_malformed' with its offset in a dead-letter view, advance the offset, and block only the affected subject if the event is well-formed but not yet applicable.
  5. Make the retry policy classify instead of counting. Deserialisation failures, unknown enum values and constraint violations are terminal and must never be retried in place; IO errors and timeouts are retryable with bounded attempts and backoff. Alert on oldest-pending age per partition, which turns positive within seconds, rather than on lag, which drifts up slowly and looks like normal backlog.

Follow-up

  • The parked event turns out to be a real delivered scan. How do you replay it after the code fix without double-applying, and what makes that safe?
  • What should the customer-facing status show for legs that were queued behind it?
  • The partition key is currently the leg id. What breaks if you key by partner instead, and what if you key randomly?

Suggested practice checklist

Roughly ninety minutes on weeknights with one longer weekend block. The plan cuts scope rather than compressing everything, on the assumption that one thing finished per night beats four half-started.

Day 1 — Fix the scope and take a cold baseline

  • Read the role description and write the three things the loop will almost certainly test, then write an explicit not-doing list and keep it visible all week.
  • Take one twenty-five-minute coding problem and one fifteen-minute design prompt cold, and write the single sentence naming what blocked each, because those two sentences decide where the remaining evenings go.
  • Set the week's rule: one thing finished every night, including the night you only have forty minutes.

Deliverable: A one-page scope with a not-doing list and two cold attempts, each carrying one sentence on what blocked it.

Day 2 — One pattern, written three times from blank

  • Choose the single pattern most likely to appear in your loop and write it three times from an empty file rather than editing the previous attempt.
  • On the third pass, write the invariant as a comment before the loop body and the complexity before the first line of code.
  • Stop at ninety minutes even if the third version is imperfect, and write the one thing you would fix given another hour.

Deliverable: Three independent implementations of the same pattern plus a note on what changed between them.

Day 3 — One design, only to the depth you can defend

  • Take one system shape and go only as far as requirements, interface and data model, refusing to draw a box you could not survive a follow-up about.
  • Attach one number to each non-functional requirement, deriving it rather than asserting it, and write the assumption the number rests on.
  • Write the one tradeoff you are choosing against and the observation that would make you reverse it.

Deliverable: One design at interface-and-schema depth with derived numbers and one written reversible tradeoff.

Day 4 — Only the fundamentals you will have to defend

  • Write, in under two hundred words each, the answers to the two questions that follow almost any implementation: why this structure and not the obvious alternative, and what happens to this code at a hundred times the input.
  • Write what an index actually costs: faster lookups on the indexed columns against a write that now maintains a second structure, plus the cases where the planner declines to use it anyway, low selectivity, or a predicate wrapping the column in a function.
  • Delete any answer you cannot deliver aloud in under a minute, since an answer that needs reading is not an answer you have.

Deliverable: Three written answers, each under two hundred words and each timed aloud.

Day 5 — Your own work, timed

  • Write a ninety-second and a four-minute version of your main project and time both aloud rather than reading them.
  • Prepare the two follow-ups that always come: what you would do differently, and how you knew it worked.
  • Put one number in the first sentence and be ready to say exactly where it came from and what it excludes.

Deliverable: Two timed narratives with one defensible number in the opening line.

Day 6 — The one full rehearsal, in the weekend block

  • Run a sixty-minute mock covering a coding round and a design round in one sitting with no break, because sustained attention is the thing evenings have not trained.
  • Immediately afterwards, and before hearing any feedback, write the three moments you lost the thread.
  • Spend the rest of the block only on those three moments, and on nothing you merely feel shaky about.

Deliverable: Mock notes naming three failure moments with a specific fix written under each.

Day 7 — Taper

  • Write the twenty-minute warm-up you will actually do on the morning: one problem you can already solve from a blank file, one design you can narrate, and nothing you have never seen.
  • Re-read only your own notes from this week and open no new material.
  • Write the logistics down: the editor or shared document you will be working in, whether execution and lookups are permitted, and the sentence you will use when you do not know something.

Deliverable: A one-page card holding the design structure, the project numbers, and the logistics.

Review and mentoring: what you let through

Counting review comments or mentees proves nothing. The useful version is a specific change you approved with a reservation you stated, or one you blocked and the delay that cost. Say which standard you were holding and why it was worth the friction. A mentoring story needs the thing the other person can now do without you.

How do you handle working in an environment with shifting priorities?

Medium · behavioural and engineering judgement

Approach

  1. Give the blast radius: what could have broken, and what you measured.
  2. Close with what you would do differently, concretely.
  3. Name the disagreement and how you resolved it with evidence.

Follow-up

  • How did you know your change caused the improvement?
  • What did you decide not to do, and why?

What would your teammates say about you?

Medium · behavioural and engineering judgement

Approach

  1. State the situation in two sentences and spend the rest on the reasoning.
  2. Close with what you would do differently, concretely.
  3. Pick a story where you made the decision, not one where you watched it.

Follow-up

  • What did you decide not to do, and why?
  • How did you know your change caused the improvement?

What makes you a good fit for this role?

Medium · behavioural and engineering judgement

Approach

  1. Give the blast radius: what could have broken, and what you measured.
  2. Name the disagreement and how you resolved it with evidence.
  3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

  • What would you do differently if you ran that again?
  • How did you know your change caused the improvement?

Describe your biggest accomplishment at your most recent experience.

Medium · behavioural and engineering judgement

Approach

  1. Pick a story where you made the decision, not one where you watched it.
  2. State the situation in two sentences and spend the rest on the reasoning.
  3. Close with what you would do differently, concretely.

Follow-up

  • What would you do differently if you ran that again?
  • How did you know your change caused the improvement?