Interview guides · ARMADA Supply Chain Solutions

ARMADA Supply Chain Solutions Software Engineer Interview Questions & Guide 2026

ARMADA Supply Chain Solutions software engineer prep on databases: indexes, query plans, and the anomalies an isolation level still permits.

The 60-second brief

The Software Engineer role at ARMADA Supply Chain Solutions is central to maintaining the high-performance infrastructure that powers modern supply chain logistics. As part of a team that bridges the gap between complex hardware systems and software-driven optimization, you will be tasked with solving high-stakes problems that impact global connectivity and efficiency. This is not a role for those who prefer abstract coding; it is for engineers who thrive at the intersection of infrastructure, AI-driven systems, and operational technology.

The shape of the workload matters more for your prep than the industry label does. Read-heavy serving, write-heavy ingestion and scheduled batch processing have different binding constraints and fail in different places, so find out which one the team lives in before picking design topics.

Focus areas

Reconcile a movement ledger against derived positions
Fold late, out-of-order scan events by event time
Make every external side effect safe to retry

Overview

You will be expected to contribute to products that demand high reliability and scalability. Whether you are working on Outside Plant (OSP) & Connectivity projects or supporting AI Infrastructure Products, your work directly influences how the company manages physical and digital assets. The environment is fast-paced and rigorous, requiring a blend of deep technical expertise and a pragmatic, solutions-oriented mindset. Success here means you are comfortable navigating ambiguity and building systems that stand the test of real-world deployment.

The ARMADA Supply Chain Solutions interview process

ARMADA Supply Chain Solutions candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Technical Screen

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

System Design Interview

An unlabelled round is first an information problem, and the cheapest information is free. Whoever schedules it can usually tell you how long it runs, who will be in the room and what they work on, whether you will be writing code and in what environment, and whether anything is being sent beforehand. Ask in writing so the answer is on record, then prepare for the two or three formats those answers still leave open instead of betting on one. What separates a strong candidate is not guessing right; it is having an opening that works whichever one it turns out to be.

What to demonstrate

Whether you can start work from an ambiguous brief, since tolerating a vague scope without stalling is the same thing the job asks for
Whether the questions you asked beforehand were ones that change your preparation, such as duration, medium and who is joining, rather than ones whose answers you could not have acted on
Whether you adapt when the round turns out to be something other than what you were told, instead of spending the first ten minutes visibly recalibrating

How to prepare

Send one short scheduling message asking four things: how long, who is joining and what they work on, whether you will be writing code and where, and whether to prepare anything in advance. Treat a vague reply as real information, since it means the round is loosely structured and you will be shaping it yourself.
Write one opening that works in any of the formats still open: restate in your own words what you have been asked to do, then ask which of two directions is more useful to them. Say it aloud until it stops sounding recited.
Set up for the two most likely formats before the call starts, with a blank editor in the language you would choose and a shared document you can type into, so a format surprise costs you nothing in the first minutes

Behavioral Interview

What you say here is written down by each interviewer and compared afterwards, so the unit of evaluation is a claim someone else could check, not a well-told narrative. Two things make a story checkable: detail only a participant would hold, and a clean line around which part was yours. Vague ownership is the usual failure and it is usually accidental, because engineers say we about the team's work and we about their own, so the thing they personally built disappears into the plural. Name the part you wrote, and name who did the rest.

What to demonstrate

Whether your details are ones a participant would hold and an observer would not: the constraint that ruled out the obvious approach, the first attempt that failed, the person who objected and on what grounds
Whether ownership survives a direct question, since a follow-up to we decided is routinely who decided, and an answer that stays plural at that point is read as the work belonging to someone else
Whether the numbers you quote are ones you would say identically to a former colleague with the dashboard open

How to prepare

Go through each story replacing every we with either I or a named role (the on-call engineer, the reviewer, the other team) and check the story still holds together. Wherever it stops making sense you have found a part you cannot actually speak to
Open the artefacts for two of your stories, the pull request, the design doc, the incident notes, and read them for dates and figures you have been rounding in the retelling. Correct your version to match
For each story write the single sentence you would least want repeated to a former teammate, then either make it accurate or take it out

Final Decision-Making

Where the day includes a partner from product, design or data, that conversation is weighted like the technical ones and prepared for least. They are deciding one thing: whether having you in the room makes their decisions cheaper. That means options with costs attached, not implementation detail and not "it depends". An estimate someone can plan against — a range, the assumption that would push it to the high end, and what you would drop to hit the low one — is worth more than a confident single number, which everyone present already knows is wrong.

What to demonstrate

Whether an estimate comes as a range with the assumption most likely to break it, and states what a specific scope cut would actually buy
Whether a technical constraint is handed over as a choice with consequences on their side, rather than as a verdict they have no standing to argue with
Whether you establish what decision is on the table before proposing anything
Whether risk is raised while it can still change the plan, with the trigger that would confirm it, instead of reported afterwards as a slip

How to prepare

Take a project that shipped late and write the two-sentence warning you could have given three weeks earlier, naming what you would have needed decided at that point
Rehearse one estimate out loud until it arrives in three parts: the range, the single assumption that would blow it, and the smallest thing you would cut to protect the date
Rewrite an objection you have actually made — the "we can't do that" version — as two options with their costs, so the choice ends up with the person who owns it

Where candidates lose points

Judging delivery windows and cut-offs in UTC

A delivery appointment, a receiving cut-off and a same-day promise are all statements about local wall-clock time at a specific stop, so evaluating them in UTC misclassifies on-time performance for any leg that crosses a zone, and misses cut-offs by an hour on the two days a year a region changes offset. The subtlety that catches experienced engineers is that converting a future local time to a UTC instant at write time bakes in the offset rules as they were known then: when a jurisdiction changes its rules, an appointment stored that way silently moves, which is why a future window is better stored as local wall time plus an IANA zone identifier and resolved at read time, while a past event is correctly stored as an instant. Duration arithmetic needs the same care, because a transit that spans a DST boundary is genuinely 23 or 25 hours long in local terms even though the elapsed instant count is unchanged. Storing only a date with no zone - common on partner feeds - is the version of this that cannot be fixed downstream, so the zone has to be captured at ingestion.

Treating the database as the truth about physical stock

Every system in this domain drifts from reality, through miscounts, unrecorded damage, mis-scans and theft, so a design with no adjustment path forces the drift somewhere worse. The usual shape is a CHECK constraint or application guard that clamps a position at zero: the negative that would have exposed a missing receipt is silently swallowed, and the first visible symptom is a picker standing at an empty location for an order the system promised. The correct posture is to let the invariant fail loudly - record the cycle-count adjustment as a first-class movement with its own reason code, surface the variance, and let planning see that this location is unreliable - because an adjustment that is indistinguishable from a normal pick destroys the only signal that would have found the root cause. A related version of the same mistake is deleting or updating a movement row to fix a mistake, which leaves the position correct and the history a lie.

Hardcoding to the sample inputs

Solve the stated problem rather than the two examples; special-casing a literal to make a sample pass is obvious immediately and reads as either a misunderstanding or an attempt to fake progress. If you genuinely cannot generalise yet, say which part is a stub and what would replace it.

Tests that assert on the implementation rather than the behaviour

Assert on what a caller can observe, not on the number of internal calls or the shape of a private field. A test that breaks on every refactor but still passes when the answer is wrong costs more than it protects.

Questions & practice

Re-sum a movement ledger and report positions that disagree

Easy · aggregation · reconciliation · append-only ledger

inventory_movement holds 3 billion immutable rows: item_id, node_id, lot_id (NULL when the item is lot-untracked), state, delta_qty (signed int64 in the item's smallest transacting unit), uom_code, and reversed_by. inventory_position holds one row per (item_id, node_id, lot_id, state) with qty, version and last_movement_id, using lot_id = 0 as the sentinel for untracked. Up to 80 million distinct keys. Report every key whose summed movements disagree with the stored qty, while writes continue. State your time and space bounds and how you bound the comparison.

Approach

Fix the key first. inventory_movement.lot_id is NULL for untracked items and inventory_position.lot_id is 0, so the group key is (item_id, node_id, COALESCE(lot_id, 0), state). Getting this wrong does not error — it silently produces two groups that each look like a variance, and the report becomes noise nobody reads.
Cut the ledger at a watermark, but do not take W = max(movement_id) at the start of the scan. A sequence hands out an id before the inserting transaction commits, so at the instant you read that maximum there are ids below it still in flight and invisible to your snapshot. Summing movement_id <= W misses them on this run, and advancing reconciled_through_movement_id to W makes the miss permanent: every later incremental run starts above those ids and they are never summed again. That is a hole in the ledger's own re-derivation, not a transient skew.
Take a watermark that is provably settled instead. Bound write transactions with a statement or transaction timeout so 'the longest write' is a number T, record (observed_at, max_movement_id) samples periodically, and use as W_safe the largest sampled id whose observed_at is older than T — every id at or below it has committed or rolled back. Sum rows with movement_id <= W_safe, compare only against position rows with last_movement_id <= W_safe, treat anything newer as a write that raced you rather than a variance, and advance reconciled_through_movement_id only to W_safe so the next run starts there instead of re-reading 3 billion rows. If the ledger carries a commit timestamp, cutting on that is the same guarantee without the sampling table.
Sum delta_qty as int64, and include reversal rows. reversed_by is a back-pointer for audit, not an exclusion filter: an original of +10 and its reversal of -10 must both be summed to reach 0. Excluding the original while keeping the reversal produces -10 and a false variance on every corrected key.
Guard the denomination rather than trusting it. uom_code is stored per row because pack factors change over time, so reject — do not sum — any row whose uom_code is not the item's smallest transacting unit, and report those keys separately. A mixed-denomination sum is arithmetically meaningless and looks exactly like a real variance.
Hash aggregation is O(n) time and O(distinct keys) space: 80 million entries at roughly 40-56 bytes each in a typical runtime is 3-5 GB, so quote the number. The bounded-memory alternative is an external sort-merge on the group key — O(n log n) comparisons, resident memory bounded by the merge fan-in rather than by key count, and it streams — or hash-partition by hash(item_id) % P and run P independent passes for 1/P of the peak.

Follow-up

A key shows a variance of exactly one pick, repeatedly, at one node. What do you look at first, and what would distinguish a duplicate movement from a missed one?
The job takes six hours and the variance report is stale by the time anyone reads it. How would you make it incremental without losing the guarantee that it re-derives from the ledger?
Who writes the correcting movement, and what reason code does it carry?

Size dock doors from overlapping appointment windows

Medium · sweep line · intervals · time zones

For one node and one local calendar day you have up to 200,000 planned dock appointments from shipment_leg: leg_id, planned_arrive_at, planned_depart_at (both TIMESTAMPTZ), plus the node's IANA zone. A door is occupied over [arrive, depart). Return the minimum number of doors that lets every appointment start on time, and the maximal interval over which that peak is sustained. Target O(n log n). Say how you derive the day's boundaries and how you break ties between an arrival and a departure at the same instant.

Approach

Name the result before computing it: with one interchangeable door type, the minimum door count equals the maximum number of simultaneously occupied intervals. That equality is not a heuristic — interval graphs are perfect, so their chromatic number equals their clique number, and greedy left-to-right assignment achieves it. Say this, because it is what licenses solving a scheduling question with a counter.
Emit 2n endpoints, (t, +1) at each arrival and (t, -1) at each departure, sort by t, and at equal t order the -1 before the +1. Half-open occupancy means a trailer leaving at 10:00 frees the door for one arriving at 10:00; ordering arrivals first inflates the answer by exactly the number of back-to-back handoffs, which on a well-packed schedule is most of them.
Scan once with a running counter, tracking the maximum and the endpoint index where it was first reached. The peak is attained on a half-open interval between two consecutive endpoints, [t_i, t_{i+1}), not at an instant — report it that way or the operations team cannot act on it. Extend the interval while the counter stays at the maximum.
Complexity: O(n log n) dominated by the sort, O(n) space. If rows already arrive ordered by planned_arrive_at from an index, use the min-heap variant instead — push each departure, pop all departures at or before the current arrival, and the heap size is the current occupancy. Same time bound, O(peak) space rather than O(n), which matters when peak occupancy is 40 and n is 200,000.
Derive the day's boundaries from the zone, not from arithmetic on instants. Convert local midnight and the next local midnight to instants in the node's IANA zone; on a transition day that span is 23 or 25 hours, so adding 86,400 seconds silently drops or duplicates an hour of appointments. Decide explicitly what a zero-length dwell means — under half-open semantics it occupies nothing — and state it.

Worked solution (25 min)

Write the endpoint tuple with the tie-break baked into the sort key — for example (t, kind) where departure sorts as 0 and arrival as 1 — rather than handling ties in the scan body.
Implement the scan, returning both the maximum and the [t_i, t_{i+1}) span at which it holds.
Build a fixture of six appointments where three are strictly back-to-back at the same minute, and hand-compute the answer before running.
Add an appointment spanning the local DST transition and derive the day window from the zone, asserting the appointment count inside the window.
Re-run with the input shuffled to confirm the result depends only on the set of intervals.

Expected result: The back-to-back fixture needs one door, not three; the reported peak is a half-open interval between two endpoints; on the transition day the local window is 23 or 25 hours and no appointment is dropped or double-counted.

Verify it yourself

Flipping the tie-break to arrivals-first changes the answer on the back-to-back fixture — if it does not, the fixture is not actually testing the tie.
The maximum equals the largest number of intervals containing any single probe instant, checked by brute force on a small fixture.
The result is identical under any permutation of the input.

Follow-up

Doors are typed: some break pallets, some are parcel-only. Does max-overlap still equal the door count?
Appointments have a tolerance — a trailer may start up to 20 minutes late without penalty. How does that change the objective, and is it still solvable by a sweep?
Half the appointments are actuals and half are plans. Which do you sweep for tomorrow's staffing, and which for last week's utilisation report?

Find the hottest write keys under a fixed memory budget

Medium · top-k · heap · streaming sketches

One day of writes to inventory_movement is 200 million rows; each carries (item_id, node_id). Distinct pairs reach 60 million. Return the 200 pairs with the most writes so they can be moved behind a single writer before the next peak. First give the exact answer assuming you may hold 60 million counters, then give an answer under a 200 MB budget, and state the error guarantee that second answer carries. Both must be one pass over the day's rows unless you argue for a second.

Approach

Exact version: one hash pass to build (item_id, node_id) -> int64, then a single scan of the map maintaining a min-heap of size K = 200, pushing and popping when the incoming count exceeds the heap root. O(n + d log K) with n = 2e8 and d = 6e7, versus O(d log d) if you sort all 60 million entries — a factor of roughly log d / log K, about 3.4x, for no benefit. The heap must be a min-heap: the root is the weakest survivor, so eviction is O(log K).
Cost the map honestly. 60 million entries at a 16-byte key, an int64 count and per-entry overhead is 2-3 GB in most runtimes, which is the whole reason for the second version rather than a theoretical aside.
Bounded version: Misra-Gries with m counters. Keep at most m keys with counts; on a key already present increment it; on a new key when fewer than m are held, insert at 1; otherwise decrement every held counter and drop those reaching zero. With m = 1,000,000 at roughly 40 bytes per counter, that is about 40 MB, comfortably inside 200 MB alongside the heap.
State the guarantee as a number, not as 'approximate'. Misra-Gries returns an estimate satisfying f - n/(m+1) <= f_hat <= f, so with n = 2e8 and m = 1e6 every count is an undercount by at most 200. If the 200th hottest pair has hundreds of thousands of writes, a 200-count error cannot reorder the top of the list — and if the counts near rank 200 are within 200 of each other, the ranking there was never meaningful for the decision being made. The sketch also merges: run one per shard, combine, prune back to m, and the bound holds with n the combined length.
If exactness is genuinely required, argue for two passes rather than more memory: the Misra-Gries pass yields a candidate superset of size m, and a second pass counts only those candidates exactly in O(m) memory. That is the standard way to get an exact top-K under a memory bound, and the cost is one extra read of the day's rows.

Follow-up

The decision downstream is which keys get a single writer. Does a rank error at position 190 change that decision, and what does the answer tell you about how much accuracy to buy?
Write volume on one key is bursty — a promotion concentrates it into twenty minutes. Does a daily top-K find it, and what window would?
How do you keep the sketch across a worker restart without replaying the day?

Model the movement ledger so a retried write is a no-op

Easy · append-only ledger · idempotency · audit history

inventory_movement is append-only: movement_id, item_id, node_id, lot_id, state, delta_qty (signed BIGINT), uom_code, reason_code, ref_type, ref_id, idempotency_key, occurred_at, recorded_at, reversed_by. inventory_position is keyed (item_id, node_id, lot_id, state) with qty, version, last_movement_id. Give the DDL constraints and the exact statements a pick confirmation runs so that a client which times out after the commit and retries changes nothing. Then say how you correct a pick recorded against the wrong lot two days ago, and why the position carries CHECK (qty >= 0) rather than clamping at zero.

Approach

Put the guarantee in the schema, not the handler: UNIQUE (idempotency_key) on inventory_movement. A SELECT-then-INSERT cannot work, because two concurrent retries both read nothing before either commits and both then insert.
Claim with INSERT ... ON CONFLICT (idempotency_key) DO NOTHING RETURNING movement_id, and branch on rowcount. Zero rows means the first attempt already committed, so the handler returns success without touching the position. This branch is the whole point: applying the position update on the conflict path is what makes the ledger and the shelf disagree by one pick forever.
Keep the insert and the position update in one transaction: UPDATE inventory_position SET qty = qty + :delta, version = version + 1, last_movement_id = :id WHERE item_id = :i AND node_id = :n AND lot_id = :l AND state = :s AND version = :v, then test the rowcount and retry the transaction on zero.
Use lot_id BIGINT NOT NULL DEFAULT 0 rather than NULL. In a default unique index NULLs are distinct from each other, so a NULL lot would let two position rows exist for the same untracked item and split the running sum in half.
Correct the mis-keyed pick with two compensating movements, never an UPDATE or a deleted_at flag: one reversal restoring the wrong lot, one fresh pick against the right lot, both with reason_code = 'reversal' or 'pick' and ref pointing at the original movement_id. The original row keeps its values; reversed_by is a back-pointer written in the same transaction so a reader does not need a reverse scan to see that the row was undone.
CHECK (qty >= 0) aborts the transaction and surfaces a real defect. GREATEST(qty + :delta, 0) writes a plausible number instead, so the missing receipt that caused the negative is never investigated and the first symptom is a picker at an empty location.

Worked solution (20 min)

Write the two CREATE TABLE statements including UNIQUE (idempotency_key), the composite primary key on the position, CHECK (qty >= 0) and lot_id NOT NULL DEFAULT 0.
Write the handler as one transaction: the ON CONFLICT DO NOTHING RETURNING insert, the rowcount branch, then the versioned conditional UPDATE with its own rowcount test.
Send the same pick twice with one idempotency key and assert one movement row and a position that moved by delta once, not twice.
Apply the lot correction as two new movements and show that summing delta_qty per lot now matches the shelf while both the wrong and the right lot retain a legible history.

Expected result: One movement row per idempotency key, a position equal to the signed sum of its movements, and a corrected pick represented as three rows (original, reversal, re-pick) with no row ever updated except reversed_by and the position aggregate.

Verify it yourself

SUM(delta_qty) per (item_id, node_id, lot_id, state) equals inventory_position.qty after the double-send and after the correction.
Drop the UNIQUE constraint and confirm the double-send test now fails; if it still passes, the retry is not actually racing.
Force a negative by replaying a pick against an empty position and confirm the transaction aborts rather than storing zero.
Query the history of the mis-keyed pick and confirm the original quantity and its reason code are still readable.

Follow-up

Two retries of the same pick arrive concurrently on different application instances. Walk through both transactions statement by statement and say which one commits a position change.
occurred_at from a handheld is five minutes ahead of recorded_at because the device clock drifted. Which of your queries break, and which column should each of them actually be using?
How long do you keep idempotency_key unique for, given the table grows by tens of millions of rows a month?

Add a required unit column and uniqueness to a live ledger

Hard · online migration · lock timeouts · unique constraints

inventory_movement holds 4.2 billion rows on PostgreSQL 15 and takes about 2,000 inserts per second around the clock. Two changes are required with no downtime: add uom_code TEXT NOT NULL, whose truthful value must come from the conversion factors in force at each row's occurred_at rather than from today's item master; and make idempotency_key UNIQUE, although duplicates predating the retry fix already exist. Give the ordered steps, the lock each one takes, its expected duration and its rollback.

Approach

ADD COLUMN with a non-volatile DEFAULT has been metadata-only since PostgreSQL 11, so no rewrite, but it still takes ACCESS EXCLUSIVE for an instant. That lock queues behind any long-running reader while blocking every statement behind it, so one 10-minute analytics scan turns a millisecond change into a 10-minute outage. Set lock_timeout to 2s and retry in a loop rather than waiting.
Reach NOT NULL without a scan under an exclusive lock: ADD CONSTRAINT ... CHECK (uom_code IS NOT NULL) NOT VALID, which is instant, then VALIDATE CONSTRAINT, which takes only SHARE UPDATE EXCLUSIVE and runs concurrently with inserts, then SET NOT NULL, which PostgreSQL 12 and later proves from the validated check and so skips its own scan. Drop the redundant check afterwards.
The truthful backfill is the long pole and must read the versioned conversion factor as of each row's occurred_at, not the current item master, or a supplier moving a case from 12 units to 10 retroactively reinterprets last quarter. Batch by primary-key range, a few thousand rows per transaction, with a sleep between batches and a resumable cursor in a control table. At 5,000 rows a second, 4.2 billion rows is roughly ten days of throttled work: that is a schedule, not a failure, and it must not block the deploy.
The UNIQUE index cannot be built until the duplicates are gone. CREATE UNIQUE INDEX CONCURRENTLY aborts on the first violation and leaves an INVALID index behind that is ignored by queries but still carries update overhead on every insert, and it can only be removed with DROP INDEX CONCURRENTLY. So: find the duplicates, resolve each one as a compensating reversal rather than a DELETE, since every duplicate key is a double-applied movement and deleting it fixes the constraint while leaving the position wrong.
Then CREATE UNIQUE INDEX CONCURRENTLY, which makes two passes over the table and waits out concurrent transactions, so budget hours here, and note it cannot run inside a transaction block: a migration framework that wraps every step in BEGIN will fail on this one. Promote it with ALTER TABLE ... ADD CONSTRAINT ... UNIQUE USING INDEX, which is a brief ACCESS EXCLUSIVE.
Question whether the constraint should cover all history. At roughly 40 bytes per key plus overhead, a btree over 4.2 billion rows is in the hundreds of gigabytes, while a retry never arrives a year later. A partial unique index bounded by a fixed occurred_at literal, rotated forward on a schedule, gives the same protection for a fraction of the size, and the predicate must be a literal because now() is not immutable.

Follow-up

Sequence the application deploy against these steps. When exactly may the code start writing uom_code, and when may it start requiring it?
The backfill is on day six and the conversion-factor table is itself corrected. What happens to the rows already written?
How do you roll back after SET NOT NULL is in place and half the traffic depends on the column?

How do you troubleshoot intermittent failures in complex, distributed hardware-software systems?

Medium · system design and architecture

Approach

Name the failure you are designing for, then the recovery path.
State the consistency you need, and where you are willing to be stale.
Fix the scope first: who calls this, how often, and what they do when it fails.

Follow-up

What breaks first when traffic grows ten times?
How does this behave when that dependency is down for an hour?

How do you approach the design of scalable infrastructure for large-scale deployments?

Medium · system design and architecture

Approach

Name the failure you are designing for, then the recovery path.
Fix the scope first: who calls this, how often, and what they do when it fails.
Choose a partition key and say what query it makes expensive.

Follow-up

How does this behave when that dependency is down for an hour?
What would you drop to keep the system up under load?

Walk us through a time you had to design a system that had to account for significant physical constraints.

Medium · system design and architecture

Approach

Name the failure you are designing for, then the recovery path.
Name the read and write paths separately; they rarely have the same bottleneck.
Fix the scope first: who calls this, how often, and what they do when it fails.

Follow-up

How does this behave when that dependency is down for an hour?
What breaks first when traffic grows ten times?

Describe a scenario where you had to pivot your technical approach due to unforeseen site or infrastructure limitations.

Medium · system design and architecture

Approach

State the consistency you need, and where you are willing to be stale.
Name the failure you are designing for, then the recovery path.
Name the read and write paths separately; they rarely have the same bottleneck.

Follow-up

What would you drop to keep the system up under load?
How does this behave when that dependency is down for an hour?

Rate limit a retrying caller without extending unknown windows

Hard · rate limiting · load shedding · partial failure

Availability reads run a few thousand per second inside checkout against a low-tens-of-milliseconds p99; allocation writes are far lower but concentrate on a few hundred hot item-node pairs on a promotion day. Every caller retries on 5xx and on timeout. Design the limiting and shedding contract: what is metered and keyed by what, the response headers, which operations may degrade to a conservative answer instead of failing, and how throttling interacts with an allocation whose outcome is already unknown. Deliver the policy table and the caller behaviour you require.

Approach

Meter two different resources with two mechanisms. A token bucket per client credential protects the read path, where cost is CPU and fan-out; a concurrency limiter per (item_id, node_id) protects the write path, where cost is contention on one row. A client-level limit alone does not protect a hot row, and a row-level limit alone lets one client drown every row.
Fix the shed order and publish it: new reads shed first, then new allocation attempts, and the resolution path never sheds. Throttling a GET of an allocation by idempotency key strands the caller in an unknown state whose only other resolution is re-issuing the write - the limiter would then be manufacturing the duplicate it exists to prevent.
Keep the retry classes unambiguous at the wire level. 429 with a Retry-After delta (RFC 9110) means nothing happened, retry later with the same key; 504 means unknown, resolve by key, do not re-issue. Give each a distinct machine code, and publish remaining quota and window in response headers - the IETF RateLimit and RateLimit-Policy fields, or your own documented equivalent - so a well-behaved client paces itself instead of discovering the limit by hitting it.
Write the caller's obligations into the contract, not the wiki: full-jitter exponential backoff, and a retry budget capping retries at a fraction of successful requests. A bare Retry-After value returned to thousands of callers at once re-synchronises the herd on a single instant, which is the second outage.
Degrade reads, never writes. Under shed, availability may answer from the last known position with a staleness stamp and a conservative floor that subtracts in-flight allocations, and the response must say it is degraded so checkout can decide whether to show the item. Allocation must never read a replica or a cache - that is the over-allocation invariant - so under pressure the write path refuses rather than guesses, and the error direction is under-promising.
Give each hot key a single-writer queue with a bounded depth and a fast reject when full. State the depth and what it implies: queue depth divided by service rate is the wait a caller inherits, and that number must fit inside the checkout budget or the queue is just a slower failure.

Worked solution (40 min)

Write the policy table: resource, key, mechanism, limit, response on exceed, caller action, and whether it may shed.
Compute the hot-key numbers - allocations per second on one item-node pair, service time under the row lock, queue depth - and show the implied wait against the checkout budget.
Write the three wire outcomes side by side (429, 504, degraded 200) with their codes and the caller's branch for each.
Specify the degraded availability payload: the floor, the staleness stamp, and the flag that tells the caller it is degraded.
Simulate a retry storm on paper: 2,000 callers receive an identical Retry-After, then add full jitter and re-derive the arrival curve.

Expected result: A table with per-credential read metering and per-item-node write concurrency, a published shed order that exempts by-key resolution, 429 and 504 carrying distinct retry semantics, and a degraded read that under-promises and says so while allocation still refuses to read a replica.

Verify it yourself

Under full shed, a caller can still resolve an ambiguous allocation and reach a definite answer.
A degraded availability response is distinguishable from a fresh one without inspecting latency or guessing.
Load on one hot item-node pair does not raise error rates for unrelated items sharing the fleet.

Follow-up

Your largest caller is also the hardest retrier. How do you stop their retries consuming the quota their first attempts need?
What does checkout render on a degraded availability answer, and who owns the decision - you or them?
At what measured point do you stop shedding reads and start refusing to accept new orders at all?

One partner's leg statuses freeze while every other feed advances

Easy · poison message · head-of-line blocking · event ingest

Since 06:40 UTC no leg fed by one partner has changed status; every other feed is current. Ingest is healthy and CPU is flat, and observation_event rows from that source are still landing with current received_at and processing_status 'pending'. Consumer lag is growing on exactly one stream partition, whose head row has over 900 attempts and a byte-identical last_error on each one. Give the ordered diagnosis and the change that stops one unapplicable observation from stalling a partition again.

Approach

Separate the partner not sending from us not applying, because both look identical to a customer. Rows landing with current received_at and processing_status 'pending' prove ingestion is fine and the fold is stuck; if received_at had gone stale instead, this would be a partner or transport incident and nothing in the consumer would be at fault.
Read the blocked partition at its head, ordered by offset, and classify the error rather than counting it. An identical error string across 900 attempts is deterministic and implicates the payload; a varying set of connection resets, timeouts and 5xx implicates the downstream store. Only the deterministic case is a poison message.
Reproduce outside the consumer by running the fold over that one event. The usual cause here is an observed_status the partner added that has no rank in the progress lattice, or a subject_id whose leg does not exist yet, so the fold throws before it can write.
Decide the exit against the ordering contract, which is per subject and not per partition. Many legs hash into one partition, so blocking on one of them is a bug and not a guarantee. Park the row as processing_status 'rejected_malformed' with its offset in a dead-letter view, advance the offset, and block only the affected subject if the event is well-formed but not yet applicable.
Make the retry policy classify instead of counting. Deserialisation failures, unknown enum values and constraint violations are terminal and must never be retried in place; IO errors and timeouts are retryable with bounded attempts and backoff. Alert on oldest-pending age per partition, which turns positive within seconds, rather than on lag, which drifts up slowly and looks like normal backlog.

Follow-up

The parked event turns out to be a real delivered scan. How do you replay it after the code fix without double-applying, and what makes that safe?
What should the customer-facing status show for legs that were queued behind it?
The partition key is currently the leg id. What breaks if you key by partner instead, and what if you key randomly?

Suggested practice checklist

For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.

Day 1 — Measure before reasoning

Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.

Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.

Day 2 — References, copies, and the bugs they produce

Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.

Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.

Day 3 — Types, once, in a language that checks them

Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.

Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.

Day 4 — Concurrency, starting with what actually runs at the same time

Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.

Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.

Day 5 — Debugging as a procedure rather than an instinct

Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.

Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.

Day 6 — Tests that catch the bug you are about to write

Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.

Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.

Day 7 — Debug something broken, out loud

Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.

Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.

'We rewrote it in a better framework' is not a story

A migration is a cost you chose to pay, not an achievement. The story is what the old system made expensive, what you measured before committing, what kept serving traffic during the cutover, and what you would have done if the numbers had come back flat. Without those, a rewrite reads as taste.

Tell us about a time you identified a critical flaw in a project—how did you communicate this to your team?

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
Name the disagreement and how you resolved it with evidence.
Pick a story where you made the decision, not one where you watched it.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?

How do you handle performance bottlenecks in a system that requires real-time data processing?

Medium · behavioural and engineering judgement

Approach

State the situation in two sentences and spend the rest on the reasoning.
Pick a story where you made the decision, not one where you watched it.
Close with what you would do differently, concretely.

Follow-up

How did you know your change caused the improvement?
What would you do differently if you ran that again?

Argue against serving available-to-promise from a cache

Hard · caching · replication lag · over-allocation · design review

The availability read sits inside checkout with a p99 budget in the low tens of milliseconds, and it is missing it. A senior engineer proposes serving available-to-promise from a two-second in-memory cache, falling back to a read replica, and leaving the allocation write untouched. You believe this is either harmless or an overselling bug depending on one detail nobody has stated. Describe a design you argued against: the distinction you drew, the evidence you brought, the alternative you offered, and what was actually decided. Be specific about the failure you predicted and whether it happened.

Approach

Draw the distinction the room is missing before arguing anything: an advisory availability read that decorates a page is a different object from the check that gates an allocation write. Caching the first is ordinary; caching the second removes the atomicity that no-over-allocation depends on, because the check and the write must be one step.
Make the argument falsifiable with the schema rather than with principle. If the allocation still commits through UPDATE inventory_position SET qty = qty - :n, version = version + 1 WHERE (item_id, node_id, lot_id, state) = (...) AND version = :v AND qty >= :n, with the affected-row count tested, then a stale cache produces failed attempts and a worse conversion rate, not oversold stock. If the write trusts the cached number, the cache is the bug. Ask which one is on the branch.
Quantify the exposure instead of asserting it. On a hot item-node pair taking 40 allocation attempts per second with a two-second staleness window, roughly 80 attempts can be decided against an unchanged number; multiply by the fraction of pairs whose available quantity is below that to get the oversell rate, and note that both factors peak on the same day. Replication lag has the same shape and is worse, because lag grows precisely under the write burst that makes availability tight.
Offer a cheaper path to the latency goal so the argument is not simply obstruction: serve the browse and search paths from the cache with an explicit staleness label, keep the authoritative check on the primary row, and attack the p99 where it actually is by measuring whether the cost is the query, the connection pool wait, or fan-out across nodes for a multi-node promise.
Say where you were willing to lose. If the measurement showed the check was already conditional and the cache only chose candidates, the correct move is to concede quickly and say so, because an engineer who argues the same way regardless of the evidence stops being listened to.
Report the outcome honestly, including the version that shipped and whether your predicted failure appeared. A strong answer names the metric that would have proved you wrong and whether anyone put it in place.

Follow-up

The team keeps the primary check but wants the cache to prefilter which nodes are even considered. What can now go wrong that could not before?
How would you bound overselling on one very hot item-node pair without adding latency to every other pair?
What would you measure for a week to decide whether the cache is safe, and what result would change your mind?