cswork / libraryOpen practice workspace

Interview guides · Americold Logistics

Americold Logistics Software Engineer Interview Questions & Guide 2026

Americold Logistics software engineer prep on operations: decide what you would need to see to know a release is going wrong.

The 60-second brief

As a Software Engineer at Americold Logistics, you are at the heart of the digital transformation of the global cold chain. Americold Logistics is the world’s largest owner and operator of temperature-controlled warehouses, and the software you build directly impacts the efficiency, safety, and reliability of the global food supply chain. Your work ensures that complex logistics operations run smoothly, from inventory management to automated facility control.

For a design discussion, a catalogue of architectures is worth less than the ability to turn a vague requirement into a data model and an API contract. A box diagram with no schema under it collapses at the first follow-up question.

Focus areas

  • Model units, lots and state machines precisely
  • Reconcile a movement ledger against derived positions
  • Reserve inventory without overselling under concurrent allocation

Overview

This role requires a blend of technical precision and operational awareness. You will move beyond simple development tasks to solve high-stakes challenges involving real-time data, complex middleware, and large-scale infrastructure. Whether you are working on internal tools, cloud integrations, or specialized logistics software, your contributions are essential to maintaining the operational excellence that defines Americold Logistics.

The Americold Logistics interview process

Americold Logistics candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Half of this call is the part candidates treat as small talk: start date, notice period, work authorisation and its timing, location and time zone, on-call, and the number. Those are what kill offers late, after several engineers have each spent a day. Surfacing a hard constraint now costs you nothing and occasionally buys you something, since a loop compressed to fit a competing deadline can usually only be arranged if it is asked for early. The common failure is deflecting the compensation question twice, then discovering at offer stage that the band never reached your number.

What to demonstrate

  • Whether your hard constraints are compatible with the role before a loop gets booked: earliest start, notice period, what authorisation you hold and when it needs action, days on site, willingness to carry a pager
  • Whether you give a compensation range with something behind it, such as current total compensation or a competing timeline, rather than leaving the band untested
  • Whether your stated timeline is real, since a competing deadline raised now is something scheduling can sometimes work around and the same deadline raised at offer stage usually is not

How to prepare

  • Write each constraint down in one line before the call and state them as facts rather than negotiating them live under a question you were not expecting
  • Set your range from two or three current data points for that level and location, and name the structure you are quoting in, so the number is comparable to the one they are holding
  • If another process is running, say where it stands and by when, and ask directly whether this loop can be scheduled inside that window

Video Interviews

An unlabelled round is first an information problem, and the cheapest information is free. Whoever schedules it can usually tell you how long it runs, who will be in the room and what they work on, whether you will be writing code and in what environment, and whether anything is being sent beforehand. Ask in writing so the answer is on record, then prepare for the two or three formats those answers still leave open instead of betting on one. What separates a strong candidate is not guessing right; it is having an opening that works whichever one it turns out to be.

What to demonstrate

  • Whether you can start work from an ambiguous brief, since tolerating a vague scope without stalling is the same thing the job asks for
  • Whether the questions you asked beforehand were ones that change your preparation, such as duration, medium and who is joining, rather than ones whose answers you could not have acted on
  • Whether you adapt when the round turns out to be something other than what you were told, instead of spending the first ten minutes visibly recalibrating

How to prepare

  • Send one short scheduling message asking four things: how long, who is joining and what they work on, whether you will be writing code and where, and whether to prepare anything in advance. Treat a vague reply as real information, since it means the round is loosely structured and you will be shaping it yourself.
  • Write one opening that works in any of the formats still open: restate in your own words what you have been asked to do, then ask which of two directions is more useful to them. Say it aloud until it stops sounding recited.
  • Set up for the two most likely formats before the call starts, with a blank editor in the language you would choose and a shared document you can type into, so a format surprise costs you nothing in the first minutes

Technical Assessments

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

  • Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
  • Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
  • Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

  • Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
  • Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
  • Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Final Interviews

A day like this is several different games in a row, and the expensive mistake is carrying the previous one into the next room. Coding rewards narrow precision and finishing inside a timer. Design rewards breadth, stated assumptions and naming what you are deliberately not building. Behavioural rewards specificity about people and decisions. Candidates who over-engineer a coding problem they were supposed to finish, or who start sketching class hierarchies before anyone has agreed what the system has to do, are usually still playing the last round. Between rooms, name out loud which game the next one is.

What to demonstrate

  • Whether the coding round ends with something that runs and has been traced against a degenerate input, rather than an extensible design that was never finished
  • Whether a design discussion opens by agreeing on traffic shape, read-to-write ratio and what is allowed to be stale, instead of proceeding from an architecture you arrived with
  • Whether a behavioural answer names a person, a disagreement and what you did about it, rather than describing the system the story happened inside
  • Whether the opening habits still appear late in the day: restating the problem, asking for constraints, saying the plan before typing

How to prepare

  • Book three mocks of different types back to back on one afternoon and ask each interviewer afterwards which round you answered in the wrong mode
  • Write a three-line opening script per round type — coding: restate, name the approach and its cost, then type; design: ask for scale, read-write mix and what must not break; behavioural: name the person, the stakes and the decision — and run it off a card so the switch is mechanical rather than remembered
  • Practise coding with a timer you do not extend, stopping when it stops, so the trained reflex is to finish a correct solution rather than to keep improving one

Where candidates lose points

Holding a database transaction open across a physical or third-party operation

It is natural to open a transaction, lock the position, call the rating or tendering API, and commit on the response, and it works perfectly until the partner's p99 goes from 200 milliseconds to thirty seconds. At that point every request holding a lock on a hot item-node pair queues behind it, the connection pool fills with transactions that are waiting on the network rather than on the database, and an unrelated service sharing the pool fails at the same moment. In this domain the effect is amplified because demand concentrates on a few hot rows during a promotion or a seasonal peak, exactly when partner latency is also degraded. The structural fix is to keep transactions short and local - commit the state change together with an outbox row, let a relay perform the external call, and reconcile asynchronously - accepting at-least-once delivery and making the effect idempotent rather than trying to stretch a transaction over something the database cannot roll back.

Retrying an irreversible external call without an owned idempotency key

Timeouts and connection resets are ambiguous by construction - the partner may have committed before the response was lost - so a generic retry policy around an HTTP client is, in this domain, a machine for buying two labels and dispatching two trucks. Relying on the partner's deduplication is not a substitute, because their window is usually short, their key is often derived from fields you may legitimately change on retry, and many older integrations have no such concept at all. The workable pattern is to generate the key yourself, persist it with an explicit unknown status before the call, and resolve ambiguity by querying the partner for that key rather than re-issuing; the sweeper that does this is the component that has to be correct, not the call site. Teams get this right for payments and then forget that a carrier tender, a warehouse work release and an EDI shipping notice have the same shape.

Sharing mutable state with no stated owner

Say which thread, request or task owns each mutable structure, and what protects it when the answer is more than one: a lock, a queue that hands ownership across, or an immutable copy per reader. A structure documented as safe for concurrent reads is usually not safe for a concurrent write alongside those reads.

Optimising an axis nobody named

Ask which resource is actually scarce here: wall-clock latency, throughput, memory footprint, cost per request, or engineering time. Shaving a constant factor off an in-memory step is wasted effort when the same function makes a blocking remote call inside the loop.

Questions & practice

Size dock doors from overlapping appointment windows

Medium · sweep line · intervals · time zones

For one node and one local calendar day you have up to 200,000 planned dock appointments from shipment_leg: leg_id, planned_arrive_at, planned_depart_at (both TIMESTAMPTZ), plus the node's IANA zone. A door is occupied over [arrive, depart). Return the minimum number of doors that lets every appointment start on time, and the maximal interval over which that peak is sustained. Target O(n log n). Say how you derive the day's boundaries and how you break ties between an arrival and a departure at the same instant.

Approach

  1. Name the result before computing it: with one interchangeable door type, the minimum door count equals the maximum number of simultaneously occupied intervals. That equality is not a heuristic — interval graphs are perfect, so their chromatic number equals their clique number, and greedy left-to-right assignment achieves it. Say this, because it is what licenses solving a scheduling question with a counter.
  2. Emit 2n endpoints, (t, +1) at each arrival and (t, -1) at each departure, sort by t, and at equal t order the -1 before the +1. Half-open occupancy means a trailer leaving at 10:00 frees the door for one arriving at 10:00; ordering arrivals first inflates the answer by exactly the number of back-to-back handoffs, which on a well-packed schedule is most of them.
  3. Scan once with a running counter, tracking the maximum and the endpoint index where it was first reached. The peak is attained on a half-open interval between two consecutive endpoints, [t_i, t_{i+1}), not at an instant — report it that way or the operations team cannot act on it. Extend the interval while the counter stays at the maximum.
  4. Complexity: O(n log n) dominated by the sort, O(n) space. If rows already arrive ordered by planned_arrive_at from an index, use the min-heap variant instead — push each departure, pop all departures at or before the current arrival, and the heap size is the current occupancy. Same time bound, O(peak) space rather than O(n), which matters when peak occupancy is 40 and n is 200,000.
  5. Derive the day's boundaries from the zone, not from arithmetic on instants. Convert local midnight and the next local midnight to instants in the node's IANA zone; on a transition day that span is 23 or 25 hours, so adding 86,400 seconds silently drops or duplicates an hour of appointments. Decide explicitly what a zero-length dwell means — under half-open semantics it occupies nothing — and state it.

Worked solution (25 min)

  1. Write the endpoint tuple with the tie-break baked into the sort key — for example (t, kind) where departure sorts as 0 and arrival as 1 — rather than handling ties in the scan body.
  2. Implement the scan, returning both the maximum and the [t_i, t_{i+1}) span at which it holds.
  3. Build a fixture of six appointments where three are strictly back-to-back at the same minute, and hand-compute the answer before running.
  4. Add an appointment spanning the local DST transition and derive the day window from the zone, asserting the appointment count inside the window.
  5. Re-run with the input shuffled to confirm the result depends only on the set of intervals.

Expected result: The back-to-back fixture needs one door, not three; the reported peak is a half-open interval between two endpoints; on the transition day the local window is 23 or 25 hours and no appointment is dropped or double-counted.

Verify it yourself

  • Flipping the tie-break to arrivals-first changes the answer on the back-to-back fixture — if it does not, the fixture is not actually testing the tie.
  • The maximum equals the largest number of intervals containing any single probe instant, checked by brute force on a small fixture.
  • The result is identical under any permutation of the input.

Follow-up

  • Doors are typed: some break pallets, some are parcel-only. Does max-overlap still equal the door count?
  • Appointments have a tolerance — a trailer may start up to 20 minutes late without penalty. How does that change the objective, and is it still solvable by a sweep?
  • Half the appointments are actuals and half are plans. Which do you sweep for tomorrow's staffing, and which for last week's utilisation report?

Find the hottest write keys under a fixed memory budget

Medium · top-k · heap · streaming sketches

One day of writes to inventory_movement is 200 million rows; each carries (item_id, node_id). Distinct pairs reach 60 million. Return the 200 pairs with the most writes so they can be moved behind a single writer before the next peak. First give the exact answer assuming you may hold 60 million counters, then give an answer under a 200 MB budget, and state the error guarantee that second answer carries. Both must be one pass over the day's rows unless you argue for a second.

Approach

  1. Exact version: one hash pass to build (item_id, node_id) -> int64, then a single scan of the map maintaining a min-heap of size K = 200, pushing and popping when the incoming count exceeds the heap root. O(n + d log K) with n = 2e8 and d = 6e7, versus O(d log d) if you sort all 60 million entries — a factor of roughly log d / log K, about 3.4x, for no benefit. The heap must be a min-heap: the root is the weakest survivor, so eviction is O(log K).
  2. Cost the map honestly. 60 million entries at a 16-byte key, an int64 count and per-entry overhead is 2-3 GB in most runtimes, which is the whole reason for the second version rather than a theoretical aside.
  3. Bounded version: Misra-Gries with m counters. Keep at most m keys with counts; on a key already present increment it; on a new key when fewer than m are held, insert at 1; otherwise decrement every held counter and drop those reaching zero. With m = 1,000,000 at roughly 40 bytes per counter, that is about 40 MB, comfortably inside 200 MB alongside the heap.
  4. State the guarantee as a number, not as 'approximate'. Misra-Gries returns an estimate satisfying f - n/(m+1) <= f_hat <= f, so with n = 2e8 and m = 1e6 every count is an undercount by at most 200. If the 200th hottest pair has hundreds of thousands of writes, a 200-count error cannot reorder the top of the list — and if the counts near rank 200 are within 200 of each other, the ranking there was never meaningful for the decision being made. The sketch also merges: run one per shard, combine, prune back to m, and the bound holds with n the combined length.
  5. If exactness is genuinely required, argue for two passes rather than more memory: the Misra-Gries pass yields a candidate superset of size m, and a second pass counts only those candidates exactly in O(m) memory. That is the standard way to get an exact top-K under a memory bound, and the cost is one extra read of the day's rows.

Follow-up

  • The decision downstream is which keys get a single writer. Does a rank error at position 190 change that decision, and what does the answer tell you about how much accuracy to buy?
  • Write volume on one key is bursty — a promotion concentrates it into twenty minutes. Does a daily top-K find it, and what window would?
  • How do you keep the sketch across a worker restart without replaying the day?

Parse partner quantity and date segments without floating point

Easy · parsing · unit-of-measure · time zones

A partner sends a delimited flat file whose segments look like QTY+17:4.500:CS and DTM+232:20260317. Quantities carry up to three decimal places in a named unit; each item has an integer pack factor to eaches that is versioned by effective date. Produce observation_event rows with an integer quantity in eaches, the unit recorded, and occurred_at resolved, rejecting any line whose conversion is not exact. Files reach 4 GB. Single pass, no floating point. State your bounds and what you do with the zoneless date.

Approach

  1. Scan with indices, not a regex. Find the segment terminator, then split the payload on : by scanning forward and slicing — O(bytes) with no backtracking and no per-line allocation beyond the output row. A regex with nested quantifiers over 4 GB is where this job silently becomes CPU-bound.
  2. Parse the decimal as an integer mantissa plus a scale. 4.500 becomes (mantissa = 4500, scale = 3): accumulate digits into an int64, count digits after the point, and never call the runtime's float parser. Reject a mantissa that would overflow int64 when multiplied by the pack factor, checking before the multiply rather than after.
  3. Convert by integer arithmetic and demand exactness: numerator = mantissa * factor; emit numerator / 10^scale only when numerator % 10^scale == 0, otherwise reject the line with a reason code. Worked: 4.500 CS at 12 eaches per case gives 54000/1000 = 54, exact. 0.125 PL at 36 eaches per pallet gives 4500/1000 = 4.5, which is half a sealed unit and must fail at the boundary, not round three services away.
  4. Select the pack factor by the file's effective date, not by the current item master. Factors are properties of a point in time — a supplier moving a case from 12 to 10 must not reinterpret last quarter's file — so the lookup is a range query on the factor's validity interval.
  5. 20260317 carries no zone and no time. Attach the IANA zone of the node the segment refers to, captured at ingestion, because it cannot be recovered downstream; then pick and document one convention for a date-only field (start of local day is the usual choice) and store the raw local string beside the resolved instant so the assumption stays visible. Complexity: O(total bytes) time, O(longest line) working space, one pass.

Follow-up

  • The same file is re-sent in full tomorrow with three lines changed. What makes the second ingest a no-op for the unchanged lines?
  • A line names a unit that is not in the ladder for that item. Reject the line, reject the file, or quarantine it — and who finds out?
  • The partner starts sending an offset like +0100 on some dates. Does that change your storage decision for a future appointment as well as a past event?

Stop two allocators promising the same unit of stock

Medium · isolation levels · lock contention · compare-and-set

inventory_position is keyed (item_id, node_id, lot_id, state) and holds qty BIGINT, which is on-hand stock and is moved only by the movement ledger; available_qty BIGINT, maintained on the same row; and version. The declared invariant is available_qty = qty - SUM(allocation.qty) over that key in status held or committed, enforced by CHECK (available_qty >= 0) and CHECK (available_qty <= qty). allocation holds allocation_id, order_line_id, item_id, node_id, lot_id, qty, tier, status (held, committed, consumed, released, expired) and expires_at. Twelve allocator threads try to claim the last eight units of one item at one node inside the same 50 ms. Give the exact statements that let at most eight units be claimed, then say what changes if identical code runs under PostgreSQL REPEATABLE READ versus InnoDB REPEATABLE READ, and what a retried allocate for the same order line must do.

Approach

  1. Collapse the check and the write into one statement whose WHERE clause is the business rule: UPDATE inventory_position SET available_qty = available_qty - :q, version = version + 1 WHERE item_id = :i AND node_id = :n AND lot_id = :l AND state = 'on_hand' AND available_qty >= :q. qty is deliberately untouched: a hold is a promise, not a physical move, so only a movement may change on-hand and the ledger-equals-position invariant survives. Rowcount zero is a refusal, not an error, and the allocation row is inserted in the same transaction only when the rowcount is one.
  2. If the deployment carries no available_qty column and availability must be derived, the alternative is SELECT ... FOR UPDATE on the position row followed by the SUM over held and committed allocations inside the same transaction. That is correct but holds the row lock for two statements instead of one, which matters on a key taking hundreds of writes a second.
  3. Isolation is not a substitute for the atomic step, and it changes the error contract. Under PostgreSQL READ COMMITTED the losing UPDATE blocks, then re-evaluates its predicate against the newly committed row version and returns rowcount 0, which the handler already handles. Under PostgreSQL REPEATABLE READ the same statement instead raises SQLSTATE 40001 and the whole transaction must be retried, so a handler that only tests rowcount fails closed on an exception it never expected. InnoDB REPEATABLE READ does neither: a locking read or UPDATE performs a current read of the latest committed row, so the loser blocks and then sees rowcount 0, with a deadlock rather than a serialization failure as its exceptional case.
  4. Make the retry a no-op with a partial unique index: UNIQUE (order_line_id, node_id, lot_id) WHERE status IN ('held','committed'). Then a retried allocate conflicts instead of double-claiming, and the handler reads the existing allocation and returns it.
  5. Never source the availability number from a replica or a cache for this path. Replication lag is largest during the write burst that made the key hot, so the guarantee fails precisely when it is load-bearing.
  6. For the hottest item-node pairs, shed load without overselling: route allocations for that key through a single writer with a bounded queue and refuse with a conservative not-available when the queue is full. Splitting the position into N sub-rows cuts contention N-fold but strands the remainder across sub-rows and produces false out-of-stock when the last units are spread thin.

Worked solution (30 min)

  1. Write the conditional UPDATE, the rowcount branch and the allocation INSERT, plus the partial unique index DDL and the two CHECK constraints the invariant needs.
  2. Set a position to qty = 8 and available_qty = 8, launch 12 threads each claiming 1, and assert exactly 8 succeed, available_qty lands at 0 and qty is still 8.
  3. Re-run the same race under PostgreSQL REPEATABLE READ and record how many transactions fail with 40001 rather than returning rowcount 0.
  4. Send one allocate twice for the same order line and node, and assert one allocation row and the same allocation_id returned both times.
  5. Remove the available_qty >= :q predicate and confirm CHECK (available_qty >= 0) aborts the losing transactions, proving the test exercises the race rather than serialising by luck.

Expected result: Exactly 8 allocation rows in status held, available_qty = 0 with qty unchanged at 8, 4 refusals returned as a clean not-available response, and a duplicate allocate that returns the existing allocation without creating a second row.

Verify it yourself

  • Repeat the 12-way race 50 times and require 8 successes on every run, not on average.
  • Sample mid-run and require available_qty = qty - SUM(allocation.qty) over held and committed at every sample, with that sum never exceeding 8.
  • Confirm the REPEATABLE READ run produces 40001 errors and that the handler retries rather than surfacing them.
  • Drop the partial unique index and confirm the duplicate allocate now creates two rows.

Follow-up

  • One order line needs two units from two different nodes, atomically. What do you do when those positions live in different shards and no distributed transaction is available?
  • Soft-tier allocations expire. Show the sweeper statement, and say what stops it from releasing an allocation that is being committed at that instant.
  • Under sustained contention your CAS retries. What is the retry budget, and what does the caller see when it is exhausted?

Model the movement ledger so a retried write is a no-op

Easy · append-only ledger · idempotency · audit history

inventory_movement is append-only: movement_id, item_id, node_id, lot_id, state, delta_qty (signed BIGINT), uom_code, reason_code, ref_type, ref_id, idempotency_key, occurred_at, recorded_at, reversed_by. inventory_position is keyed (item_id, node_id, lot_id, state) with qty, version, last_movement_id. Give the DDL constraints and the exact statements a pick confirmation runs so that a client which times out after the commit and retries changes nothing. Then say how you correct a pick recorded against the wrong lot two days ago, and why the position carries CHECK (qty >= 0) rather than clamping at zero.

Approach

  1. Put the guarantee in the schema, not the handler: UNIQUE (idempotency_key) on inventory_movement. A SELECT-then-INSERT cannot work, because two concurrent retries both read nothing before either commits and both then insert.
  2. Claim with INSERT ... ON CONFLICT (idempotency_key) DO NOTHING RETURNING movement_id, and branch on rowcount. Zero rows means the first attempt already committed, so the handler returns success without touching the position. This branch is the whole point: applying the position update on the conflict path is what makes the ledger and the shelf disagree by one pick forever.
  3. Keep the insert and the position update in one transaction: UPDATE inventory_position SET qty = qty + :delta, version = version + 1, last_movement_id = :id WHERE item_id = :i AND node_id = :n AND lot_id = :l AND state = :s AND version = :v, then test the rowcount and retry the transaction on zero.
  4. Use lot_id BIGINT NOT NULL DEFAULT 0 rather than NULL. In a default unique index NULLs are distinct from each other, so a NULL lot would let two position rows exist for the same untracked item and split the running sum in half.
  5. Correct the mis-keyed pick with two compensating movements, never an UPDATE or a deleted_at flag: one reversal restoring the wrong lot, one fresh pick against the right lot, both with reason_code = 'reversal' or 'pick' and ref pointing at the original movement_id. The original row keeps its values; reversed_by is a back-pointer written in the same transaction so a reader does not need a reverse scan to see that the row was undone.
  6. CHECK (qty >= 0) aborts the transaction and surfaces a real defect. GREATEST(qty + :delta, 0) writes a plausible number instead, so the missing receipt that caused the negative is never investigated and the first symptom is a picker at an empty location.

Follow-up

  • Two retries of the same pick arrive concurrently on different application instances. Walk through both transactions statement by statement and say which one commits a position change.
  • occurred_at from a handheld is five minutes ahead of recorded_at because the device clock drifted. Which of your queries break, and which column should each of them actually be using?
  • How long do you keep idempotency_key unique for, given the table grows by tens of millions of rows a month?

Describe your understanding and application of OOPs concepts.

Medium

Approach

  1. Clarify what is being asked and what a complete answer contains.
  2. Say what you would check first and why it is the highest-information step.
  3. State your assumptions explicitly before working the problem.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

How comfortable are you with network engineering concepts such as SD-Wan, routing, trunks, and ACLs?

Medium

Approach

  1. Say what you would check first and why it is the highest-information step.
  2. Clarify what is being asked and what a complete answer contains.
  3. Work from the requirement backwards to the design.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

How do you approach project requirements when you are asked to use specific, proprietary enterprise tools?

Medium

Approach

  1. Work from the requirement backwards to the design.
  2. State your assumptions explicitly before working the problem.
  3. Clarify what is being asked and what a complete answer contains.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

Redesign a tender endpoint whose retry books two trucks

Medium · idempotency · irreversible effects · retries

POST /shipments/{shipment_id}/tender asks the Carrier Integration Gateway to tender a load and returns the partner booking reference. The HTTP client has a 10 s timeout and a three-attempt retry decorator; partner p99 is 4 s with occasional 40 s stalls. shipment_leg already carries booking_idempotency_key TEXT NOT NULL UNIQUE. Last week two trailers arrived at one dock for one load. Redesign the contract so the physical effect happens at most once: give the resource shape, the status model including the ambiguous outcome, who performs retries, and what the caller does on a timeout.

Approach

  1. Diagnose the shape, not the bug. A POST to an action with server-minted identity gives the caller no way to name the effect it wants, so every retry is a new request by construction. Replace it with a caller-named sub-resource - PUT /legs/{leg_id}/booking/{booking_idempotency_key} - where the key is in the URI and a second identical PUT converges on the same resource instead of creating another.
  2. Persist before you call. Write the booking row with status requested and the key committed locally before the gateway touches the partner. A crash between that commit and the call is recoverable by a sweeper; a call with no local record is not recoverable at all, because you cannot ask about a key you never stored.
  3. Make unknown a first-class status: requested, unknown, confirmed, rejected. A timeout or a connection reset moves the row to unknown and the API answers 202 with the resource and a Location, never a 5xx - a 5xx is an invitation to the decorator to re-issue an irreversible act.
  4. Move retries off the request path entirely. Delete the decorator for this call and let a sweeper read rows in requested or unknown and resolve each by querying the partner for that key, or by their list-by-reference endpoint when they have no key concept. The sweeper is now the component that must be correct, and it is testable in a way an in-process retry that dies with the pod is not.
  5. Keep the database transaction off the network call. Commit the state row together with an outbox entry and let a relay make the partner call; a transaction held open across a 40 s stall parks a pooled connection, and at tender peak the pool is what fails first, taking unrelated traffic with it.
  6. State the caller's rule in one line: a timeout or 5xx from this endpoint means unknown, never not booked, and the only permitted follow-up is to read the booking resource.

Worked solution (35 min)

  1. Draw the current timeline that produces two trailers: request sent, partner commits, response lost at 10 s, decorator re-issues, partner commits again.
  2. Rewrite the endpoint as a key-in-URI PUT and write the request and response bodies for first call, duplicate call and ambiguous outcome.
  3. Write the status table with allowed transitions and mark every transition that touches the partner.
  4. Write the sweeper's query and its resolution rule for a row stuck in unknown, including the decision it makes when the partner reports nothing for that key.
  5. Re-examine the connection lifecycle and confirm no transaction is open while the partner call is in flight.

Expected result: A PUT whose URI carries the caller-owned key, a persisted requested state written before any partner contact, an explicit unknown state returned as 202, retries owned by a sweeper reading persisted rows, and no database transaction spanning the external call.

Verify it yourself

  • Replaying the same PUT ten times yields one booking reference and one partner-side booking.
  • Killing the process immediately after the partner call leaves a row a sweeper can resolve without guessing.
  • No code path both holds a row lock and awaits the gateway.

Follow-up

  • The partner supports no idempotency key and no query-by-reference. What do you do, and what residual risk do you accept?
  • Two sweeper instances pick up the same unknown row at once. What stops the resolution itself from producing a second booking?
  • How do you cancel a booking whose outcome is still unknown, and what does the leg's status machine look like during that?

One partner's leg statuses freeze while every other feed advances

Easy · poison message · head-of-line blocking · event ingest

Since 06:40 UTC no leg fed by one partner has changed status; every other feed is current. Ingest is healthy and CPU is flat, and observation_event rows from that source are still landing with current received_at and processing_status 'pending'. Consumer lag is growing on exactly one stream partition, whose head row has over 900 attempts and a byte-identical last_error on each one. Give the ordered diagnosis and the change that stops one unapplicable observation from stalling a partition again.

Approach

  1. Separate the partner not sending from us not applying, because both look identical to a customer. Rows landing with current received_at and processing_status 'pending' prove ingestion is fine and the fold is stuck; if received_at had gone stale instead, this would be a partner or transport incident and nothing in the consumer would be at fault.
  2. Read the blocked partition at its head, ordered by offset, and classify the error rather than counting it. An identical error string across 900 attempts is deterministic and implicates the payload; a varying set of connection resets, timeouts and 5xx implicates the downstream store. Only the deterministic case is a poison message.
  3. Reproduce outside the consumer by running the fold over that one event. The usual cause here is an observed_status the partner added that has no rank in the progress lattice, or a subject_id whose leg does not exist yet, so the fold throws before it can write.
  4. Decide the exit against the ordering contract, which is per subject and not per partition. Many legs hash into one partition, so blocking on one of them is a bug and not a guarantee. Park the row as processing_status 'rejected_malformed' with its offset in a dead-letter view, advance the offset, and block only the affected subject if the event is well-formed but not yet applicable.
  5. Make the retry policy classify instead of counting. Deserialisation failures, unknown enum values and constraint violations are terminal and must never be retried in place; IO errors and timeouts are retryable with bounded attempts and backoff. Alert on oldest-pending age per partition, which turns positive within seconds, rather than on lag, which drifts up slowly and looks like normal backlog.

Follow-up

  • The parked event turns out to be a real delivered scan. How do you replay it after the code fix without double-applying, and what makes that safe?
  • What should the customer-facing status show for legs that were queued behind it?
  • The partition key is currently the leg id. What breaks if you key by partner instead, and what if you key randomly?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

  • Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
  • Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
  • Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

  • Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
  • Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
  • Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

  • Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
  • Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
  • Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

  • For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
  • Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
  • Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
  • Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

  • Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
  • Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
  • Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
  • Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

  • Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
  • Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
  • Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

  • Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
  • Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
  • Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

The bug you decided not to fix

Engineers over-index on what they repaired. A stronger answer covers something you knowingly left broken: the alert you tuned down, the data inconsistency you documented instead of chasing, the cleanup you deferred past two quarters. Give the reasoning and the condition that would have reopened it, so it reads as a decision and not as neglect.

What do you consider your greatest strength and your greatest weakness?

Medium · behavioural and engineering judgement

Approach

  1. Close with what you would do differently, concretely.
  2. State the situation in two sentences and spend the rest on the reasoning.
  3. Name the disagreement and how you resolved it with evidence.

Follow-up

  • How did you know your change caused the improvement?
  • What would you do differently if you ran that again?

What do you know about Americold Logistics and why do you want to work here?

Medium · behavioural and engineering judgement

Approach

  1. State the situation in two sentences and spend the rest on the reasoning.
  2. Give the blast radius: what could have broken, and what you measured.
  3. Close with what you would do differently, concretely.

Follow-up

  • How did you know your change caused the improvement?
  • What would you do differently if you ran that again?

Tell me about a time you had trouble with a teammate and how you addressed it.

Medium · behavioural and engineering judgement

Approach

  1. State the situation in two sentences and spend the rest on the reasoning.
  2. Close with what you would do differently, concretely.
  3. Give the blast radius: what could have broken, and what you measured.

Follow-up

  • How did you know your change caused the improvement?
  • What would you do differently if you ran that again?