cswork / libraryOpen practice workspace

Interview guides · Arrow International

Arrow International Software Engineer Interview Questions & Guide 2026

Software engineer interview prep for Arrow International: original coding, system design and debugging practice, each with a worked solution.

The 60-second brief

As a Software Engineer at Arrow International, you play a pivotal role in designing, developing, and deploying state-of-the-art gaming solutions that power venues around the world. You will work within dynamic engineering teams—ranging from electronic gaming development to core infrastructure—building engaging software experiences, reliable deployment pipelines, and robust backend systems. Your contributions affect the company's ability to innovate within the charitable gaming and entertainment sectors, keeping its digital and physical product ecosystems running at scale.

The title spans product, platform and infrastructure work, and which of those the seat actually is decides whether design or algorithms carries more weight in your preparation. The posting rarely settles it; what the team is on call for usually does.

Focus areas

  • Make cross-service trades atomic through escrow sagas
  • Budget per-tick simulation, serialization and bandwidth
  • Derive idempotency keys from server-authoritative state

Overview

This position demands a strong grasp of computer science fundamentals, a passion for clean code, and the ability to collaborate across multidisciplinary teams including game designers, quality assurance, and product management. Whether you are building 3D interactive experiences using frameworks like Unity, optimizing backend deployment scripts, or scaling system integrations, you will be solving complex, real-world technical problems. The scope of the work requires both high-level architectural thinking and meticulous attention to implementation details.

You can expect an environment that values continuous learning, technical depth, and creative problem-solving. While the projects are fast-paced and carry high business visibility, Arrow International fosters a collaborative culture where engineers are empowered to take ownership of their work. Success in this role means combining rigorous technical execution with an enthusiasm for building products that delight users and drive company growth.

The Arrow International interview process

Arrow International candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Outreach

An unlabelled round is first an information problem, and the cheapest information is free. Whoever schedules it can usually tell you how long it runs, who will be in the room and what they work on, whether you will be writing code and in what environment, and whether anything is being sent beforehand. Ask in writing so the answer is on record, then prepare for the two or three formats those answers still leave open instead of betting on one. What separates a strong candidate is not guessing right; it is having an opening that works whichever one it turns out to be.

What to demonstrate

  • Whether you can start work from an ambiguous brief, since tolerating a vague scope without stalling is the same thing the job asks for
  • Whether the questions you asked beforehand were ones that change your preparation, such as duration, medium and who is joining, rather than ones whose answers you could not have acted on
  • Whether you adapt when the round turns out to be something other than what you were told, instead of spending the first ten minutes visibly recalibrating

How to prepare

  • Send one short scheduling message asking four things: how long, who is joining and what they work on, whether you will be writing code and where, and whether to prepare anything in advance. Treat a vague reply as real information, since it means the round is loosely structured and you will be shaping it yourself.
  • Write one opening that works in any of the formats still open: restate in your own words what you have been asked to do, then ask which of two directions is more useful to them. Say it aloud until it stops sounding recited.
  • Set up for the two most likely formats before the call starts, with a blank editor in the language you would choose and a shared document you can type into, so a format surprise costs you nothing in the first minutes

Technical Screenings

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

  • Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
  • Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
  • Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
  • Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

  • For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
  • For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
  • Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Comprehensive Discussions

You cannot drill a format you do not know, so put the preparation into material that travels. Three pieces of your own work, each rehearsed until you can take a follow-up you did not anticipate, will carry a conversation or a code walkthrough equally well. Specificity is what separates that from filler. A number needs its definition before it means anything: a p99 is over some window and measured at some hop, and a server-side figure excludes the queueing and network time a client would see. The number you cannot qualify is the one to leave out.

What to demonstrate

  • Whether your examples carry detail only someone who did the work would hold, such as what the binding constraint actually was, which alternative you rejected and why it was worse, and what you measured on each side of the change
  • Whether a number survives one follow-up, meaning you can say what it was measured over and whether it moved because of your change or merely alongside it
  • Whether a failure is described with the specific change that followed it, rather than a lesson stated in general terms
  • Whether your part in a team effort is stated accurately, including what other people did

How to prepare

  • Write a page on each of three projects covering the constraint, the option you rejected, the measurement before and after, and what went wrong. Cut any line you cannot take a follow-up on, since you are writing the parts you will be pressed on rather than a summary.
  • Recover the real figures while you still have access: request volume, data size, latency with its percentile and window, team size, timeline. Note where each came from, whether a dashboard, a design document or memory, and mark the estimates so you can say which they are out loud.
  • Take your weakest project story to someone who works in a different area and have them ask why four times in succession. The point where you run out of answer is the part to go and re-read before the round.

Facility Tours

When a round has no standard shape, it is often there because something is still open: an area no earlier conversation reached, a round where the signal came out mixed, or a decision someone is not ready to make alone. Work out which by going back over what each earlier round actually covered rather than how it felt, and arrive able to give evidence on that point without being asked twice. Weak answers replay the loop's earlier material at the same depth. Strong ones go a level deeper and stay consistent with what you already said.

What to demonstrate

  • Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
  • Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
  • Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
  • Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened

How to prepare

  • Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
  • Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
  • Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure

Where candidates lose points

Read-modify-write on a balance or a stack quantity - SELECT the current value, check it in application code, then UPDATE to the computed new value.

The gap between the read and the write is a lost-update window of a few milliseconds. It survives every code review and every load test run at baseline concurrency, then fires reliably when a seasonal event puts two requests on the same wallet inside that window, or when one player has two clients open. The fix is one statement - UPDATE ... SET qty = qty - :n WHERE qty >= :n - plus an affected-row check; the trap is that the broken version returns the correct answer in every single-threaded test you will write.

Representing prices and balances as floating point, or storing a display currency amount instead of integer minor units plus a currency code.

Binary floating point cannot represent most decimal fractions exactly, so repeated credits and debits accumulate drift that only surfaces when reconciliation compares the ledger sum against the cached balance and finds them a fraction apart. Regional price books make it worse: the same SKU has different minor-unit scales across currencies, and a currency with no minor unit breaks any code that assumes two decimal places.

Choosing a schema before the access patterns are known

Write the queries first, with their filters, sort orders, cardinalities and which ones sit on the latency-critical path, then design tables and indexes to serve them. An index nothing queries still costs write throughput and storage, and a hot query with no supporting index becomes a full scan that only hurts once the table is big.

Retrying a write that is not safe to repeat

A timeout tells you nothing about whether the server applied the write, so a blind retry of a create or a charge can duplicate it. Either make the operation idempotent, with a caller-supplied key the server deduplicates on or a conditional update, or do not retry it; and use exponential backoff with jitter so the retries of many clients do not synchronise into a second outage.

Questions & practice

Budget per-tick visibility queries under a 16.7 ms tick

Hard · spatial index · tick budget · bandwidth

A match instance simulates 100 entities at 60 Hz in a 1000 m by 1000 m arena. Every tick, each entity needs the set of others within 60 m so the snapshot serialiser can skip the rest. Entities move at up to 8 m/s. The whole tick, meaning inputs, physics, visibility and serialisation, must fit in 16.7 ms, and one host runs many instances. Give the naive cost, a structure that beats it, its worst case on this exact workload, and the downstream bandwidth per client at 30 Hz with 40-byte per-entity deltas.

Approach

  1. Naive all-pairs is C(100,2) = 4,950 distance tests per tick, 297,000 per second per instance, and 5.9 x 10^8 per second across a 2,000-instance fleet at 200k concurrent players. The argument is the scaling, not the absolute number: at O(P^2) per tick, doubling the roster quadruples the work.
  2. Uniform grid with cell edge equal to the query radius, 60 m: ceil(1000/60) = 17 cells per side, 289 cells, mean occupancy 100/289 = 0.35. Cell edge at least the radius means the 3x3 neighbourhood provably covers every candidate, since a point within 60 m differs by at most one cell index on each axis. That is about 3.1 candidates per query and roughly 310 distance tests per tick, 16 times fewer. Rebuild clear-and-reinsert each tick at O(P); at P = 100 and 0.13 m of movement per tick that is cheaper than incremental updates and immune to stale-cell bugs.
  3. State the worst case on this workload rather than quoting the average as the budget. All 100 entities inside one cell, which is exactly what a contested objective looks like, makes the 9-cell scan return all 100 for every query: 10,000 tests, about twice the naive triangular count, because the grid neither halves symmetric pairs nor prunes anything.
  4. Choose by density, not by fashion. A uniform grid wins on roughly even spread with one fixed radius; a k-d tree or BVH costs O(P log P) to build and only pays off at much larger P or with widely varying query radii. Halve the work in either case by visiting each pair once, scanning only cells at or after the current cell in traversal order, and by comparing squared distances so no test carries a sqrt.
  5. Bandwidth: 30 visible entities at 40 B is 1,200 B per snapshot; at 30 Hz that is 36 KB/s, or 288 kbit/s downstream per client, plus about 28 B of UDP and IPv4 header per packet for another 840 B/s. At 60 Hz it doubles to 576 kbit/s, which is the concrete argument for running the snapshot rate below the simulation rate.

Worked solution (40 min)

  1. Compute the naive pair count, C(100,2) = 4,950, and multiply by 60 Hz for the per-instance rate.
  2. Size the grid at cell edge 60 m, compute 17 by 17 cells, mean occupancy 0.35, and expected candidates per query as occupancy times 9.
  3. Construct the clustered case with all 100 entities in one cell and recompute the test count, then compare it against the naive figure rather than against the uniform figure.
  4. Convert 30 visible entities at 40 B into bytes per snapshot, then into kbit/s at both 30 Hz and 60 Hz.
  5. Implement both the grid query and all-pairs on the same positions and diff the resulting visibility sets.

Expected result: 4,950 naive pair tests per tick and 297,000 per second per instance; about 310 grid tests per tick under uniform spread, a 16x reduction; 10,000 tests per tick when all 100 share one cell, roughly twice the naive cost; 288 kbit/s per client at 30 Hz and 576 kbit/s at 60 Hz.

Verify it yourself

  • The grid and all-pairs visibility sets must be identical, element for element. A mismatch means the cell edge is smaller than the radius, or the neighbourhood scan is 3x3 where the cell size demands wider.
  • Measured candidates per query should sit near mean occupancy times 9. If they run far above it, the distribution is not uniform and the average-case number is not your budget.
  • Per-instance egress, 100 clients at 36 KB/s, is 3.6 MB/s or about 29 Mbit/s, which must be checked against the host NIC once you multiply by instances per host.

Follow-up

  • Entities converge on one objective and the grid degrades to its worst case. What do you change first: cell size, a cap on the visible set, or the snapshot rate, and what does each cost the player?
  • How does the visibility set interact with lag compensation, where the server rewinds to the shooter's view time to resolve a hit?

Trace a duplication ring through time-respecting trade edges

Medium · temporal graph · reachability · fraud detection

An investigation gives you a seed set of accounts known to hold duplicated ItemInstances as of time t0, plus the day's completed trades as edges (from_account, to_account, instance_id, traded_at): 40 million edges over 20 million accounts. Return every account that could have received tainted supply, meaning reachable from a seed along a path whose timestamps are non-decreasing and all at or after t0. A trade that happened before its sender was tainted carries nothing. Give the complexity, the memory bound, and say why an ordinary BFS over the edge set is wrong.

Approach

  1. Recognise this as earliest-arrival temporal reachability, not static reachability. Sort the edges by traded_at ascending, O(E log E) over 40 million edges, sorting on a packed (timestamp, u, v) key rather than dragging the full row through the sort.
  2. Keep taint_time[account], initialised to t0 for seeds and to positive infinity elsewhere. With dense account ids a flat int64 array over 20 million accounts is 160 MB; with sparse ids use an open-addressed map sized to the seed's expected reach rather than to the account space.
  3. One forward pass in time order: for edge (u, v, t), if taint_time[u] <= t then taint_time[v] = min(taint_time[v], t). One pass suffices because every edge that could have tainted u carries a timestamp at most t and has therefore already been processed. O(E) after the sort, O(V) space.
  4. Fix the equal-timestamp rule explicitly and defend it. Letting same-timestamp edges chain lets a single millisecond propagate arbitrarily far through the graph; the defensible default applies a whole equal-timestamp batch against the taint set as it stood before the batch.
  5. Explain why static BFS is a strict over-approximation: it accepts paths that run backwards in time, which in a transfer graph asserts that an account received supply it could not yet have held. On a graph with heavy repeat trading between the same accounts, that inflates the rollback set well past what the evidence supports, and a rollback list is something you have to defend to real players.

Follow-up

  • The rollback has to be defensible to the player whose items you claw back. What do you record per tainted account so you can produce the path that implicated them?
  • Trades keep arriving. Make it incremental, then say what happens when a late edge shows up out of timestamp order.
  • Propose two signals over these same edges that separate a duplication ring from a high-volume legitimate trader.

Find peak escrow lock concurrency with a sweep line

Medium · sweep line · intervals · capacity planning

You have 2 million market_listing rows from one day, each with created_at and expires_at, where the escrow lock is held over the half-open interval [created_at, expires_at). Timestamps are millisecond precision inside a 24-hour window. Report the peak number of simultaneously held escrow locks and every maximal interval over which that peak holds. Separately, the reclaim worker that releases expired locks handles 500 per second: report the first instant at which expiries per second exceed that rate. State time and space complexity for both answers.

Approach

  1. Emit 2N endpoint events, +1 at created_at and -1 at expires_at, then sort by timestamp with end events ordered before start events at an equal timestamp. The half-open interval means a lock ending at t does not overlap one starting at t. O(N log N) time, O(N) space.
  2. Sweep with a running counter, tracking the maximum and the [t_i, t_i+1) span in which it is held. A peak sustained across several consecutive events is one maximal interval, so coalesce adjacent spans that sit at the maximum rather than emitting one per event.
  3. For the reclaim question use a difference array instead: bucket the -1 events into 86,400 one-second buckets and take the first bucket exceeding 500. O(N + T) time with T = 86,400 buckets at 8 bytes, about 700 KB, and no sort at all.
  4. Name the trade-off rather than picking by habit. At millisecond resolution over a day, T = 86.4 million buckets is 690 MB as int64 and the sort wins. At one-second resolution the bucket array wins outright. The decision is T against N log N.
  5. If the listings stream in rather than being loadable at once, sorting is unavailable: keep a min-heap of active expiry times, pop everything at or before the current timestamp on each arrival, and accept that the reported peak is only correct over the prefix seen so far.

Follow-up

  • The sweeper falls behind and locks outlive expires_at. What does the seller see on their inventory page during that window, and what does the listing row have to say to make that view honest?
  • Same question but peak concurrency per item_def_id, where the top one percent of def ids carry most listings. Does your memory bound still hold, and which def ids actually need the per-key answer?

Two one-to-many joins that triple a grant report's totals

Medium · join fan-out · aggregation · polymorphic reference

Finance wants, per account for one day, the soft currency granted and the number of item stacks received. grant_receipt(grant_id UUID PK, account_id BIGINT, source_type TEXT, source_ref_id UUID, status TEXT, created_at TIMESTAMPTZ). currency_ledger(entry_id BIGSERIAL PK, wallet_id BIGINT, delta_minor BIGINT, source_type TEXT, source_ref_id UUID NULL) — one leg per currency per grant. item_instance(instance_id UUID PK, origin_grant_id UUID NULL, quantity INT) — one row per stack granted. An analyst joins all three on the grant id and reports totals roughly triple the ledger's own sum. Explain the arithmetic that produces that, then write the corrected query.

Approach

  1. Name the shape rather than the symptom: two independent one-to-many branches hang off the same parent, and joining both in one FROM multiplies them. A grant with two ledger legs and three item rows emits six rows, so SUM(delta_minor) counts every leg three times and COUNT(instance_id) counts every stack twice.
  2. Show why COUNT(DISTINCT instance_id) is only half a fix: it rescues the count, but nothing rescues the sum. Two legs can legitimately carry the same delta_minor, so SUM(DISTINCT delta_minor) would collapse genuine rows and be wrong in the opposite direction.
  3. Aggregate each branch to one row per key before joining — a CTE per branch grouped to grant_id or account_id, then join the two aggregates. Use LEFT JOIN with COALESCE so a grant that gave currency but no items still appears with a stack count of zero.
  4. Pin the discriminator: source_ref_id on currency_ledger is a polymorphic reference that points at matches, orders or listings depending on source_type, so the join is only valid with source_type fixed. No foreign key can enforce a polymorphic column — that is the price of the modelling choice, and the query has to pay it explicitly.
  5. Filter grant_receipt.status = 'applied', and decide out loud what happens to reversed grants — netted against the total or excluded — because leaving it unstated makes the number wrong either way.
  6. Validate against independent ground truth: the corrected currency total must equal SUM(delta_minor) computed over currency_ledger alone for the same day and reason codes.

Follow-up

  • One grant in the window is reversed after the report has been sent. Does the number change retroactively, and should it?
  • Rewrite the corrected version with LATERAL subqueries and say when you would prefer that to the CTE form.
  • The report now has to run per-hour over a 10^9-row ledger. What changes?

Design the unique constraint that stops a reward duplicating

Medium · idempotency · at-least-once · partial index · reconciliation

grant_receipt(grant_id UUID PK, account_id BIGINT, idempotency_key TEXT NOT NULL UNIQUE, source_type TEXT, source_ref_id UUID, payload_hash BYTEA, status TEXT CHECK (status IN ('pending','applied','reversed','rejected')), created_at TIMESTAMPTZ, applied_at TIMESTAMPTZ NULL). purchase_order(order_id UUID PK, platform TEXT, receipt_id TEXT, capture_status TEXT, fulfil_status TEXT, fulfil_grant_id UUID NULL, captured_at TIMESTAMPTZ NULL). Match rewards arrive over an at-least-once queue; store receipts can be re-presented by a reinstalled client. Specify how idempotency_key is derived for a match reward and for an order fulfilment, what the insert does on a key collision, and write the sweep that finds captured payments never fulfilled.

Approach

  1. Derive the key from server state only: for a match reward, 'match:' || match_id || ':' || account_id || ':' || slot_index, taken from the signed match_result row; for fulfilment, 'order:' || platform || ':' || receipt_id, taken from the validated receipt. Nothing the client sends enters the key, so a client that regenerates a request id on retry cannot mint a second reward.
  2. Explain what one global UNIQUE (idempotency_key) buys: the source is encoded in the string, so a single index covers every source type, and the database rather than application code is the thing enforcing at-most-once. A broker that redelivers on consumer failure is at-least-once by construction, so the sink has to dedupe.
  3. Handle the collision correctly. On 23505, read the existing row and compare payload_hash: equal means a genuine retry, so return the original grant_id and the original contents; different means a key reused with different contents and must be rejected, not quietly honoured with the old payload.
  4. Know the ON CONFLICT sharp edge: DO NOTHING RETURNING grant_id returns zero rows on conflict and does not hand back the existing row. Either follow it with a SELECT, or use a no-op DO UPDATE to force RETURNING at the cost of a row lock and a dead tuple on every retry.
  5. Write the sweep for the opposite failure — captured but never fulfilled — with a grace window so it does not race an in-flight fulfilment, and a partial index on (captured_at) WHERE capture_status='captured' AND fulfil_status='unfulfilled' so it costs the size of the matching set rather than the table.
  6. Say what the sweep does on a hit: re-drive the same grant with the same derived key, which is safe precisely because the key is idempotent and a genuine double-run collides instead of double-granting.

Worked solution (25 min)

  1. Write the key derivation for both source types as literal string expressions over named columns.
  2. Insert the same grant twice with identical payloads and confirm the second returns the original grant_id and applies nothing.
  3. Insert again with the same key and a different payload, and confirm it is rejected rather than returning the old grant.
  4. Write the captured-but-unfulfilled sweep with its grace window, create the matching partial index, and EXPLAIN it.
  5. Simulate a crash between the grant write and the queue acknowledgement, replay the message, and confirm nothing duplicates.

Expected result: Keys of the form 'match:'||match_id||':'||account_id||':'||slot_index and 'order:'||platform||':'||receipt_id; a collision path that compares payload_hash before returning the original; and SELECT order_id, account_id, sku_id, captured_at FROM purchase_order WHERE capture_status='captured' AND fulfil_status='unfulfilled' AND captured_at < now() - interval '5 minutes' ORDER BY captured_at LIMIT 500, backed by a partial index with that predicate.

Verify it yourself

  • Replaying one grant 100 times leaves exactly one grant_receipt row and one txn_id in currency_ledger.
  • The same-key, different-payload replay is rejected rather than returning the earlier grant.
  • EXPLAIN on the sweep shows an index scan on the partial index, not a sequential scan of the order table.
  • The sweep returns zero rows in steady state and a bounded set immediately after an induced fulfilment failure.

Follow-up

  • A reward is granted, then the match's integrity_flag flips to voided. How do you take it back, and what does the player see?
  • Two reward slots on one match happen to resolve to identical items. Does your key still separate them?
  • The queue redelivers a batch of 50,000 after a consumer OOM. What does your database do for the next minute?

Walk me through how you would architect a real-time interactive game loop using modern frameworks.

Medium · system design and architecture

Approach

  1. Name the read and write paths separately; they rarely have the same bottleneck.
  2. Name the failure you are designing for, then the recovery path.
  3. State the consistency you need, and where you are willing to be stale.

Follow-up

  • What breaks first when traffic grows ten times?
  • How does this behave when that dependency is down for an hour?

How would you design a scalable deployment pipeline for distributed software environments?

Medium · system design and architecture

Approach

  1. Fix the scope first: who calls this, how often, and what they do when it fails.
  2. Name the failure you are designing for, then the recovery path.
  3. Name the read and write paths separately; they rarely have the same bottleneck.

Follow-up

  • How does this behave when that dependency is down for an hour?
  • What breaks first when traffic grows ten times?

Given a specific performance bottleneck in a multi-threaded application, how would you diagnose and resolve it?

Medium · system design and architecture

Approach

  1. State the consistency you need, and where you are willing to be stale.
  2. Name the failure you are designing for, then the recovery path.
  3. Name the read and write paths separately; they rarely have the same bottleneck.

Follow-up

  • What would you drop to keep the system up under load?
  • How does this behave when that dependency is down for an hour?

How do you approach debugging and modifying unfamiliar, legacy codebases?

Medium

Approach

  1. Clarify what is being asked and what a complete answer contains.
  2. Work from the requirement backwards to the design.
  3. State your assumptions explicitly before working the problem.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Can you explain object-oriented programming principles and how you apply them in C#?

Medium

Approach

  1. Clarify what is being asked and what a complete answer contains.
  2. Work from the requirement backwards to the design.
  3. State your assumptions explicitly before working the problem.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Escrow saga for an item-for-currency trade

Medium · saga · escrow · compensation · cross-service

Ownership lives in the inventory service (item_instance.owner_account_id, escrow_id, row_version) and money in the ledger service (currency_ledger legs sharing one txn_id). There is no shared transaction. A completed trade must debit the buyer, move the stack, credit the seller net of a fee, and release the lock. Give the ordered saga with the durable state after each step, the compensation for each, what buyer and seller see in every intermediate state, and how a lock whose saga died is reclaimed without racing a saga that is still alive.

Approach

  1. Write the steps with the durable state each one leaves behind: reserve (listing.status='reserved'), debit buyer (ledger leg, idempotency_key = listing_id + ':debit'), transfer instance (owner change plus escrow_id set to NULL, guarded on escrow_id = :listing_id), credit seller net of fee (two more legs under the same txn_id, the fee leg targeting the sink wallet so the three legs sum to zero), settle (listing.status='sold', settle_txn_id recorded).
  2. Mark the crash point after each step and say what recovery does. Before the transfer, compensate: refund with idempotency_key = listing_id + ':refund', which is itself idempotent because a re-run collides on the unique index. After the transfer, only roll forward.
  3. Justify forward-recovery past the transfer rather than unwinding: reversing an ownership change writes a second transfer edge that a later duplication investigation cannot distinguish from a genuine trade, and the graph is the only evidence you will have.
  4. Reclaim dead sagas with a reaper over status='reserved' AND reserved_at < now() - interval '2 minutes', and fence it. Bump an escrow epoch on reclaim so a slow-but-alive saga's transfer, guarded on the old epoch, affects zero rows instead of completing after the item was returned.
  5. Name what each side sees: the seller's stack reads as locked from listing until sold or cancelled; the buyer is debited before the item lands, so the client shows 'purchasing' and not 'failed'; no state in the sequence may show the stack in both inventories.
  6. State what you gave up against two-phase commit: there is a window where money has moved and the item has not, and clients must render it. In exchange there is no coordinator that can block both services while holding locks across a network call.

Worked solution (35 min)

  1. List the five steps with the exact row state after each, then annotate every crash point with its recovery direction.
  2. Write both idempotency keys and show that re-running each compensation changes nothing.
  3. Write the guarded transfer statement and add the epoch check that makes reclaim safe.
  4. Enumerate the intermediate states and the client copy for each one.
  5. Write the reaper query with its index and pick the timeout, justifying it from the expected saga duration.

Expected result: A five-step saga with durable state per step, per-step idempotent compensation keyed off the listing id, forward-only recovery after the transfer, and a fenced expiry reaper whose epoch bump makes a late transfer affect zero rows.

Verify it yourself

  • Replaying every step twice leaves exactly one debit, one transfer, one credit and one fee leg.
  • The legs sharing a txn_id sum to zero once the fee leg is included.
  • A saga killed after the debit either completes or refunds, and never leaves the stack unlocked while the buyer stays debited.
  • A saga resumed after its reclaim cannot transfer the stack.

Follow-up

  • The ledger service is unavailable for ten minutes. Do you queue reservations, reject new listings, or keep browsing read-only?
  • Support asks you to prove a given instance was never duplicated. Which rows answer that, and what would you have had to keep?
  • How do you cap the number of simultaneous escrows one account can hold, and why does that limit matter?

Reward consumer lag climbs while one partition stops advancing

Medium · poison message · at-least-once delivery · idempotency

Match-reward grants are consumed from a partitioned log. Consumer group lag on partition 7 grows linearly for forty minutes while every other partition stays at zero. The consumer process exits and restarts roughly every 30 s, and each restart triggers a group rebalance that briefly stalls all partitions. After you redeploy, support reports a burst of duplicated rewards. grant_receipt declares idempotency_key TEXT NOT NULL UNIQUE. Give the ordered diagnostic checklist, the root cause of both symptoms, and the fix.

Approach

  1. Read the crash, not the lag: pull the consumer's stack trace at exit and the offset it was processing, then fetch the record at that offset and reproduce locally. One partition stalled behind a crash-looping consumer is head-of-line blocking on a single record, not a throughput shortfall.
  2. Confirm the shape before theorising. In a partition-ordered log a poison record stops that partition's committed offset advancing at all, so its lag rises at exactly the production rate while other partitions are untouched. If every partition had degraded together you would be looking at capacity or a downstream dependency instead.
  3. Account for the second-order damage: each crash-restart rejoins the consumer group and triggers a rebalance, which stalls partitions that have nothing wrong with them. That is why the blast radius looks wider than one record and why the incident reads as a group-wide problem.
  4. Explain the duplicate burst separately. The consumer wrote the grant and died before committing its offset, so redelivery is guaranteed by at-least-once semantics — that part is correct behaviour. Duplicates prove the dedupe never fired, so inspect what idempotency_key is actually built from: any component that is client-supplied or regenerated per attempt, such as a fresh UUID or a wall-clock stamp, makes a collision impossible.
  5. Fix both. Make the sink idempotent by deriving the key from match_id, the result signature and the reward slot index, and compare payload_hash so a key replayed with different contents is rejected rather than honoured. Bound the poison record with an attempt counter plus a dead-letter destination so the partition advances and the record is quarantined for inspection.
  6. State the cost: dead-lettering restores availability but breaks per-key ordering for that account, so any downstream consumer that depends on ordering must either tolerate the gap or re-resolve state from the ledger.

Follow-up

  • What changes if the broker acknowledges per message rather than per partition offset? Does the poison record still block anything, and what does it do instead?
  • The grant was written but the offset was not committed. What should the redelivered attempt return, and how does the caller tell that apart from a genuinely new grant?

Suggested practice checklist

For someone who has spent the last few years shipping features and reading other people's code, and who has not solved a timed problem from a blank file in a long time. Five days rebuild the primitives and the patterns that sit on them, working from invariants rather than remembered solutions, and the last two attach that back to the rest of the loop.

Day 1 — Rebuild the primitives by implementing them

  • Implement a dynamic array with doubling growth and an operation counter, then change the growth rule to add a fixed sixteen slots instead, and time both for n of ten thousand, a hundred thousand and a million. The fixed-increment version resizes n/16 times at O(n) each, so its total work is quadratic; doubling is what makes append amortised constant.
  • Implement a hash map with separate chaining and a load-factor resize, then insert ten thousand keys engineered to land in one bucket and record what happens to lookup time, so that average-case O(1) becomes a claim with a stated precondition rather than a reflex.
  • For dynamic-array append and hash-map insert, write down which cost is amortised rather than worst-case, which single operation pays the whole bill, and what a system with a hard per-operation deadline would have to do instead.

Deliverable: Two working implementations plus a timing table showing the input at which each structure's advertised complexity stops holding.

Day 2 — Arrays under an invariant: two pointers, sliding window, binary search

  • Solve longest-subarray-with-sum-at-most-K using a sliding window, then run it on an input containing negative numbers and watch it return the wrong answer: extending the window only moves the sum monotonically when every element is non-negative, and that precondition is the whole reason the technique works.
  • Write the binary search that finds the first index satisfying a predicate rather than an exact value, put the loop invariant above the loop in a comment, and verify termination on the two inputs that break careless versions: the empty range, and a range where every element satisfies the predicate.
  • Compute the midpoint as lo + (hi - lo) / 2 and write one line on why the obvious (lo + hi) / 2 is a genuine defect in a fixed-width integer type and a non-issue in a language with arbitrary-precision integers.

Deliverable: Three solved problems, each with its invariant written above the loop, plus one recorded input on which the sliding window is provably wrong.

Day 3 — Sorting, heaps, and the greedy argument that has to be proved

  • Solve one top-k problem three ways, by full sort, by a size-k heap, and by quickselect, then write the values of n and k at which each becomes the right choice, along with quickselect's quadratic worst case and why a randomised pivot makes that unlikely rather than impossible.
  • Implement bottom-up heapify and count sift-down steps to confirm it does linear work rather than n log n, because most nodes sit near the bottom of the tree and therefore move only a short distance.
  • Take interval scheduling by earliest finishing time and write the exchange argument out in full: given any optimal schedule, swapping in the earliest-finishing interval keeps it feasible and no smaller. Then construct the weighted variant where that same greedy fails and name what has to replace it.

Deliverable: A three-way top-k comparison with measured crossover points, one written exchange argument, and one counterexample to a greedy rule that looks almost identical.

Day 4 — Recursion, memoisation, and the step to a table

  • Take one problem with overlapping subproblems, such as edit distance or coin change, instrument the plain recursion with a call counter to show the blow-up, then add memoisation and re-count.
  • Convert the memoised version to a bottom-up table and state the two properties you relied on: each subproblem's result depends only on its arguments, and the dependencies form a DAG you can enumerate in order.
  • Rewrite one deep recursion with an explicit stack, then find the input length at which the original hits the interpreter's frame limit, which defaults to about a thousand frames in CPython, so you know when the rewrite is required rather than decorative.

Deliverable: One problem in three forms, naive, memoised and tabulated, with call counts for each and the input length at which recursion depth becomes the binding constraint.

Day 5 — Graphs, where most of the work is choosing the traversal

  • Implement BFS and DFS over one adjacency list, then answer for each which finds a shortest path in an unweighted graph and which you would use to detect a cycle in a directed graph, including why the in-progress versus finished distinction matters for the second.
  • Implement topological sort by in-degree, feed it a graph containing a cycle, and confirm the failure signature is that fewer than V nodes come out rather than an exception, then note that the order it produces is one of several valid ones.
  • Run a shortest-path search on a graph with a single negative edge weight and show the wrong answer, then write the precondition Dijkstra actually needs, non-negative weights, because it finalises a node's distance the first time that node is popped, and name the algorithm you would switch to and its own limit.

Deliverable: A small graph library with BFS, DFS and topological sort, plus two inputs that produce documented wrong answers under the wrong algorithm choice.

Day 6 — One day for everything that is not an algorithm

  • Sketch one system only to the depth a coding-heavy loop tends to reach: the endpoints, what the service stores, and the single query pattern that decides the schema. Stop at twenty-five minutes.
  • Prepare the project answer for an interviewer who codes, which means rehearsing the two levels they push to: the specific thing you built, and why you chose that approach over the alternative they will name. Open with a number and be ready to say what it excludes.
  • Prepare the answer to what you would do differently, choosing a real technical mistake with a specific fix rather than a complaint about process or staffing.

Deliverable: One design sketch at endpoint-and-schema depth, plus a project answer rehearsed to two levels of follow-up.

Day 7 — Solve out loud, under time

  • Do three timed problems at twenty-five minutes each in a plain editor with no autocomplete and no execution until the end, then tally separately the failures that were syntax and the ones that were approach, because those two numbers call for different fixes.
  • Narrate one solution from the first sentence, stating the approach and its complexity before writing any code, and rehearse the sentence you will use when you realise mid-solution that the approach is wrong.
  • Re-solve from blank the two problems you were slowest on this week and compare the times against the day they first appeared.

Deliverable: A recording of one fully narrated solution and a tally that separates syntax failures from approach failures.

Risky changes and the reversal path

For anything that touched live traffic, be ready to say how you would have undone it: a flag, a staged rollout, dual writes with the old path still authoritative. Once the old column is dropped or the source rows are overwritten there is no reverse, so name what you kept a copy of and for how long.

What is your experience with version control systems and agile development workflows?

Medium · behavioural and engineering judgement

Approach

  1. Give the blast radius: what could have broken, and what you measured.
  2. Close with what you would do differently, concretely.
  3. Pick a story where you made the decision, not one where you watched it.

Follow-up

  • What did you decide not to do, and why?
  • What would you do differently if you ran that again?

What are you looking for in your next workplace and team culture?

Medium · behavioural and engineering judgement

Approach

  1. Pick a story where you made the decision, not one where you watched it.
  2. State the situation in two sentences and spend the rest on the reasoning.
  3. Close with what you would do differently, concretely.

Follow-up

  • What did you decide not to do, and why?
  • How did you know your change caused the improvement?

Tell me about a challenging project you worked on and how you handled unexpected roadblocks.

Medium · behavioural and engineering judgement

Approach

  1. Name the disagreement and how you resolved it with evidence.
  2. State the situation in two sentences and spend the rest on the reasoning.
  3. Give the blast radius: what could have broken, and what you measured.

Follow-up

  • How did you know your change caused the improvement?
  • What would you do differently if you ran that again?