cswork / libraryOpen practice workspace

Interview guides · Keyrock

Keyrock Software Engineer Interview Questions & Guide 2026

Keyrock software engineer prep on partial failure: a timed-out request has an unknown outcome, not a failed one.

The 60-second brief

As a Software Engineer at Keyrock, you will be at the forefront of the digital asset market-making industry. You will build and optimize low-latency trading infrastructure, ensuring that Keyrock maintains its competitive edge in high-frequency trading (HFT) environments. This role is critical to the firm’s mission of providing liquidity and efficiency across global crypto markets.

Treat capacity estimation as a conversion skill rather than a table to memorise: turn a user count and an action rate into requests per second and bytes per day, then name the component that number breaks first. The figure only matters if it changes the design.

Focus areas

  • Detect sequence gaps before pricing off a book
  • Treat a send timeout as unknown, not failed
  • Replay recorded input through the same build

Overview

You will work on complex, performance-sensitive systems where every microsecond matters. Your contributions will directly influence the reliability, scalability, and speed of the firm’s proprietary trading engines. This is a high-impact position designed for engineers who thrive on solving intricate concurrency challenges and pushing the limits of the Rust programming language in a production-grade trading environment.

The environment at Keyrock is highly technical and fast-paced. Focus your preparation on demonstrating how you handle performance constraints and maintain code quality under pressure.

The Keyrock interview process

Keyrock candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

Half of this call is the part candidates treat as small talk: start date, notice period, work authorisation and its timing, location and time zone, on-call, and the number. Those are what kill offers late, after several engineers have each spent a day. Surfacing a hard constraint now costs you nothing and occasionally buys you something, since a loop compressed to fit a competing deadline can usually only be arranged if it is asked for early. The common failure is deflecting the compensation question twice, then discovering at offer stage that the band never reached your number.

What to demonstrate

  • Whether your hard constraints are compatible with the role before a loop gets booked: earliest start, notice period, what authorisation you hold and when it needs action, days on site, willingness to carry a pager
  • Whether you give a compensation range with something behind it, such as current total compensation or a competing timeline, rather than leaving the band untested
  • Whether your stated timeline is real, since a competing deadline raised now is something scheduling can sometimes work around and the same deadline raised at offer stage usually is not

How to prepare

  • Write each constraint down in one line before the call and state them as facts rather than negotiating them live under a question you were not expecting
  • Set your range from two or three current data points for that level and location, and name the structure you are quoting in, so the number is comparable to the one they are holding
  • If another process is running, say where it stands and by when, and ask directly whether this loop can be scheduled inside that window

Technical Assessments

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

  • Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
  • Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
  • Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

  • Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
  • Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
  • Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Interviews with Senior Engineers

You cannot drill a format you do not know, so put the preparation into material that travels. Three pieces of your own work, each rehearsed until you can take a follow-up you did not anticipate, will carry a conversation or a code walkthrough equally well. Specificity is what separates that from filler. A number needs its definition before it means anything: a p99 is over some window and measured at some hop, and a server-side figure excludes the queueing and network time a client would see. The number you cannot qualify is the one to leave out.

What to demonstrate

  • Whether your examples carry detail only someone who did the work would hold, such as what the binding constraint actually was, which alternative you rejected and why it was worse, and what you measured on each side of the change
  • Whether a number survives one follow-up, meaning you can say what it was measured over and whether it moved because of your change or merely alongside it
  • Whether a failure is described with the specific change that followed it, rather than a lesson stated in general terms
  • Whether your part in a team effort is stated accurately, including what other people did

How to prepare

  • Write a page on each of three projects covering the constraint, the option you rejected, the measurement before and after, and what went wrong. Cut any line you cannot take a follow-up on, since you are writing the parts you will be pressed on rather than a summary.
  • Recover the real figures while you still have access: request volume, data size, latency with its percentile and window, team size, timeline. Note where each came from, whether a dashboard, a design document or memory, and mark the estimates so you can say which they are out loud.
  • Take your weakest project story to someone who works in a different area and have them ask why four times in succession. The point where you run out of answer is the part to go and re-read before the round.

Where candidates lose points

Reporting mean or median latency, and benchmarking with a harness that stops issuing load while the system is stalled.

The distribution that matters is the tail - p99.9 is where a stall meets a burst, and it is precisely the sample an average hides. The common closed-loop harness sends a request, waits for the reply, then sends the next, so it never issues the requests that would have arrived during a stall and never samples the stall at the arrival rate it would have faced. That is coordinated omission, and it can understate the tail by an order of magnitude. Measure with an open-loop generator that holds the intended send schedule, record latency from intended send time rather than actual, and publish percentiles from a high-dynamic-range histogram rather than from a running average.

Reading the wall clock inside strategy or matching logic instead of taking time from the event stream.

It breaks replay outright - the same inputs stop producing the same outputs - and it adds a subtler failure. CLOCK_REALTIME is adjustable, so an NTP correction can step it backward and produce a negative interval or a timer that fires twice. CLOCK_MONOTONIC never steps backward but is not comparable across machines and, on Linux, does not advance while the machine is suspended, so it is the right clock for measuring a duration and the wrong one for stamping an event. Event time has to flow through the system as data, with the wall clock consulted only at the ingestion boundary where a receipt timestamp is taken once and then carried.

Reading the constraints as preamble rather than as part of the problem

The bounds are usually there to eliminate the obvious approach: n up to 10^5 makes an O(n^2) scan roughly 10^10 operations, far outside any per-test time budget, and an input larger than memory rules out loading it at all. When a bound is not given, ask for it, then say out loud which approach it kills.

Assuming the input fits in memory

Ask how large the input is in bytes before committing to an in-memory algorithm; beyond that point the options are a single streaming pass, an external sort with bounded buffers, or a sketch that trades exactness for constant memory. An algorithm that assumes random access to the whole input is a different algorithm from one that sees each element once.

Questions & practice

Find every overlapping effective window before adding the constraint

Hard · interval sweep · skew · null semantics

instrument_version holds about 10^7 rows of (instrument_id, effective_from, effective_to nullable). Versions for one instrument_id must never overlap, and at most one may have effective_to NULL. Adding the EXCLUDE constraint aborts on the first conflicting pair it meets, so you need the whole picture first. Produce every row that overlaps another row for its instrument, per-instrument overlap counts, and every instrument holding more than one open-ended row. The median instrument has about ten versions; the worst has 10^5 after repeated vendor reloads. Explain why the pairwise comparison is correct but cannot ship.

Approach

  1. Cost the naive version properly rather than dismissing it. Grouped by instrument_id it is the sum of g^2 over groups, and at a median g of ten that is trivial — the blow-up lives entirely in the tail, where one 10^5-version group contributes 5 x 10^9 comparisons. Worse, its output is also quadratic: if that group is a repeated reload, nearly every pair overlaps, so the self-join emits billions of rows describing a few thousand bad versions. The output size, not the runtime, is the reason it cannot ship.
  2. Restate the deliverable as rows rather than pairs, which is what makes a linear output possible, then sort by (instrument_id, effective_from, effective_to nulls last) for O(n log n) and sweep each group once.
  3. In the sweep, carry running_max_end = MAX(COALESCE(effective_to, infinity)) over the rows already seen in this group, plus the row that set it. A row overlaps iff its effective_from is strictly less than running_max_end; flag both that row and the max-setter, so both sides of every overlap appear in the output. O(1) extra state per group.
  4. Do not shortcut to comparing adjacent rows only. It is sufficient to answer the boolean does any overlap exist — if every adjacent pair is disjoint then end_i <= start_(i+1) <= start_j for all j > i, so nothing overlaps — but it misses nested intervals when you must list every offending row, and a long open-ended version nesting later ones is the single most common real shape.
  5. If you write it in SQL, the window-function form is MAX(COALESCE(effective_to, 'infinity'::timestamptz)) OVER (PARTITION BY instrument_id ORDER BY effective_from ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING). The COALESCE is load-bearing, and in two distinct ways: MAX ignores NULLs, so an earlier still-open version contributes nothing to the running maximum, and where it is the only preceding row the maximum is NULL outright, which makes effective_from < running_max evaluate to NULL rather than true. Either way the comparison is not true and the overlap disappears from the report.
  6. Handle the open-row rule as its own aggregate in the same pass — COUNT(*) FILTER (WHERE effective_to IS NULL) > 1 per instrument — since a second open row is a violation whether or not your range model makes it overlap, and since it is the only part of the report that survives the COALESCE being dropped. Only then add EXCLUDE USING gist (instrument_id WITH =, tstzrange(effective_from, effective_to) WITH &&), which needs btree_gist for equality on a bigint, and expect to iterate because each failed build names one pair.

Worked solution (35 min)

  1. Build instrument 7 with r1 = [2026-01-01, NULL), r2 = [2026-03-01, 2026-04-01), r3 = [2026-05-01, 2026-06-01), and instrument 9 with two rows both open from 2026-01-01 and 2026-02-01.
  2. Run the sweep with COALESCE to infinity and record which rows are flagged for each instrument.
  3. Run the same sweep with plain MAX(effective_to) and diff the two outputs, evaluating the comparison for r2 and r3 separately.
  4. Run an adjacent-pairs-only variant and diff again.
  5. Attempt to add the EXCLUDE constraint on the unfixed fixture and record what it reports.

Expected result: The correct sweep flags all three rows of instrument 7 (r2 and r3 both nest inside r1's open window) and both rows of instrument 9, and lists instrument 9 in the multiple-open-rows output. Plain MAX flags nothing at all for either instrument: over r2's preceding frame the maximum is MAX(NULL) = NULL, so 2026-03-01 < NULL is NULL rather than true; over r3's frame it is MAX(NULL, 2026-04-01) = 2026-04-01, and 2026-05-01 is not less than that; instrument 9's second row compares against MAX(NULL) = NULL too. The only thing the plain-MAX report still surfaces is instrument 9, and only via the separate open-row count. Adjacent-only, with the COALESCE kept, catches r1/r2 but misses r3, which nests inside r1 and does not touch r2. The constraint build fails naming exactly one pair.

Verify it yourself

  • On a small fixture, the sweep's flagged-row set equals the set produced by a brute-force O(g^2) pairwise comparison — same rows, regardless of pair count.
  • Every row flagged has a named counterpart row that it actually overlaps, checkable by hand from the two ranges.
  • After the flagged rows are repaired, the EXCLUDE constraint builds on the first attempt; if it does not, the report was incomplete rather than the data newly broken.

Follow-up

  • tstzrange(a, a) is empty and empty ranges never overlap, so a zero-width version passes the constraint. Is that acceptable, and what would catch it?
  • The fix requires closing thousands of rows. How do you do that without editing history that a replay depends on?
  • How would you run this check continuously, so the next bad vendor load is caught at write time instead of at constraint-build time?

Attribute every order to the limit version in force

Medium · as-of join · two pointers · temporal data

You have 10^7 order_event rows of type submit sorted by sent_ts, and 10^5 risk_limit versions carrying (limit_id, limit_version, scope_type, scope_id, limit_type, limit_value, effective_from, effective_to nullable). For every order and every scope that applies to it — firm, desk, account, strategy, instrument, asset_class — produce the (limit_id, limit_version) that was in force at sent_ts. Windows are half-open. Write the offline attribution job, state its complexity against a per-order binary search, and define exactly what happens to an order sent at the microsecond a new version becomes effective, and to one whose scope has no version at all.

Approach

  1. Group the version table by (scope_type, scope_id, limit_type) and sort each group by effective_from. Assert non-overlap inside each group first; the whole job assumes at most one version is in force per group per instant, and an unchecked assumption there produces a plausible answer rather than an error.
  2. Partition the orders by the same key. Orders already arrive sorted by sent_ts, so each group is a sorted merge: advance the version pointer while the next version's effective_from is at or before the order's sent_ts, then emit the current one. O(n + v) after partitioning, against O(n log v) for a binary search per order.
  3. Be honest about why the merge wins. At v = 10^5, a binary search is about 17 probes into an array that fits in L2, so the log factor is not the problem; the merge wins because it is a single sequential pass over both inputs with no random access, which at 10^7 orders is a memory-bandwidth story rather than a complexity one.
  4. Pin the boundary in code and in a test: the window is [effective_from, effective_to), so a version effective at 10:00:00.000000 does not bind an order sent at 09:59:59.999999 and does bind one sent at 10:00:00.000000. The columns are microsecond-precision, so exact ties are common enough to hit in production, not a theoretical edge.
  5. Use sent_ts rather than decision_ts or the ack's received_ts, because the property being audited is that the order was evaluated against the limits in force when it went out. Record which timestamp you used in the output schema so a later reader cannot silently reinterpret it.
  6. Emit an explicit no-limit-in-force marker with a reason code when a group has no covering version, rather than omitting the row. An absent row reads as unlimited, and unlimited is the one answer a risk report must never produce by accident.

Follow-up

  • Move this into the live pre-trade path. How does the check read a versioned snapshot without a lock, and how does it record the version it evaluated against?
  • A limit tightens at 10:00:00 while three orders are in flight. What should happen to them, and which component decides?
  • Scopes nest — a firm limit and an account limit both apply. Do you attribute all of them or only the binding one, and what does the answer cost you at 10^7 rows?

Choose a book structure for constant-time top-of-book reads

Medium · data structure choice · cache locality · order book

You maintain a price-level book for one instrument on one venue. Deltas arrive as (side, price, new_aggregate_qty), where price is an integer at the instrument's price_scale and tick_size is constant within the session; new_aggregate_qty of zero deletes the level. Sustained 10^5 to 10^6 deltas per second, over 90 percent landing within ten ticks of the touch, and top of book is read after every delta. Choose the representation, then give apply(delta) and best_bid()/best_ask() with their expected cost, their worst case, and the memory per instrument per side. Say what happens when the price leaves your band.

Approach

  1. Derive the structure from the access pattern rather than from the abstract operation set: reads are overwhelmingly of one element (the touch), writes cluster within ten ticks of it, and the price domain is a dense integer lattice because every valid price is a multiple of tick_size. That argues for a flat array of aggregate quantities indexed by (price - base_px) / tick_size, with cached best_bid_idx and best_ask_idx, over any comparison-based container.
  2. apply(delta) writes one slot in O(1). An insert at a price better than the current best updates the cached index in O(1) with no scan. A delete of a non-best level is O(1). Only a delete of the best level requires finding the next occupied level inward, which is where the worst case lives.
  3. Make that scan cheap by shadowing the array with a bitset, one bit per level, 64 levels per 64-bit word: the next occupied level is a find-first-set from the current index, so a 4096-tick band costs at most 64 word reads and typically one. Memory is 4096 x 8 bytes plus 512 bytes of bitset per side, about 32.5 KiB per side, 65 KiB per instrument.
  4. Price the alternative honestly. A sorted map is O(log L) per update with correct semantics and no band, but it allocates a node per new level on the decode path and chases pointers on every read; the disqualifier is the allocation and the cache misses, not the logarithm. Keep the flat array for the actively quoted set (1,000 instruments is about 65 MB) and a map for the long tail, because 10^5 instruments at 65 KiB each is 6.5 GB.
  5. Handle the band exit explicitly: when a price falls outside [base_px, base_px + width x tick], rebase by memmoving the live window and clearing the rest, an O(width) operation that must be counted and alarmed rather than hidden, since it is a latency spike correlated with fast markets.
  6. Tie it to the gap invariant: the book carries a staleness flag, and an unrecovered sequence gap sets it so best_bid()/best_ask() report unusable rather than returning the last well-formed state.

Follow-up

  • Move from price-level to order-level (each resting order tracked individually). What does that cost you, and what does queue-position modelling need from it?
  • The venue sends a snapshot plus increments after a gap. Where do you buffer the increments, and at which sequence number do you start applying them?
  • How do you verify this book against an independent full-depth snapshot mid-session without stalling the decode thread?

As-of symbology resolution over an effective-dated instrument table

Easy · scd2 · temporal-queries · constraints · audit-history

instrument_version holds one row per instrument_id per effective-dated version: instrument_id BIGINT, venue_mic CHAR(4), venue_symbol VARCHAR(32), price_scale SMALLINT, effective_from TIMESTAMPTZ NOT NULL, effective_to TIMESTAMPTZ NULL. A vendor correction currently arrives as UPDATE ... SET venue_symbol = ... on the open row, and retired instruments carry an is_deleted BOOLEAN flag. Write the query that resolves (venue_mic, venue_symbol) to an instrument_id as of an arbitrary past instant, state the constraint that makes overlapping versions impossible, and explain why the in-place update and the is_deleted flag each break a replay.

Approach

  1. Fix the interval convention before writing anything: treat the window as half-open [effective_from, effective_to) with a NULL upper bound meaning open-ended, so every instant matches at most one version, an instant in a coverage gap matches none, and a boundary instant is not claimed by two.
  2. Write the predicate as effective_from <= $ts AND (effective_to IS NULL OR effective_to > $ts). Both halves are load-bearing, and dropping the upper bound in favour of ORDER BY effective_from DESC LIMIT 1 is not an equivalent query: at any instant inside a coverage gap, where a symbol has been released and not yet reassigned, the correct answer is zero rows, and the lower-bound-only form returns the last closed version instead, which is the issuer that used to hold the symbol. Keep both bounds. A btree on (venue_mic, venue_symbol, effective_from) still serves the full predicate with one descent and a backward walk, and INCLUDE (effective_to, instrument_id) lets the upper-bound test be evaluated from the index tuple, holding Heap Fetches at zero for as long as the visibility map is current for those pages. LIMIT 1 then buys a stopping condition rather than the semantics: under the non-overlap constraint at most one row can qualify anyway, and in a gap the walk reads every earlier version of that symbol before correctly returning nothing.
  3. Enforce non-overlap in the database rather than in the loader: EXCLUDE USING gist (instrument_id WITH =, tstzrange(effective_from, effective_to) WITH &&), which needs the btree_gist extension to get an equality operator class for a bigint. UNIQUE (instrument_id, effective_from) alone happily admits two windows that overlap.
  4. Replace the in-place update with close-the-current-row-then-insert-the-new-one inside one transaction, and replace is_deleted with a new version carrying trading_status = 'delisted'. A row that was true yesterday has to stay readable exactly as it was, because a replay reads it again.
  5. State the consequence that motivates all of it: venue symbols are reassigned to new issuers, so resolving today's symbol against today's open row can return a different instrument_id than the session being replayed actually traded.

Worked solution (20 min)

  1. Insert three versions for one instrument_id: an original symbol, a symbol change, and an open-ended current row. Then insert a fourth row under a different instrument_id that reuses the first symbol some months after it was released, leaving a deliberate coverage gap between the release and the reuse.
  2. Run the as-of lookup at five instants: before the first version, inside each of the three windows, exactly at a boundary timestamp, and inside the coverage gap.
  3. Run the lower-bound-only ORDER BY effective_from DESC LIMIT 1 variant at the gap instant and diff it against the full predicate.
  4. Add the EXCLUDE constraint and attempt an insert whose window overlaps an existing one; confirm it is rejected rather than accepted.
  5. Re-run the lookup for the reused symbol at an old instant and at now, and compare the two instrument_ids.

Expected result: One query returning exactly one row for any instant covered by a version and zero rows otherwise, including every instant in the coverage gap; the boundary instant resolving to the newer version only; the reused symbol resolving to the old instrument_id in the past and the new one today; the overlapping insert failing with an exclusion-constraint violation. At the gap instant the lower-bound-only variant returns the released issuer's version where the full predicate correctly returns nothing.

Verify it yourself

  • For a sample of random instants, SELECT count(*) from the lookup is 0 or 1, never 2.
  • At an instant inside the coverage gap the full predicate returns zero rows while the lower-bound-only form returns one; that difference is the reason the two forms are not interchangeable.
  • SELECT instrument_id, count() FROM instrument_version WHERE effective_to IS NULL GROUP BY 1 HAVING count() > 1 returns no rows.
  • EXPLAIN shows an index scan on (venue_mic, venue_symbol, effective_from) rather than a filter over every version of every symbol.

Follow-up

  • A version was loaded three weeks ago with the wrong effective_from. How do you correct it without editing history, and what second time axis does that force into the table?
  • What does the lookup return during the instant a corporate action closes one version and opens the next, and what guarantees those two writes are one transaction?
  • A replay resolves a million symbol lookups per run. What changes about the query shape at that volume?

Changing a price column's type on a live four-billion-row table

Hard · online-migration · ddl-locks · backfill · data-integrity

execution_report.last_px is NUMERIC(18,6) and must become a BIGINT integer at the instrument's price_scale, which lives on the effective-dated instrument_version row. The table holds 4 x 10^9 rows, takes writes continuously through the session, and is replicated. Produce the migration plan with no write downtime: the exact DDL steps and the lock each takes, how the backfill is batched and throttled, how you verify before cutting reads over, and the rollback point at every stage. Name the correctness bug specific to resolving price_scale during the backfill.

Approach

  1. Add the column cheaply and defensively. ALTER TABLE ... ADD COLUMN last_px_scaled BIGINT is a catalog-only change in PostgreSQL 11 and later for a nullable column or a non-volatile default, so there is no table rewrite. It still takes ACCESS EXCLUSIVE briefly, and that lock queues ahead of every new query, so set lock_timeout to a couple of seconds and retry rather than letting one long-running transaction stall the entire table behind the ALTER.
  2. Start dual-writing before backfilling, from the application or a BEFORE INSERT trigger, so the backfill only ever has to catch up on a closed set of old rows instead of chasing a moving tail.
  3. Backfill in bounded batches over a key range, one commit per batch, throttled on replication lag and autovacuum progress. Every UPDATE writes a new row version, so a full-table backfill roughly doubles the physical size until vacuum reclaims it; unthrottled, it bloats the table and lags the replicas at the same time, and the second symptom hides the first.
  4. Resolve price_scale as of the execution's venue_ts, not as of today. instrument_version is effective-dated, so a scale change since the trade makes a today-resolved backfill mis-scale every historical row of that instrument by a power of ten, silently, because the values stay plausible. Assert per row that the stored NUMERIC carries no more decimals than the target scale, and fail the batch rather than round away real precision.
  5. Verify before cutting reads over: a full pass returning zero rows for last_px_scaled IS DISTINCT FROM round(last_px * power(10::numeric, price_scale)) (numeric power, not the float8 ^ operator), plus a zero count of NULLs, run twice with the second run after further writes have landed. Then move reads behind a flag, soak, and only then DROP COLUMN last_px, which is catalog-only with space reclaimed on the next rewrite.
  6. State the rollback at each stage: before dual-write, drop the column; during backfill, stop and the old column is still authoritative; after read cutover, flip the flag back because both columns remain populated; after the drop there is no rollback, which is why the drop is a separate change weeks later.

Follow-up

  • Where do you add NOT NULL, and what does SET NOT NULL cost on this table compared with a NOT VALID check constraint you validate afterwards?
  • The backfill needs an index that does not exist yet. How do you add it without blocking writes, and what do you do when it fails halfway through?
  • A replica serves reporting and cannot tolerate an hour of lag. What changes about the batch size and the schedule?

How would you design a low-latency order book handler?

Medium · system design and architecture

Approach

  1. Name the read and write paths separately; they rarely have the same bottleneck.
  2. State the consistency you need, and where you are willing to be stale.
  3. Fix the scope first: who calls this, how often, and what they do when it fails.

Follow-up

  • How does this behave when that dependency is down for an hour?
  • What breaks first when traffic grows ten times?

What strategies would you use to minimize jitter in a trading system?

Medium · system design and architecture

Approach

  1. State the consistency you need, and where you are willing to be stale.
  2. Name the read and write paths separately; they rarely have the same bottleneck.
  3. Name the failure you are designing for, then the recovery path.

Follow-up

  • What breaks first when traffic grows ten times?
  • How does this behave when that dependency is down for an hour?

How do you approach the design of a system that must handle sudden spikes in market volatility?

Medium · system design and architecture

Approach

  1. State the consistency you need, and where you are willing to be stale.
  2. Name the read and write paths separately; they rarely have the same bottleneck.
  3. Name the failure you are designing for, then the recovery path.

Follow-up

  • How does this behave when that dependency is down for an hour?
  • What would you drop to keep the system up under load?

How do you profile a Rust application to identify bottlenecks in a hot path?

Medium

Approach

  1. Work from the requirement backwards to the design.
  2. Clarify what is being asked and what a complete answer contains.
  3. State your assumptions explicitly before working the problem.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

How do you ensure thread safety when working with shared state in a high-frequency trading context?

Medium

Approach

  1. Clarify what is being asked and what a complete answer contains.
  2. State your assumptions explicitly before working the problem.
  3. Say what you would check first and why it is the highest-information step.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Evolve the as-of instrument contract without breaking clients

Medium · versioning · backward-compatibility · reference-data

GET /v1/instruments/{instrument_id}?as_of=<ts> serves instrument_version rows: venue_mic, venue_symbol, tick_size NUMERIC(18,9), lot_size, contract_multiplier, price_scale, asset_class, trading_status ENUM('active','halted','auction_only','expired','delisted'), expiry_date, effective_from, effective_to. Six services consume it; two are operated by another team and cannot be redeployed on your schedule. Three changes are requested: add a new asset_class value, replace the scalar tick_size with a price-tiered schedule, and make expiry_date mandatory. Classify each as compatible or breaking, separately for readers and writers, and give the rollout.

Approach

  1. State the compatibility rule before classifying anything: for a response consumer, adding a field is safe, removing or repurposing one is breaking, and loosening a value domain is breaking. Direction matters, so classify per role rather than per field.
  2. New asset_class value: breaking for any client that switches exhaustively or validates against a closed schema. The only cheap mitigation is one you had to write into v1 before you needed it, a documented unknown-value rule such as 'an unrecognised trading_status is treated as halted' (fail closed, because the cost of quoting a halted instrument exceeds the cost of skipping a live one). Absent that rule, a new enum value needs a new version or a per-client value mapping.
  3. Tiered tick size: keep tick_size as the value applicable at the current reference price and add tick_schedule as a new optional array. Old clients continue to read a scalar that is correct for the common case; new clients read the array. Cut a major version only when the scalar becomes actively wrong rather than merely incomplete, and say which instruments those are.
  4. Mandatory expiry_date: for readers this is a tightening of the response and therefore safe, they simply stop seeing NULL. For writers, it is breaking, because a loader that omits the field now fails. Two different answers for one field is the point of classifying by role.
  5. Roll out with the version in the URL path or a media type, run N and N-1 concurrently, and gate the sunset on measured per-client usage rather than on an announcement, because the two services you do not control are exactly the ones whose deploy you cannot schedule.
  6. Pin the contract with tests over recorded responses for both versions in CI; the same recordings that make replay possible make this nearly free.

Worked solution (30 min)

  1. Write a two-column table classifying each of the three changes for readers and for writers, with one sentence of justification each.
  2. Write the v1 response and the proposed v1-plus-additions response side by side and mark every field an old client reads.
  3. Write the unknown-enum-value rule as documentation text a client team could implement from, including the fail-closed default.
  4. Write the rollout: what ships first, what metric gates the sunset, and what the sunset response looks like.
  5. Write one contract test that would have caught the enum change against a recorded old-client fixture.

Expected result: A classification in which the enum addition is breaking absent a pre-declared unknown-value rule, the tick schedule is additive as an optional field alongside a still-correct scalar, and the mandatory expiry_date is safe for readers and breaking for writers, plus a dual-version rollout gated on measured usage.

Verify it yourself

  • Deserialize a new-version payload with the old client's strict schema; if it throws, your 'additive' change is breaking.
  • Confirm an old client reading only tick_size still computes a valid price for every instrument you can name, and list the ones where it does not.
  • Confirm the sunset criterion is a number from a metric, not a date on a wiki page.

Follow-up

  • What does as_of mean for a row whose effective_from is in the future at the time of the call, and what does a correction to a past row do to a client that already cached the old answer?
  • How do you tell which clients are still on N-1 when calls arrive through a shared gateway with no client identity?

Replay diverges once in twenty runs on wide machines

Hard · determinism · concurrency · floating point

Replaying one recorded session through one build produces outbound order streams that differ in about one run in twenty. It reproduces only on 32-core hosts, never under the thread sanitizer, and never in the single-threaded debug build. Two divergence shapes appear: two orders swapping position in the stream, and one limit price differing in the last digit. Give an ordered checklist that locates the first diverging decision and separates the two causes, naming what you would pin and in what order.

Approach

  1. Make the divergence cheap to locate. Byte-diffing two output streams finds the first differing byte, which is usually far downstream of the first differing decision. Hash each decision into a running chain keyed by the input event's sequence number; the first differing chain entry names the event and the decision, turning a rerun-until-it-happens loop into a single comparison.
  2. Separate the two shapes, because their cause sets are disjoint. An ordering difference is an interleaving or an iteration-order problem. A value difference in the last digit is arithmetic, and the precise statement matters: one binary executing the same operations in the same order on the same inputs is bit-reproducible, so a value differing across runs means the order of operations changed or a different code path executed.
  3. For the ordering shape, enumerate and pin one at a time: more than one thread consuming the input stream; a work-stealing pool whose chunk boundaries vary with scheduling; a container whose iteration order is not a function of insertion order - a map keyed on a pointer whose value moves with address layout, a runtime with randomised map iteration or a per-process hash seed, or a set of objects with no stable hash. Force single-consumer ingestion inside the strategy boundary and replace those containers with ordered ones keyed on instrument_id or sequence number.
  4. For the value shape, pin the arithmetic. A parallel reduction whose partition depends on thread count or on stealing gives a different summation order and therefore different rounding; runtime dispatch to a different vector kernel does the same. Then pin the compiler: fast-math permits reassociation, and contracting a multiply and an add into a single fused operation changes the rounding of the intermediate. Build with contraction disabled and fast-math removed, and make any reduction's partition a function of input length alone.
  5. Do not treat a quiet sanitizer as a result. It perturbs timing and allocation enough to close the window and instruments only what it can see, so its silence is weak evidence. Run it, but prove the fix by making the run deterministic by construction and then replaying the recording several hundred times with a zero-tolerance comparison.
  6. Keep it from returning. Add a CI job that replays a fixed recording N times and fails on any difference, and enforce that nothing inside the strategy boundary reads a wall clock: event time arrives as data, and the wall clock is consulted once at ingestion to stamp received_ts.

Follow-up

  • It will not reproduce on an 8-core host. Why would core count matter, and does that narrow which of the two shapes you are chasing?
  • A colleague proposes comparing prices with a 1e-9 tolerance so the test goes green. What does that cost you six months from now?
  • The strategy needs a 200 ms timeout. How do you express it without reading a wall clock?

Suggested practice checklist

Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.

Day 1 — Diagnostic, scored before you study anything

  • Sit a 110-minute diagnostic in four blocks: forty-five minutes on two coding problems, twenty-five on one design prompt taken to interface and data model, twenty of short-answer fundamentals, and twenty delivering two behavioural answers aloud.
  • Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct and 0 is stuck, grading the artifact rather than how the attempt felt.
  • Allocate days two to five in proportion to 3 minus each block's score, write the allocation down, and commit to leaving it alone.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

  • Split the weakest area into named sub-skills and rate each separately. For coding those are restating the problem, choosing the structure, stating the invariant, turning the invariant into loop bounds, handling empty and single-element input, and accounting for complexity out loud.
  • Attempt three items positioned just above where the rating drops off, and for each write the first move you failed to make.
  • Re-attempt one of them from blank four hours later with nothing open.

Deliverable: A sub-skill map with the two blocking sub-skills circled.

Day 3 — Drill the blocking sub-skill by repeating the shape

  • Do eight short repetitions of the same shape rather than eight different problems, so what gets practised is the pattern and not the puzzle.
  • State the rule you now hold in one sentence, then test it against a case built to break it, a sliding window over an array containing negative values, or a cache-aside read path whose invalidation message is dropped.
  • Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on the strongest area

  • Run the same sub-skill decomposition on the second-largest gap in half the time.
  • Spend twenty-five timed minutes on the block you scored highest, choosing the hardest item you can still finish rather than a warm-up.
  • Write whether each area fails you on recall, on setup, or on execution, and set the fix accordingly: repetition for recall, a written checklist for setup, timed work for execution.

Deliverable: A second sub-skill map plus a one-line failure diagnosis for each area.

Day 5 — The gap that is not a skill

  • Record one technical and one behavioural answer, then count two things in the playback: seconds before your first clarifying question, and sentences you began without knowing where they would end.
  • Practise saying that you do not know, followed by how you would find out, without letting it soften into a guess, and practise stating a complexity or an estimate before being asked for it.
  • Redeliver one answer under a hard ninety-second cap, which forces structure ahead of detail.

Deliverable: Two recordings with a counted reduction in time-to-first-question.

Day 6 — Retest under day-one conditions

  • Sit the same 110-minute structure with new prompts of comparable difficulty and score it on the identical rubric.
  • For any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
  • Write down which single block you would still lose the offer on.

Deliverable: A second scored rubric placed beside the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

  • Run a sixty-minute mock over the two blocks that moved least, with an interviewer briefed to interrupt and change direction mid-answer.
  • Write the recovery script for going blank: restate the question, state your assumption, name the first thing you would check.
  • Say every rule from the week aloud without reading it, and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive an interruption.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

Know where your numbers came from

Bring the two or three numbers the story rests on and know how they were collected. A p99 whose timer starts inside your handler excludes the time a request spent queued, so it can sit flat while users wait longer. Give the window, the percentile and what the measurement left out, or drop the number.

How do you handle ownership and borrowing to optimize performance in a multithreaded application?

Medium · behavioural and engineering judgement

Approach

  1. State the situation in two sentences and spend the rest on the reasoning.
  2. Pick a story where you made the decision, not one where you watched it.
  3. Close with what you would do differently, concretely.

Follow-up

  • How did you know your change caused the improvement?
  • What did you decide not to do, and why?

How do you handle data consistency and fault tolerance in a distributed trading environment?

Medium · behavioural and engineering judgement

Approach

  1. Close with what you would do differently, concretely.
  2. Pick a story where you made the decision, not one where you watched it.
  3. Give the blast radius: what could have broken, and what you measured.

Follow-up

  • What would you do differently if you ran that again?
  • How did you know your change caused the improvement?

Estimate a kernel-bypass migration you have never attempted

Hard · estimation · kernel bypass · latency budget

Leadership asks how long it would take to move the order path off the conventional kernel networking stack, where it sits in the tens of microseconds, onto a kernel-bypass stack targeting single-digit microseconds. You have never done this migration. Give an estimate you are prepared to defend: the decomposition, the range and what drives its width, the spike that would narrow it most, and the conditions under which you would recommend not doing it at all. State what you would refuse to put a number on until the spike is finished.

Approach

  1. Refuse the single number first and say what you are giving instead: a range with the driver of its width named. The width is the information the requester actually needs, and handing over a point estimate destroys it.
  2. Decompose by uncertainty, not by component. The transport swap is well-bounded. What is not is everything currently leaning on kernel facilities: packet capture feeding the recording, session recovery on reconnect, failover between redundant lines, the timestamping source, and core pinning and isolation. Mark each item done-before or never-done and estimate only the first group directly.
  3. Attack the premise before estimating the work. Profile where the tens of microseconds actually sit; if a meaningful share is allocation, a lock, a synchronous log write or a page fault, the migration buys much less than the headline and the cheaper work comes first. State that as a decision gate with a threshold, not as a caveat at the end.
  4. Specify the spike as the narrowest experiment that collapses the widest uncertainty: one instrument, one venue session, send and receive only, tick-to-trade p99.9 measured against the current path on the same open-loop harness, with a fixed time box and a written question it must answer.
  5. State the costs that persist after delivery, because that is the part usually missing. Taking the network out of the kernel also takes it out of the kernel's tooling, so capture, counters and the existing packet recording need replacements, and a core is now permanently dedicated. Those are recurring costs against a one-time latency gain.
  6. Name the conditions for not doing it and what you will not estimate yet: if the strategies are not latency-sensitive at the margin, or the venue queue rather than your stack is the binding constraint, the answer is no — and the recovery and capture rework stays unestimated until the spike says what the new stack actually provides.

Follow-up

  • Give me one number anyway, with the probability you attach to it.
  • The spike shows 4 microseconds where you projected 9. What do you do with the estimate?
  • What would you cut to get half the benefit in a quarter of the time?