Intercontinental Exchange Holdings Software Engineer Interview Guide

The 60-second brief

At Intercontinental Exchange Holdings (ICE), software engineering is the backbone of the global financial ecosystem. As a Software Engineer, you will design, build, and maintain high-throughput, low-latency systems that power global exchanges, clearing houses, and mortgage technology platforms. The systems you work on are responsible for processing millions of transactions daily, meaning reliability, security, and extreme precision are non-negotiable.

Allocate preparation against your weakest link rather than your favourite topic. A strong algorithm habit usually comes with weak out-loud explanation of tradeoffs, and years of shipping usually come with rusty from-scratch implementation under a clock.

Focus areas

• Build at-least-once pipelines with explicit deduplication horizons
• Evolve APIs without breaking pinned SDK clients
• Keep money in integer minor units

Overview

Unlike typical consumer-tech companies, the engineering challenges at ICE sit at the intersection of complex financial domain logic and high-performance computing. Whether you are developing core backend services in Java, writing low-latency execution engines in C++, building modern user interfaces in React, or managing massive data pipelines with Kafka and SQL, your work directly impacts the stability of global markets.

This role requires a unique blend of deep technical expertise and a highly disciplined engineering mindset. You will collaborate with cross-functional teams of developers, business analysts, database administrators, and operations specialists to deliver robust, production-grade software. It is an inspiring and highly critical environment where your code must be designed to withstand extreme market volatility and strict regulatory compliance.

The Intercontinental Exchange Holdings interview process

Intercontinental Exchange Holdings candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

HR Screening Call

Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.

What to demonstrate

• Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
• Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
• Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled

How to prepare

• Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
• Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
• Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market

Technical Screening

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

• Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
• Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
• Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

• Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
• Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
• Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Panel/Onsite Interviews

A day like this is several different games in a row, and the expensive mistake is carrying the previous one into the next room. Coding rewards narrow precision and finishing inside a timer. Design rewards breadth, stated assumptions and naming what you are deliberately not building. Behavioural rewards specificity about people and decisions. Candidates who over-engineer a coding problem they were supposed to finish, or who start sketching class hierarchies before anyone has agreed what the system has to do, are usually still playing the last round. Between rooms, name out loud which game the next one is.

What to demonstrate

• Whether the coding round ends with something that runs and has been traced against a degenerate input, rather than an extensible design that was never finished
• Whether a design discussion opens by agreeing on traffic shape, read-to-write ratio and what is allowed to be stale, instead of proceeding from an architecture you arrived with
• Whether a behavioural answer names a person, a disagreement and what you did about it, rather than describing the system the story happened inside
• Whether the opening habits still appear late in the day: restating the problem, asking for constraints, saying the plan before typing

How to prepare

• Book three mocks of different types back to back on one afternoon and ask each interviewer afterwards which round you answered in the wrong mode
• Write a three-line opening script per round type — coding: restate, name the approach and its cost, then type; design: ask for scale, read-write mix and what must not break; behavioural: name the person, the stakes and the decision — and run it off a card so the switch is mechanical rather than remembered
• Practise coding with a timer you do not extend, stopping when it stops, so the trained reflex is to finish a correct solution rather than to keep improving one

Where candidates lose points

One shared connection pool for every tenant and every query class

A single tenant with a large table and a missing index can occupy every connection with slow queries, and every other tenant then waits in connection acquisition -- a queue invisible in database metrics, because the database itself looks healthy while the application starves. Containment is bulkheads: separate pools or per-tenant concurrency caps for interactive requests, background jobs and exports, a statement timeout low enough that a pathological query dies before it accumulates, and an idle-in-transaction timeout so a stuck client cannot pin a connection and its locks indefinitely. One caveat worth knowing in advance: if a transaction-pooling proxy sits in front of the database, session-scoped behaviour changes, so session-level advisory locks and settings applied outside a transaction do not survive the way they do on a direct connection.

Serialising a tenant's writes through select ... for update on a single counter row

It is the first change that makes a counter correct, and it caps that tenant's write throughput at roughly one divided by the lock hold time. A transaction that takes the lock, makes a network call and then commits holds it for the entire round trip: at 2 ms that is about 500 writes per second for the whole tenant, and the largest tenants are exactly the ones that exceed it. The damage then spreads, because every waiter holds a database connection while it queues, so one hot tenant drains the shared pool and the symptom presents as a site-wide latency incident rather than as a lock problem. The repairs are to shrink the critical section to a single statement, to shard the counter into per-(tenant, hour) or per-(tenant, bucket) rows and sum on read, or to batch in memory and flush periodically while accepting the bounded loss that batching implies.

Not asking what the system looks like if it dies halfway through

For any multi-step write, say what state remains if the process stops between step two and step three, and what brings it back: a single transaction, a saga with compensating actions, an outbox, or a reconciliation job. Partial failure is routine at any real call volume, so 'that shouldn't happen' is an answer with nothing behind it.

Writing code before the input contract is pinned down

Before the first line, state the types, the size bounds, whether duplicates, negatives or an empty input are possible, whether the input is sorted, whether you may mutate it, and what the function returns when nothing matches. Every one of those answers changes the code, and discovering one at minute twenty costs a rewrite you no longer have time for.

Questions & practice

Given an array of integers, find the contiguous subarray with the largest sum and return its sum.

Medium · data structures and algorithms

Approach

1. Walk one small example through your approach before writing the whole thing.
2. Name the brute-force solution and its complexity before improving on it.
3. Choose the data structure from the access pattern, not from familiarity.

Follow-up

• How does this change if the input no longer fits in memory?
• What is the worst case, and how likely is it on real data?

Implement a function to perform a deep copy of a complex JavaScript object.

Medium · data structures and algorithms

Approach

1. Choose the data structure from the access pattern, not from familiarity.
2. Walk one small example through your approach before writing the whole thing.
3. State the target complexity and say which constraint rules the naive version out.

Follow-up

• What is the worst case, and how likely is it on real data?
• Which test case would catch an off-by-one here?

Explain the performance characteristics and time complexity (Big O) of an ArrayList, HashSet, and HashMap.

Medium · data structures and algorithms

Approach

1. Restate the input: its shape, its size, and what is guaranteed about it.
2. Walk one small example through your approach before writing the whole thing.
3. Name the brute-force solution and its complexity before improving on it.

Follow-up

• How does this change if the input no longer fits in memory?
• What is the worst case, and how likely is it on real data?

Write a program to insert a node into a Binary Search Tree (BST) using both recursive and iterative approaches.

Medium · data structures and algorithms

Approach

1. State the target complexity and say which constraint rules the naive version out.
2. Name the brute-force solution and its complexity before improving on it.
3. Choose the data structure from the access pattern, not from familiarity.

Follow-up

• Which test case would catch an off-by-one here?
• How does this change if the input no longer fits in memory?

What are the key differences between C++11, C++14, and modern C++20 features?

Medium · languages, concurrency and fundamentals

Approach

1. Name what is shared across threads and what owns each piece of state.
2. Distinguish a value from a reference to it, and say which one you handed out.
3. Reach for the cheapest primitive that closes the race, not the broadest lock.

Follow-up

• Where could this allocate more than you expect?
• How would you prove the race exists rather than suspect it?

Describe the process of conducting a rigorous code review. What specific patterns do you look for to ensure thread safety and efficiency?

Medium · languages, concurrency and fundamentals

Approach

1. Name what is shared across threads and what owns each piece of state.
2. Distinguish a value from a reference to it, and say which one you handed out.
3. Identify the window where an invariant is briefly untrue.

Follow-up

• Where could this allocate more than you expect?
• What happens if two callers reach this at the same time?

Order a job dependency graph and find its critical path

Medium · topological-sort · dag-longest-path · cycle-detection · critical-path

A workspace defines up to 50,000 jobs with up to 200,000 dependency edges and an estimated duration_seconds per job. Given the edge list, reject the graph if it contains a cycle and name one cycle's nodes; otherwise return a valid execution order, the earliest possible completion time with unlimited workers, and the set of jobs whose slack is zero. Then say which single job to shorten in order to cut the completion time, and by exactly how much. State the complexity of each part.

Approach

1. Kahn's algorithm for the order: compute indegrees, seed a queue with zero-indegree nodes, emit and decrement. O(V + E), which at 50,000 and 200,000 is milliseconds. If fewer than V nodes are emitted, the graph contains a cycle.
2. Kahn detects a cycle but cannot name one. The nodes left with indegree above zero contain every cycle, so run one DFS restricted to that residual subgraph with three-colour marking and report the stack slice from the grey node the back edge points at. That is the difference between a usable error message and 'dependency cycle detected'.
3. Earliest completion with unlimited workers is the longest path, which is NP-hard on a general graph and linear on a DAG. State the precondition, then relax in topological order: earliest_finish[v] = duration[v] + max(earliest_finish[u] for u in preds(v)), taking the max over an empty predecessor set as zero. The makespan T is the maximum over all nodes. O(V + E).
4. Second pass in reverse topological order for latest_finish, then slack[v] = latest_finish[v] - earliest_finish[v]. Zero-slack nodes form the critical path, and there can be several disjoint critical paths, so return the set rather than one chain. slack[v] = 0 is exactly the statement that some longest path runs through v; equivalently, the longest path through v has length T - slack[v].
5. The speed-up bound is the point of the question, and the obvious form of it is wrong. Shortening a zero-slack job v by d, with 0 <= d <= duration[v], cuts the makespan by min(d, T - L_avoid(v)), where L_avoid(v) is the longest path in the graph with v deleted: the longest path that avoids v, not the second-longest path overall. The two coincide only when the runner-up path misses v. Counterexample: A of 10 s feeds both B of 5 s and C of 4 s, so T = 15 s and the second-longest path is 14 s, yet shortening A by 10 s leaves a makespan of 5 s. The realised gain is the full 10 s, because both paths ran through A and shrank together, while min(10, 15 - 14) predicts 1 s. The reason is structural: shortening v reduces every path through v by d and leaves every other path alone, so the new makespan is max(T - d, L_avoid(v)).
6. Compute L_avoid(v) the direct way: delete v and re-run the same forward relaxation, O(V + E) per candidate. The cheaper equivalent skips the deletion, since L_avoid(v) only ever matters through that max: set duration[v] := 0, recompute the makespan as T0(v) = max(T - duration[v], L_avoid(v)), and the gain is min(d, T - T0(v)), which is identical for every d <= duration[v]. Only zero-slack jobs are candidates, because shortening a job with positive slack changes the completion time not at all. One relaxation is milliseconds at this size, so ranking a critical set in the hundreds costs O(k(V + E)) and is worth doing exactly; a critical set in the tens of thousands is not, and there you evaluate a shortlist, longest jobs first, and say that the answer is the best of that shortlist rather than the optimum.

Worked solution (30 min)

1. Build four fixtures. A: 12 jobs, two branches of 100 s and 95 s that share no job. B: fixture A plus one back edge. C: two disjoint paths tied at 100 s. D: the shared-prefix case, one job of 10 s feeding a 5 s job and a 4 s job, so the longest path is 15 s and the runner-up is 14 s.
2. Run Kahn; on fixture B confirm it emits fewer than V nodes, then run the residual-subgraph DFS and print the actual cycle.
3. Compute earliest_finish forward and latest_finish backward, and list the zero-slack set for each fixture.
4. For each zero-slack job v, recompute the makespan with duration[v] := 0 to get T0(v), and record both the correct bound T - T0(v) and the wrong one, T - second_longest_path, side by side.
5. Apply the shortening for real (20 s off the critical branch of A, 10 s off the shared prefix of D) and diff the recomputed makespan against each prediction.

Expected result: Fixture A: makespan 100 s, and shortening by 20 s leaves 95 s, a gain of 5 s. Both formulas agree here, because the 95 s branch avoids the shortened job. Fixture D: makespan 15 s, and shortening by 10 s leaves 5 s, a gain of the full 10 s, which T - T0(v) = 15 - 5 = 10 predicts and T - second_longest = 1 does not. Fixture C: the zero-slack set covers both tied paths, and shortening a job on one of them alone gains nothing, since the other path still runs 100 s.

Verify it yourself

• The cycle printed by the DFS is a real cycle: every consecutive pair is an edge present in the input.
• Summing durations along any zero-slack path equals the makespan exactly.
• For every candidate, the predicted gain equals the makespan recomputed from scratch with the shortened duration. Assert the equality; do not eyeball it.
• Fixture D is the regression test for the bound, and a fixture whose branches are disjoint is not: it passes under either formula and proves nothing.

Follow-up

• Only m workers are available. What happens to your answer, and what can you still promise about the schedule you produce?
• Edges arrive incrementally as the customer edits the pipeline. How do you detect a cycle at insert time without re-running Kahn over 250,000 elements?
• Durations are estimates. How would you express completion time as a distribution, and what breaks about the critical path once you do?

Explain why the metering dashboard scans every daily partition

Medium · indexing · partition pruning · sargability · tenant skew

usage_event is range-partitioned daily on ingested_at and holds tenant_id, workspace_id, environment, sku, quantity numeric(20,6), occurred_at and ingested_at. The only relevant index is on (occurred_at). A dashboard runs select sku, sum(quantity) from usage_event where tenant_id = $1 and date_trunc('hour', occurred_at) >= $2 and environment = 'production' group by sku, and EXPLAIN shows a sequential scan of every partition. Give each distinct reason, rewrite the predicate so an index can serve it, propose the index, and state the write cost its column order adds.

Approach

1. Separate the three causes rather than blaming one. First, date_trunc('hour', occurred_at) wraps the column, so the predicate is not sargable against a btree on the bare column. Second, pruning keys off ingested_at while the query constrains occurred_at, so no partition can be excluded. Third, even made sargable, (occurred_at) is not tenant-leading, so for one tenant among thousands the scan reads the whole time range and discards nearly all of it.
2. Rewrite the bound carefully, because the obvious rewrite is only conditionally equivalent. date_trunc('hour', x) >= $2 equals x >= $2 only when $2 is already hour-aligned; for an arbitrary $2 it means x >= date_trunc('hour', $2) + interval '1 hour'. Normalise the parameter in the caller and leave the column bare.
3. Restore pruning with a second, redundant predicate on the partition key: ingested_at >= $2 - interval '<late-data horizon>'. State both sides of it. It prunes to a handful of partitions, and it silently omits any event whose ingest lagged past that horizon, which is precisely what a producer replay produces. Either document the horizon as a stated bound, or partition on occurred_at and move the problem into the dedup window instead.
4. Propose (tenant_id, occurred_at) include (sku, quantity) per partition. A partial index where environment = 'production' mostly saves size rather than selectivity, since production dominates the three environments; take it if non-production is a meaningful share and skip it otherwise.
5. Price the write path honestly. At roughly 250M rows/day each extra index is another insert plus WAL per row, and a tenant-leading key scatters inserts across one hot leaf per active tenant instead of appending to a single rightmost leaf, so page dirtying and random I/O both rise. An INCLUDE payload widens every leaf entry and enlarges the index accordingly.
6. Add the index-only-scan caveat before someone reports it as a regression: on a freshly appended table the visibility map is not yet set for recent pages, so the INCLUDE columns still cost heap fetches until autovacuum has been through, and the newest hour is exactly the data the dashboard reads.

Worked solution (30 min)

1. Build 30 daily partitions with skewed tenants, one holding about 40% of the rows, then ANALYZE.
2. Run explain (analyze, buffers) on the original query and record how many partitions were scanned and the rows removed by filter.
3. Apply the rewritten predicate and the index, re-run, and confirm the plan lists only the partitions inside the ingested_at bound.
4. Re-run with $2 set to a non-hour-aligned timestamp and confirm the rewritten and original predicates return identical rows.
5. Insert an event with ingested_at six hours past occurred_at and check whether the pruning predicate excludes it.

Expected result: Three causes named separately: a non-sargable date_trunc, pruning on ingested_at against a predicate on occurred_at, and a non-tenant-leading index. A rewrite to occurred_at >= <hour-normalised $2> plus a redundant ingested_at >= $2 - horizon, an index on (tenant_id, occurred_at) including sku and quantity, a per-row write cost stated, and the late-data omission the pruning predicate introduces stated as a known bound.

Verify it yourself

• The plan lists only partitions inside the ingested_at bound.
• Rows removed by filter on the index scan is small relative to rows returned.
• The rewritten and original predicates agree for both hour-aligned and non-aligned bounds.
• A deliberately late-ingested event is visible or omitted exactly as the stated horizon predicts.

Follow-up

• CREATE INDEX CONCURRENTLY is not supported on a partitioned parent. Give the sequence that gets this index onto 400 existing partitions without blocking ingest.
• One tenant holds 200 times the median row count and the dashboard still times out for them with the index in place. What changes?
• Should this read hit usage_rollup_hourly instead? State what that costs in freshness and what the watermark lets you promise.

Find the join that inflates every invoice total

Medium · join fan-out · aggregation grain · sql correctness

invoice_line_item holds line_id, invoice_id, tenant_id, sku, rate_tier, quantity, unit_price_micros, amount_minor (bigint), currency, kind, voided_at. invoice_payment_attempt holds attempt_id, invoice_id, tenant_id, amount_minor, status (succeeded, failed, pending), created_at, and an invoice has many attempts. A finance report runs select i.invoice_id, sum(l.amount_minor), count(p.attempt_id) from invoice i join invoice_line_item l using (invoice_id) join invoice_payment_attempt p using (invoice_id) group by 1 and the totals are wrong. Say precisely what the sum now equals, and write a version that is also correct for invoices with zero attempts.

Approach

1. Compute what the query actually returns before fixing it. The two joins form a Cartesian product per invoice, so each line row repeats once per attempt row: sum(l.amount_minor) is the true total multiplied by the attempt count, and count(p.attempt_id) is attempts times lines. Three lines and two attempts report double the money and six attempts.
2. Reject the reflex repair. count(distinct p.attempt_id) does fix the count, because attempt_id is unique. sum(distinct l.amount_minor) does not fix the sum, because two legitimate lines with equal amounts collapse into one. DISTINCT inside an aggregate deduplicates values, not rows, and the difference stays invisible until two lines happen to match.
3. Aggregate each branch to invoice grain before joining: one CTE summing lines by invoice_id, one counting attempts by invoice_id, then join the two results. A LATERAL subquery per invoice is equivalent and sometimes plans better when the outer set is small. Either way every aggregate stays at the grain it was defined at.
4. Keep invoices with no attempts by making the attempt branch a LEFT JOIN with coalesce(attempt_count, 0). An inner join here silently drops every unpaid invoice, which is usually the exact population finance is asking about.
5. Push each filter to its own grain: where l.voided_at is null belongs inside the line CTE, not the outer query, or it would also filter the attempt branch through the join. Put the tenant predicate on both branches, since the denormalised tenant_id is what stops a wrong join crossing tenants.
6. Leave yourself a standing check: an invoice total is a function of its non-voided lines and of nothing about payments, so if changing the payment filter moves the money figure, the fan-out is back.

Follow-up

• Add a third branch for credit notes applied to the invoice. Does the CTE shape still hold, and when would a single pass with filter (where ...) be better?
• Over 500k invoices this report takes minutes. Which grain would you materialise, and how do you keep it correct when a line is voided?
• The same report is needed per tenant per month. What index makes the line CTE cheap?

Walk through the lifecycle of a request in a Spring Boot controller and explain how dependency injection is managed.

Medium

Approach

1. Work from the requirement backwards to the design.
2. State your assumptions explicitly before working the problem.
3. Say what you would check first and why it is the highest-information step.

Follow-up

• What assumption would you test first?
• How would you know your answer was wrong?

Webhook fan-out with per-endpoint isolation and backoff

Medium · fan-out · backpressure · circuit breaker · ordering

One domain event fans out to every matching subscription, producing a webhook_delivery row per (subscription_id, event_id, redelivery_seq). Peak unique event rate is 20k/second; attempts run five to ten times that once fan-out and retries are counted. One customer endpoint has returned 503 for six hours and its backlog holds days of events; every other customer must be unaffected. Design the delivery system: how a worker claims work, the backoff schedule, the per-endpoint circuit breaker, the queue partitioning, and whether you offer ordering per subscription. State the delivery guarantee in one sentence.

Approach

1. State the guarantee first, because it determines the rest: at-least-once with a stable event_id, and the consumer documented as responsible for idempotency. Exactly-once over HTTP is not deliverable - the 200 can be lost after the customer has already committed - so any design that promises it is either lying or is really offering at-most-once.
2. Partition work per subscription rather than into one global pool, with a concurrency cap per subscription. With a shared pool, the endpoint that has been dead for six hours consumes workers on retries that will fail, and every other customer's delivery latency rises: head-of-line blocking across tenants is the exact failure being designed against here.
3. Claim by compare-and-set with a fencing token: UPDATE webhook_delivery SET status = 'in_flight', lease_token = $new, leased_until = now() + interval '60 seconds' WHERE delivery_id = $1 AND status IN ('pending','failed_retryable') AND (leased_until IS NULL OR leased_until < now()), and make the terminal write carry AND lease_token = $new so a paused worker's late write is rejected rather than overwriting a newer attempt. Find due work through the partial index on next_attempt_at WHERE status IN ('pending','failed_retryable'), so the scan is proportional to live rows rather than to the terminal rows that outnumber them by orders of magnitude.
4. Use full jitter: sleep uniformly in [0, min(cap, base x 2^(attempt-1))]. Plain exponential backoff hands a recovering endpoint its entire backlog as one synchronised herd and knocks it over again; full jitter de-correlates it. Then check the schedule actually spans the retention you promise - with base 1 s and a 3,600 s cap, twenty attempts have an expected total elapsed time of only about 4.6 hours, so a twenty-four-hour promise needs roughly fifty-nine attempts or a larger cap.
5. Trip a circuit per endpoint on consecutive failures or a failure ratio over a rolling window: stop dispatching, push next_attempt_at out or mark new deliveries dropped_circuit_open, and half-open with exactly one probe rather than a batch. Bound the backlog explicitly with a per-subscription cap or retention, and decide in advance whether a recovered endpoint receives six hours of events at full rate or a pointer telling it to fetch what it missed.
6. Offer ordering only as an opt-in mode of one in-flight attempt per subscription, and price it honestly: with parallel attempts a retried event overtakes a newer one, so ordering requires serialisation, and serialisation means one slow endpoint blocks its own queue entirely. That converts a shared problem into that customer's own problem, which is the right place for it, but it is still a real cost.

Worked solution (35 min)

1. Compute the attempt rate: 20k unique events/second x mean fan-out x retry multiplier, and size worker pools and the per-subscription concurrency cap from it.
2. Write the claim statement and the terminal write, and point at the clause that rejects a resumed worker's stale write.
3. Tabulate the backoff for attempts 1 to 20 with base 1 s and cap 3,600 s, take the expected value of each full-jitter sleep as half its ceiling, and sum to get total expected elapsed coverage.
4. Decide the policy for a subscription down six hours: events buffered, bytes held, and what the customer actually receives when it returns.

Expected result: At-least-once with a stable event_id; per-subscription lanes with a concurrency cap; lease-and-fence claiming; full jitter whose schedule matches the retention promised - with base 1 s and a one-hour cap, twenty attempts covers only about 4.6 hours of expected elapsed time, so twenty-four hours needs roughly fifty-nine attempts or a larger cap; a per-endpoint breaker with a single half-open probe; ordering offered only as opt-in single-in-flight with head-of-line blocking named as its price.

Verify it yourself

• Sum the expected sleeps in your schedule and compare against the retention window you promised the customer. If the two numbers disagree, one of them is wrong and it is usually the promise.
• Execute the terminal write with a stale lease_token and confirm it affects zero rows. If it succeeds, the lease was doing the work alone and the token is decorative.
• The dead endpoint's backlog must not be visible in any queue another subscription's workers poll. If one index scan serves all subscriptions, the isolation is nominal.

Follow-up

• The endpoint recovers. Does it receive six hours of events at full rate, and what does that do to it?
• Trace the exact code path by which an event belonging to one tenant could be signed and sent to another tenant's endpoint.
• A customer insists they never received an event your row marks delivered. What evidence do you have, and what does payload_digest let you prove?

Regional error rate explodes after a dependency merely slows

Hard · retry storm · circuit breaker · partial outage

A control-plane read replica in one region degrades from 4 ms to 120 ms. Within ninety seconds that region's gateway error rate rises from 0.01% to 40% and its p99 becomes bimodal, one mode near the old p99 and one at the client timeout. The other two regions are unaffected. The gateway retries control-plane reads three times with exponential backoff and no jitter. Give an ordered checklist that separates trigger from amplifier, the offered-load arithmetic, and the controls that break the loop.

Approach

1. Split the incident into three questions before touching a control: what started it, what amplified it, and what would make recovery slow. Here they are the replica slowdown, the retry policy interacting with queueing, and a synchronised unjittered herd at recovery. They are different mechanisms and each needs its own control.
2. Read the distribution rather than the mean. A bimodal p99 with one mode pinned at the client timeout is two populations, not one degraded path; split latency by cache hit and miss and confirm the fast mode is hits and the timeout mode is misses that reached the replica.
3. Do the load arithmetic. Three retries turn one client request into up to four upstream requests, so offered load reaches roughly 4x on a dependency that is already slower, and it arrives at the worst moment. With utilisation approaching one, queueing delay grows superlinearly, which is why a 30x latency increase upstream does not produce a 30x increase downstream, it produces timeouts.
4. Break the loop with controls that bound offered load rather than with more attempts: a concurrency limit on the control-plane client so at most N calls are in flight and the remainder fail fast, a circuit breaker scoped per dependency and region, and a retry budget capping retries at a small fraction of base traffic so amplification has a ceiling that does not depend on how many clients are retrying.
5. Add full jitter to whatever retries survive, sleeping uniformly in [0, min(cap, base x 2^attempt)], so attempts de-correlate instead of arriving in waves aligned to the moment of failure.
6. Decide the unreachable-dependency behaviour in advance, because it is the actual product decision underneath: serving from an expired credential cache keeps the product available while extending a revoked key's life past the stated bound, and failing closed converts a dependency degradation into a total outage. State the mode and the staleness number rather than letting the timeout choose.

Follow-up

• The replica recovers. Describe what happens in the first ten seconds with your controls in place versus without them.
• Which single metric would have paged before the error rate moved, and why is upstream latency by itself not it?
• Requests that fail fast under the concurrency limit still need an answer. What does the gateway return, and what does it do to the usage event it would otherwise have emitted?

Suggested practice checklist

Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.

Day 1 — Coding, one pass at shallow depth

• Solve one problem from each of six families, an array with two pointers, hash counting, binary search, a tree traversal, a graph traversal and one dynamic program, under a hard twenty-minute cap with no extensions, marking each finished, late, or stalled.
• For every stall, write the exact move you could not make rather than the subject, so the note reads could not turn the recurrence into a loop rather than bad at dynamic programming.
• Fix nothing today. The value of the pass is the unfixed record.

Deliverable: Six timed attempts marked finished, late or stalled, each stall carrying a named blocking move.

Day 2 — Design, one pass at shallow depth

• Spend twenty minutes each on three different shapes, a read-heavy feed, a write-heavy ingest path, and something needing a transaction across two entities, stopping each at requirements, interface and data model.
• After each, write the first question you could not answer, which is usually a number you could not estimate or a failure mode you had no vocabulary for.
• Mark which of the three you would be most relieved not to be asked, and treat that as data rather than as a preference.

Deliverable: Three shallow designs, each with the first unanswerable question written at the bottom.

Day 3 — Fundamentals and the practical rounds

• Answer eight short questions in writing at four minutes each, covering the material that fills the gaps between the big rounds: what happens between a URL and a rendered page, what an index costs on write, when a process is preferable to a thread, and what conditions a deadlock requires.
• Do one thirty-minute practical task of the kind a take-home compresses: read an unfamiliar two-hundred-line file and write what it does, what you would change, and the one thing you remain unsure of.
• Score every answer fluent, correct but slow, or absent, and keep the absent ones visible.

Deliverable: Eight scored short answers and one written reading of unfamiliar code.

Day 4 — The rounds that are about you, and the map

• Deliver three behavioural answers aloud against a timer, a conflict, a failure you owned, and a decision made without enough information, marking any that ran past three minutes or contained no number.
• Assemble the map: every marked item from days one to three on a single page, sorted by how likely it is to appear in your loop rather than by how uncomfortable it felt.
• Choose exactly two areas for the remaining three days and write down what you are deliberately abandoning.

Deliverable: A one-page scored map of the whole surface area with two areas chosen and the rest explicitly abandoned.

Day 5 — First chosen area, to the depth you skipped

• Work the higher-ranked area in four focused blocks, choosing items one level above where you stalled rather than repeating what already works.
• After each block write the rule you extracted in one sentence with its precondition attached, since a rule carrying no precondition is exactly what fails under a variation.
• Re-attempt the day-one or day-two item that exposed this area and compare against the original timing.

Deliverable: Four worked blocks, a timed re-attempt against the original, and three one-sentence rules with preconditions.

Day 6 — Second chosen area, where the gap is coverage rather than speed

• Treat the second area differently from the first. Day five drilled something you could already half-do; this one is usually a topic you had simply never met, so build one worked reference example end to end and keep it, rather than attempting six problems badly.
• Write down the vocabulary you were missing on day two or three, five terms at most, each with the one sentence that makes it usable in an answer rather than the textbook definition.
• Redo the shallow attempt that exposed this area and note whether you now fail later in the problem, because moving the failure point is the realistic gain from a single day and is worth more than a score that did not change.

Deliverable: One worked reference example for the newly covered area, a five-term vocabulary list, and a note on where the failure point moved.

Day 7 — Reassemble the loop

• Sit two rounds back to back with no gap, ordering them so the area you chose second comes last, because the map was built from rested, isolated attempts and the loop will reach your weaker area when you are already spent.
• Write where the second round suffered from the first, which is normally the point at which structure collapses into narration.
• Reduce the week to one page holding only the rules you can state without reading them.

Deliverable: Mock notes on cross-round carryover plus a one-page card of rules you can recite from memory.

Risky changes and the reversal path

For anything that touched live traffic, be ready to say how you would have undone it: a flag, a staged rollout, dual writes with the old path still authoritative. Once the old column is dropped or the source rows are overwritten there is no reverse, so name what you kept a copy of and for how long.

How do you handle a situation where an interviewer or teammate disagrees with your technical approach or solution?

Medium · behavioural and engineering judgement

Approach

1. State the situation in two sentences and spend the rest on the reasoning.
2. Pick a story where you made the decision, not one where you watched it.
3. Give the blast radius: what could have broken, and what you measured.

Follow-up

• What did you decide not to do, and why?
• What would you do differently if you ran that again?

Describe a situation where you had to deliver a feature under a tight deadline with incomplete requirements. How did you proceed?

Medium · behavioural and engineering judgement

Approach

1. State the situation in two sentences and spend the rest on the reasoning.
2. Pick a story where you made the decision, not one where you watched it.
3. Close with what you would do differently, concretely.

Follow-up

• What did you decide not to do, and why?
• What would you do differently if you ran that again?

Reverse a webhook ordering decision after measuring its cost

Medium · reversing decisions · head-of-line blocking · at-least-once · api contracts

You argued for strict per-subscription ordering in webhook-delivery, which means one in-flight attempt per subscription. It shipped. Three months later a single unresponsive endpoint holds one subscription's queue at a six-hour backlog, and two customers report events arriving out of order anyway once their own retries are counted. Describe a decision you reversed: what you originally optimised for, the measurement that changed your mind, what the reversal cost in engineering time and customer change, and how you told the people who had already built on the original guarantee.

Approach

1. State the original decision as a trade you made knowingly. Ordering across a network requires a single in-flight attempt per subscription, and its price is head-of-line blocking whenever one endpoint is slow. 'We priced it wrong' is a much stronger opening than 'we did not realise', and it is usually the true one.
2. Bring the measurement that flipped it, not the anecdote: backlog age at the ninety-ninth percentile per subscription, the share of subscriptions where one slow endpoint gated an otherwise healthy queue, and the delivery throughput lost to serialisation. A reversal justified by complaints is indistinguishable from a reversal justified by fatigue.
3. Name what you learned about the guarantee itself, which is the engineering content of this story. At-least-once delivery means a retried event already arrives after newer ones and the consumer already must be idempotent, so a guarantee the customer has to defend against anyway was never worth what it cost to provide.
4. Describe the migration, because reversing a published contract is the hard half and the part candidates skip. Parallel attempts behind a per-subscription flag, a monotonically increasing sequence number added to the envelope so order-sensitive consumers can sort or discard, documentation that states at-least-once and unordered in those words, and a deprecation measured in quarters because the client is a pinned SDK inside a build pipeline you cannot see or redeploy.
5. Give the cost in the two currencies that matter: engineer-weeks, and how many customers had to change code. Then say who you told before it shipped rather than in a changelog afterwards, and which large customer you left on the old behaviour and for how long.
6. Close with the signal you now weight differently, stated as something you would do earlier next time: measuring the blocking cost on the slowest decile of endpoints before committing to the guarantee, rather than after a customer noticed.

Follow-up

• A customer insists they need ordering. What do you offer them that is not global serialisation?
• How did you choose the deprecation window given that you cannot see or redeploy the clients?
• What would have to be true for you to reverse back?