Crusoe Software Engineer Interview Guide

The 60-second brief

Crusoe builds vertically integrated AI infrastructure, from power down to token delivery, and runs Crusoe Cloud. Software Engineers, as the role is described, write the control planes, storage engines, telemetry pipelines and software-defined networking behind large GPU clusters. Named examples include the Crusoe Cloud container registry, eBPF and DPDK packet processing, and IAM control planes.

This guide covers the three rounds candidates report for Crusoe's Software Engineer role: the initial screening call, the technical screen and the final evaluation. The worked exercises on online SQL migration, rate limiting and webhook fan-out are original drills. They use the same infrastructure skills but are not reported Crusoe questions. The seven-day plan moves from interval and tree coding to infrastructure design and then to your project deck.

Focus areas

• Keep money in integer minor units
• Bound blast radius with per-tenant concurrency limits
• Evolve APIs without breaking pinned SDK clients

Overview

The described responsibilities mix building services, operating them and profiling them. That covers cutting latency and resource overhead on compute nodes, building in observability with tools such as Prometheus, Grafana, Vector or OpenTelemetry, taking on-call shifts and running incident post-mortems, writing design docs, and working with hardware operations, site reliability, network engineering and product teams. For preparation, be ready to discuss sharding, replication, consensus and consistency models with concrete examples, to talk through Linux, Docker and Kubernetes work you have done, and to code in a systems language you can debug quickly.

In practice, this means the interview looks more like infrastructure engineering than puzzle solving. The reported coding prompts are applied: intervals across streams, aligned tree printing and a staged matrix validator. Most reported design prompts are infrastructure systems (telemetry ingestion, a multi-tenant container registry, a storage control plane), alongside a Slack-like chat service. The final evaluation reportedly includes a project review that you present yourself. Spend your time on those three areas, not on a broad algorithm list.

The Crusoe interview process

Crusoe candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening Call

Candidates report that this first call is with a recruiter or a hiring manager and covers your background, your technical interests and the scope of the role. Reports say that when the hiring manager runs it, the call can go past a résumé summary into direct questions about past architecture decisions, low-level trade-offs and operational problems you handled. Treat it as a short technical conversation. Have one system ready to describe at design-review depth. Use the scope discussion to find out which area the team owns. The described role spans control planes, storage, telemetry pipelines, networking and IAM, and later preparation is easier to aim once you know which one applies.

What to demonstrate

• Whether your background and technical interests fit the scope the team is hiring for
• When a hiring manager runs the call: whether you can explain past architecture decisions and low-level trade-offs directly
• Whether you can describe a concrete operational problem you handled: what failed, how you found it and what you changed

How to prepare

• Pick one system you built and prepare its architecture, one trade-off you would defend and one operational problem, all at design-review depth
• Ask which area the team owns (control plane, storage, telemetry, networking, IAM or frontend) and which language it works in, so you can aim later preparation at the right prompts
• List the systems languages and tools you have used in production, for example Go, C++, Rust, Linux or Kubernetes, and have one concrete example ready for each

Technical Screen

Candidates report either practical coding, such as interval manipulation or string and tree visualization, or a preliminary system design discussion. Ask early which format you will get. The reports do not say which specific questions come from this round, so prepare the reported coding questions as a set: they are applied rather than puzzle-like and centre on intervals, tree rendering and staged validation. In a coding version, most of the risk is in edge cases (touching endpoints, unsorted input, empty streams, an unbalanced tree) and in keeping the code readable as requirements are added. Talk through the invariant and the tests while you write. If it turns into a design discussion, have a short outline of one infrastructure system ready.

What to demonstrate

• Whether your interval code handles unsorted input, overlapping and touching endpoints, and multiple streams without an off-by-one error
• Whether a tree rendering derives row width from the tree's height, so parents stay centred over their children and empty slots are marked
• Whether your code stays clean when the interviewer adds or changes a requirement partway through
• Whether you state complexity and edge cases while writing, rather than after being asked

How to prepare

• Write merge intervals, insert interval and a k-way heap merge of sorted interval streams from a blank file, with tests for touching endpoints, containment and empty input
• Write an aligned tree printer: compute the height h, use a row width of 2^h - 1 slots, put each node at the midpoint of its range and recurse into each half
• Build a Sudoku-style validator as separate row, column and box checks, then add a new rule as a new check without editing the old ones
• Prepare a short outline of a telemetry ingestion path in case the screen turns into a preliminary design discussion

Final Evaluation

Candidates report a final evaluation held virtually or onsite. It combines practical system design, a technical project review that often requires a prepared slide deck or structured presentation, and behavioral conversations with engineering leaders. The reports do not tie specific design questions to this round, and design can also come up in the technical screen, so prepare the reported design questions for either discussion. For the project review, expect follow-up questions on your own choices rather than a summary. For design, answers are stronger when replication, failure recovery, backpressure and observability are part of the design itself rather than a closing remark.

What to demonstrate

• Whether your design names its partitioning, replication and consistency choices and what each one costs
• Whether failure handling and observability (metrics, logs, recovery paths) are built into the design rather than listed at the end
• Whether you can defend a past project's data-store choices and scaling bottlenecks, and say what you would redesign today, under follow-up questions
• Whether your behavioral answers show a decision you owned under ambiguity, and how you explained an architecture to people outside software engineering

How to prepare

• Build a deck for one production system with a topology diagram, the data flow, the reason for each data store, the bottleneck you hit, one incident and what you would redesign
• Take each reported design question end to end. Each time, state the read and write paths, the partition key, and what happens when a node or region fails
• For a container registry design, work out how layers are cached or deduplicated so parallel pulls from thousands of nodes do not saturate the origin
• Prepare stories for ambiguity with little documentation, explaining an architecture to non-engineers, and technical debt versus feature work under a deadline

Where candidates lose points

Merging intervals from one sorted list when the prompt has several out-of-order streams

The reported coding prompts involve multiple incoming streams and out-of-order events. Ask whether each stream is sorted and whether endpoints are closed or half-open, because that decides whether [1,3] and [3,5] merge. If each stream is sorted, a k-way merge with a heap keeps the combined order in O(N log k). If they are not sorted, sort first and say so. Before you call it done, test empty streams, a single interval, full containment and touching endpoints.

Printing the tree level by level without fixing the grid width first

A plain BFS gives you the levels but not the alignment. Compute the height, set each row's width from it (2^h - 1 slots), and put every node at the midpoint of the slot range it owns. Write * where a node is missing so children stay under their parent. Ask whether the children of a missing node should also print as *, and pad every cell to the same width so multi-character values do not break the alignment.

Hard-coding the first stage of a progressive problem so the next stage forces a rewrite

The reported matrix validator changes its rules at each stage. Write each rule as its own check function over the grid and run them from a list. A new stage then adds a check instead of editing a nested loop you wrote under pressure. Name the structure out loud before stage one so the interviewer can see why stage two goes quickly.

Walking into the project review with a feature tour instead of a design review

Candidates report being asked to present a past project, often with slides. Build the deck around decisions. Cover why you chose each data store over the alternatives, the scaling bottleneck and how you measured it, how the system behaved under upstream latency or node crashes, and what you would redesign today. Rehearse it with someone who interrupts with 'why not X?', because the follow-up questions are where the review actually happens.

Designing a telemetry pipeline, registry or storage control plane without failure recovery or observability

Most reported design prompts are infrastructure serving many nodes or tenants: the telemetry pipeline, the container registry and the storage control plane. For each one, say what happens when a node, a replica or a downstream store fails, where backpressure applies, and which metrics and alerts would tell you the system is degrading. For the telemetry prompt, also cover how high-cardinality series are bounded. For the registry, cover how concurrent pulls avoid saturating bandwidth.

Questions & practice

Solve a progressive, multi-stage matrix validator (e.g., custom state checks or Sudoku solver variants) with evolving rules at each stage.

Medium · data structures and algorithms

Approach

1. Choose the data structure from the access pattern, not from familiarity.
2. Walk one small example through your approach before writing the whole thing.
3. Restate the input: its shape, its size, and what is guaranteed about it.

Follow-up

• Which test case would catch an off-by-one here?
• How does this change if the input no longer fits in memory?

Implement an interval insertion and merging utility designed to handle out-of-order data event streams.

Medium · data structures and algorithms

Approach

1. Name the brute-force solution and its complexity before improving on it.
2. Restate the input: its shape, its size, and what is guaranteed about it.
3. Walk one small example through your approach before writing the whole thing.

Follow-up

• Which test case would catch an off-by-one here?
• How does this change if the input no longer fits in memory?

Print a binary tree level by level into a formatted grid structure, replacing missing nodes with an asterisk (*) and maintaining visual alignment between parent and child nodes.

Medium · data structures and algorithms

Approach

1. Name the brute-force solution and its complexity before improving on it.
2. Choose the data structure from the access pattern, not from familiarity.
3. Walk one small example through your approach before writing the whole thing.

Follow-up

• How does this change if the input no longer fits in memory?
• Which test case would catch an off-by-one here?

Implement a multi-stream log parser that handles overlapping interval ranges and merges data windows efficiently across incoming streams.

Medium · data structures and algorithms

Approach

1. Choose the data structure from the access pattern, not from familiarity.
2. Name the brute-force solution and its complexity before improving on it.
3. Restate the input: its shape, its size, and what is guaranteed about it.

Follow-up

• How does this change if the input no longer fits in memory?
• What is the worst case, and how likely is it on real data?

Hold a tenant to a trailing sixty-second request limit

Medium · sliding-window · two-pointer · rate-limiting · tenant-skew

The gateway must hold each tenant to R requests in any trailing 60 seconds, in aggregate across three regions and every pod, within a budget of under 10 ms added p99. Peak is 30,000 requests/second across 200,000 active tenants, and traffic is heavily skewed toward a handful of them. Give an exact single-process algorithm with its amortised per-request cost and its memory per tenant, then a bounded-memory approximation and the worst-case overshoot it actually admits. Say what the distributed version does when the counter store is unreachable.

Approach

1. Exact, single process: a per-tenant deque of request timestamps. On arrival, pop from the front while front <= now - 60s, then admit if the remaining length is below R and push. Each timestamp is pushed once and popped once, so the cost is O(1) amortised. The O(R) version is the one that re-filters the whole deque on every request.
2. Quote the memory. R = 1,000 across 200,000 active tenants is up to 2 x 10^8 timestamps at 8 bytes, about 1.6 GB, and that is the worst case rather than the mean, because the long tail of small tenants holds almost nothing. Skew helps you here and hurts you in the sharding decision.
3. Bounded alternative, with its real bound stated: a fixed 60-second counter is O(1) memory but admits close to 2R across a 60-second span straddling a boundary. The weighted two-bucket estimate, prev * (60 - elapsed)/60 + cur, is better on smooth traffic but assumes the previous window's arrivals were uniform; an adversary packing them at the end of that window is undercounted and can still approach 2R. Say that rather than calling it exact.
4. Token bucket is the usual gateway answer and a different contract: O(1) state per tenant (tokens, last_refill), a sustained rate, and a deliberate burst allowance equal to the bucket size. Choose it when a burst is acceptable and the log when the limit is contractual.
5. Distributed: the limit is per tenant in aggregate, so a local bucket of R/N per pod is wrong in both directions under skew. A tenant landing on one pod is throttled at R/N, and a tenant spread evenly across pods exceeds R. The shared check must be a single atomic round trip, one script or one increment-and-compare, never read-then-write, and it must fit inside the 10 ms p99 budget.
6. Decide the unavailable case in advance and write it down. Failing open keeps the product up and lets a tenant exceed its limit for the duration; failing closed converts a counter-store outage into a full outage. Most gateways fail open on rate limits and closed on authorisation, and those are two separate decisions made separately.

Worked solution (25 min)

1. Implement the deque version and instrument the per-request pop count, then confirm total pops equal total pushes over a run.
2. Generate a burst that places R requests in the last 100 ms of one minute and R more in the first 100 ms of the next.
3. Run that burst through the exact deque, a fixed 60-second counter, and the weighted two-bucket estimate, recording admissions in the trailing 60 seconds at every instant.
4. Size the memory as R x active tenants x 8 bytes at R = 1,000 and 200,000 tenants, and compare it against what a token bucket would need.

Expected result: The exact deque never admits more than R in any trailing 60-second window. The fixed counter admits close to 2R across the boundary. The weighted estimate lands between the two on this burst and approaches 2R once the previous window's requests are packed at its end.

Verify it yourself

• Total pops equal total pushes, which is what makes the amortised O(1) claim true rather than merely asserted.
• Sliding the whole burst by 30 seconds changes the fixed counter's result and leaves the deque's unchanged.
• Running two processes each limiting to R/2, against a tenant that sends everything to one of them, shows that tenant throttled at half its contractual limit.

Follow-up

• One tenant sends 40% of all traffic. What does that do to a single counter key, and what do you shard on instead?
• Quotas rather than rate limits: the check is select used; if used < limit then insert. Name the isolation level that still permits the overshoot, and the two fixes.
• How do you return an accurate Retry-After from the exact algorithm without a second scan?

Enforce a concurrent-run quota that survives simultaneous requests

Medium · isolation levels · write skew · quotas · lock contention

A plan allows at most 20 concurrently running rows in job_run per tenant. The table holds run_id, tenant_id, workspace_id, status (queued, leased, running, succeeded, failed, timed_out, cancelled, lost), lease_token, leased_until, started_at and finished_at. Today the service runs select count(*) from job_run where tenant_id = $1 and status = 'running', compares the result to 20, then inserts. Under load a tenant exceeds the cap by exactly the number of concurrent requests. Name the anomaly, say which isolation levels do and do not prevent it, and give a version that holds, as SQL.

Approach

1. Name it: write skew. Each transaction reads a predicate (the count of running rows), neither modifies what the other read, and both then insert rows that jointly violate an invariant no single row expresses. Read committed permits it. So does repeatable read, because snapshot isolation's first-updater-wins check fires only on conflicting row updates, and these are inserts touching disjoint rows.
2. Enumerate the fixes with their real costs. SERIALIZABLE works: PostgreSQL's SSI tracks the predicate read and aborts one transaction with SQLSTATE 40001, which obliges the caller to retry and makes the abort rate rise with contention on a hot tenant. Folding the predicate into the write as insert ... select ... where (select count(*) ...) < 20 narrows the race to the statement's snapshot but does not close it under read committed.
3. Give the version that holds at read committed: serialise on a row both transactions must touch. update tenant_concurrency set running = running + 1 where tenant_id = $1 and running < 20 returning running updates zero rows when the cap is reached, and zero rows is the rejection. This works because at read committed a blocked UPDATE re-evaluates its WHERE clause against the newly committed row; at repeatable read the same statement raises a serialisation error instead, so the isolation level changes the calling contract.
4. State the cost you just bought. That row is now a per-tenant serialisation point, so admission throughput for the tenant is bounded by one divided by the lock hold time; at a 2 ms hold that is roughly 500 admissions/second. Keep the critical section to the single UPDATE, with no network call or scheduling decision inside the transaction, and decrement in the same transaction that writes the terminal status.
5. Close the leak the status enum implies: a run can end as lost, so a crashed worker otherwise consumes a slot forever. Reconcile on a schedule against status = 'running' and leased_until < now(), and treat the counter as a fast path over job_run, which stays the system of record.

Follow-up

• Write the retry loop for the SERIALIZABLE version. What does the caller see when it keeps aborting, and what bounds the retries?
• Two regions each keep a counter. What is the effective cap, and what does admission do when the counter store is unreachable?
• The cap changes mid-flight on a plan upgrade. Do running jobs get killed, and what does the counter row look like during the change?

Migrate a live partitioned event table without blocking ingest

Hard · online migration · ddl locks · expand and contract · partitioning

usage_event is range-partitioned daily on ingested_at, holds roughly 250M rows per day across 400 live partitions, and is written at 10-40k rows/second. Two changes are required: quantity must move from double precision to numeric(20,6), and a new environment column must become NOT NULL with a default of 'production'. Ingest cannot stop. Give the ordered plan, naming for each step the lock it takes, what that lock blocks, and roughly how long it is held. Identify the one step that cannot be rolled back cleanly once traffic depends on it.

Approach

1. Classify the two changes before planning anything. Adding a column with a non-volatile default has been metadata-only since PostgreSQL 11, so it is cheap. Changing double precision to numeric is not binary-coercible, so alter column ... type rewrites every partition under ACCESS EXCLUSIVE and rebuilds its indexes; on this volume that is hours of blocked ingest and is simply not an option, which is why the plan is expand-and-contract rather than one statement.
2. Expand: add quantity_numeric numeric(20,6) and environment with its default on the parent. Both are catalogue-only but both take a brief ACCESS EXCLUSIVE that cascades to partitions, so run each with lock_timeout set to a second or two and retry on failure. A queued ACCESS EXCLUSIVE request blocks every reader behind it, which is how a metadata-only change turns into an outage.
3. Dual-write: deploy producer code that populates both columns on every insert, and leave it running before anything reads the new column. This is the step that cannot be reverted cleanly. Once readers depend on quantity_numeric, reverting the writer leaves rows with a null there, and the gap is only discoverable by re-reading the old column, which the readers have stopped doing.
4. Backfill older partitions in batches keyed on the primary key, oldest first, committing every few thousand rows with a pause between batches, and skipping the partition still receiving writes until it rotates. Each batch is an ordinary UPDATE taking row locks only. The cost is bloat and WAL rather than blocking, so watch dead tuples and let autovacuum keep pace instead of wrapping 400 partitions in one transaction.
5. Make NOT NULL cheap with the three-step form: add constraint ... check (environment is not null) not valid (brief ACCESS EXCLUSIVE, no scan), then validate constraint (SHARE UPDATE EXCLUSIVE, scans while reads and writes continue), then set not null, which from PostgreSQL 12 uses the validated check and skips its own full scan. Do this per partition, then on the parent.
6. Switch and contract: move reads to the new column behind a flag, verify over a full period that both columns agree on freshly written rows, drop the old column (metadata-only), and only then remove the dual-write. Any index on the new column goes on with CREATE INDEX CONCURRENTLY per partition, since CIC is not supported on a partitioned parent: create the parent index with ONLY, build each child concurrently, then ALTER INDEX ... ATTACH PARTITION until the parent index becomes valid.

Worked solution (45 min)

1. On a scratch cluster, build 10 partitions of 2M rows each and run a writer at a few thousand inserts/second.
2. Run the naive type change and measure how long writes stall and how far ingest lag grows before killing it.
3. Run the expand step with lock_timeout = '2s' while the writer runs, and observe a clean lock timeout and retry instead of a pile-up of blocked readers.
4. Backfill in 5k-row batches and chart dead tuples and WAL generated per batch.
5. Run the not-valid, validate, set-not-null sequence and confirm from pg_stat_activity and timings that nothing held an exclusive lock through a full scan.
6. Add an index with CIC per partition plus ATTACH PARTITION and confirm the parent index reports valid only after the last attach.

Expected result: An ordered expand, dual-write, backfill, switch, contract plan with the lock named per step: metadata-only ADD COLUMN under a brief ACCESS EXCLUSIVE taken with lock_timeout; batched UPDATE backfill under row locks; CHECK NOT VALID, then VALIDATE under SHARE UPDATE EXCLUSIVE, then SET NOT NULL; CREATE INDEX CONCURRENTLY per partition with ATTACH PARTITION; and the dual-write-to-read switch identified as the step that cannot be reverted cleanly.

Verify it yourself

• No step in the plan holds ACCESS EXCLUSIVE for longer than the configured lock_timeout.
• The writer's error rate stays at zero through the rehearsal apart from the deliberate lock timeout.
• After VALIDATE, SET NOT NULL completes in milliseconds on a table where the naive form scanned every row.
• Freshly written rows agree on both columns for a full verification period before the old column is dropped.

Follow-up

• A CREATE INDEX CONCURRENTLY fails halfway through the partition list. What state is the table in, how do you detect it, and what do you run?
• The producer computes quantity itself. What happens to a request already in flight when the dual-write deploy lands, and does it matter?
• Give two queries that prove the backfill is complete: one cheap enough to run every minute, one authoritative.

Prepare a slide deck or technical walkthrough of a past project: What were the critical scaling bottlenecks, how did you choose the underlying data stores, and what would you redesign today?

Medium · system design and architecture

Approach

1. State the consistency you need, and where you are willing to be stale.
2. Choose a partition key and say what query it makes expensive.
3. Fix the scope first: who calls this, how often, and what they do when it fails.

Follow-up

• How does this behave when that dependency is down for an hour?
• What breaks first when traffic grows ten times?

How do you measure, profile, and isolate performance bottlenecks when your application interacts directly with Linux kernel components or hardware accelerators?

Medium · system design and architecture

Approach

1. Name the read and write paths separately; they rarely have the same bottleneck.
2. Name the failure you are designing for, then the recovery path.
3. Choose a partition key and say what query it makes expensive.

Follow-up

• What would you drop to keep the system up under load?
• How does this behave when that dependency is down for an hour?

Build a live dashboard component using React and TypeScript that inges…

Medium

Build a live dashboard component using React and TypeScript that ingests a real-time data endpoint for a transport system (e.g., train station arrival board) and displays status parameters cleanly under high-frequency updates.

Approach

1. State your assumptions explicitly before working the problem.
2. Work from the requirement backwards to the design.
3. Clarify what is being asked and what a complete answer contains.

Follow-up

• What assumption would you test first?
• How would you know your answer was wrong?

Webhook fan-out with per-endpoint isolation and backoff

Medium · fan-out · backpressure · circuit breaker · ordering

One domain event fans out to every matching subscription, producing a webhook_delivery row per (subscription_id, event_id, redelivery_seq). Peak unique event rate is 20k/second; attempts run five to ten times that once fan-out and retries are counted. One customer endpoint has returned 503 for six hours and its backlog holds days of events; every other customer must be unaffected. Design the delivery system: how a worker claims work, the backoff schedule, the per-endpoint circuit breaker, the queue partitioning, and whether you offer ordering per subscription. State the delivery guarantee in one sentence.

Approach

1. State the guarantee first, because it determines the rest: at-least-once with a stable event_id, and the consumer documented as responsible for idempotency. Exactly-once over HTTP is not deliverable - the 200 can be lost after the customer has already committed - so any design that promises it is either lying or is really offering at-most-once.
2. Partition work per subscription rather than into one global pool, with a concurrency cap per subscription. With a shared pool, the endpoint that has been dead for six hours consumes workers on retries that will fail, and every other customer's delivery latency rises: head-of-line blocking across tenants is the exact failure being designed against here.
3. Claim by compare-and-set with a fencing token: UPDATE webhook_delivery SET status = 'in_flight', lease_token = $new, leased_until = now() + interval '60 seconds' WHERE delivery_id = $1 AND status IN ('pending','failed_retryable') AND (leased_until IS NULL OR leased_until < now()), and make the terminal write carry AND lease_token = $new so a paused worker's late write is rejected rather than overwriting a newer attempt. Find due work through the partial index on next_attempt_at WHERE status IN ('pending','failed_retryable'), so the scan is proportional to live rows rather than to the terminal rows that outnumber them by orders of magnitude.
4. Use full jitter: sleep uniformly in [0, min(cap, base x 2^(attempt-1))]. Plain exponential backoff hands a recovering endpoint its entire backlog as one synchronised herd and knocks it over again; full jitter de-correlates it. Then check the schedule actually spans the retention you promise - with base 1 s and a 3,600 s cap, twenty attempts have an expected total elapsed time of only about 4.6 hours, so a twenty-four-hour promise needs roughly fifty-nine attempts or a larger cap.
5. Trip a circuit per endpoint on consecutive failures or a failure ratio over a rolling window: stop dispatching, push next_attempt_at out or mark new deliveries dropped_circuit_open, and half-open with exactly one probe rather than a batch. Bound the backlog explicitly with a per-subscription cap or retention, and decide in advance whether a recovered endpoint receives six hours of events at full rate or a pointer telling it to fetch what it missed.
6. Offer ordering only as an opt-in mode of one in-flight attempt per subscription, and price it honestly: with parallel attempts a retried event overtakes a newer one, so ordering requires serialisation, and serialisation means one slow endpoint blocks its own queue entirely. That converts a shared problem into that customer's own problem, which is the right place for it, but it is still a real cost.

Worked solution (35 min)

1. Compute the attempt rate: 20k unique events/second x mean fan-out x retry multiplier, and size worker pools and the per-subscription concurrency cap from it.
2. Write the claim statement and the terminal write, and point at the clause that rejects a resumed worker's stale write.
3. Tabulate the backoff for attempts 1 to 20 with base 1 s and cap 3,600 s, take the expected value of each full-jitter sleep as half its ceiling, and sum to get total expected elapsed coverage.
4. Decide the policy for a subscription down six hours: events buffered, bytes held, and what the customer actually receives when it returns.

Expected result: At-least-once with a stable event_id; per-subscription lanes with a concurrency cap; lease-and-fence claiming; full jitter whose schedule matches the retention promised - with base 1 s and a one-hour cap, twenty attempts covers only about 4.6 hours of expected elapsed time, so twenty-four hours needs roughly fifty-nine attempts or a larger cap; a per-endpoint breaker with a single half-open probe; ordering offered only as opt-in single-in-flight with head-of-line blocking named as its price.

Verify it yourself

• Sum the expected sleeps in your schedule and compare against the retention window you promised the customer. If the two numbers disagree, one of them is wrong and it is usually the promise.
• Execute the terminal write with a stale lease_token and confirm it affects zero rows. If it succeeds, the lease was doing the work alone and the token is decorative.
• The dead endpoint's backlog must not be visible in any queue another subscription's workers poll. If one index scan serves all subscriptions, the isolation is nominal.

Follow-up

• The endpoint recovers. Does it receive six hours of events at full rate, and what does that do to it?
• Trace the exact code path by which an event belonging to one tenant could be signed and sent to another tenant's endpoint.
• A customer insists they never received an event your row marks delivered. What evidence do you have, and what does payload_digest let you prove?

Hourly rollups merge one hour and lose another

Easy · timezone · rollups · reconciliation

Reconciliation flags one tenant on one day. Summing usage_event.quantity by hour of occurred_at gives 24 non-empty hours, but usage_rollup_hourly holds 23 rows for that tenant, workspace and SKU, one of which carries roughly the sum of two adjacent hours. Other days reconcile exactly, and the affected date matches a civil-time transition. hour_start is documented as truncated to the hour in UTC. You have both tables, the rollup job source, and its runtime environment. Give an ordered checklist, the mechanism, and the correction path for a day that may already be sealed.

Approach

1. Bisect by dimension until one cell explains the whole difference: tenant, then day, then SKU, then hour. A defect confined to a single transition date already rules out deduplication and late arrival, both of which are indifferent to which hour an event lands in.
2. Read the truncation with its precondition stated: date_trunc on a timestamptz value is evaluated in the session TimeZone, not in UTC. If the job connects without pinning that setting, it inherits the server or container default.
3. Follow that to the collision: in a zone that observes daylight saving, two distinct UTC hours map to the same local wall-clock label at the autumn transition, so both fold into one key under the unique constraint on (tenant_id, workspace_id, sku, hour_start) and their quantities sum into one row. At the spring transition a label never occurs and the row is simply absent.
4. Confirm from data rather than from reading code: run the same aggregate twice, once with the session pinned to UTC and once with the job host zone, and check that the second reproduces the stored rollup exactly.
5. Fix at the source by pinning the connection to UTC explicitly, or by truncating on occurred_at AT TIME ZONE 'UTC', rather than relying on a default that differs between a developer machine, CI and production.
6. Correct according to status, not convenience: an open hour is recomputed with revision incremented, a sealed hour is frozen and the difference becomes an adjustment line on the next invoice with voided_by_line_id pointing at the line it reverses.

Follow-up

• The same job also emits a daily figure for a dashboard. Why can a correct hourly rollup still produce a wrong day, and what does the tenant's billing timezone have to do with it?
• How would you detect this class automatically rather than waiting for reconciliation, given that it only manifests twice a year per zone?

Suggested practice checklist

Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.

Day 1 — Intervals across streams

• Write merge intervals and insert interval from a blank file, then add tests for touching endpoints, full containment, a single interval and empty input.
• Extend the merge to k sorted streams with a heap, then to unsorted, out-of-order arrivals, and state each version's complexity aloud.
• Work the reported multi-stream log parser with overlapping interval ranges and the reported interval insertion utility for out-of-order event streams under a timer, explaining the invariant as you write.

Deliverable: Tested merge, insert and k-way merge implementations, plus a list of the edge cases you missed on the first attempt.

Day 2 — Tree rendering and the staged matrix validator

• Solve the reported aligned tree printer (bank: Print a binary tree as aligned text): compute the height, fix the row width, place nodes at midpoints and fill missing slots with *.
• Test it on a skewed tree, a full tree and multi-character values, and decide how the children of a missing node are drawn.
• Build the reported multi-stage matrix validator as separate row, column and box checks, then add a second-stage rule without editing the first-stage code.

Deliverable: A tree printer that passes skewed and full-tree cases, and a validator where a new rule is a new function rather than an edit.

Day 3 — Stateful practical coding and the frontend prompt

• Work the trailing sixty-second rate-limit exercise, comparing the deque, fixed-window and token-bucket versions and the overshoot each admits.
• Solve the bank's Calculate charge with a single price override and Implement Interval Overrides and Top-K Strings, and note where the interval logic from day 1 comes back.
• If the team covers frontend, sketch the reported React and TypeScript arrival-board component: where the streamed state lives and how you keep re-renders down under frequent updates.

Deliverable: Working rate limiter and override solutions with stated complexity, plus a one-page component sketch if frontend applies.

Day 4 — Chat and telemetry system design

• Design the Slack-like chat and file-sharing service (bank: Design a Slack-Like Messaging Platform): delivery, durable history, search, attachment processing and presence.
• Design the telemetry ingestion pipeline for high-cardinality metrics across GPU nodes, then check it twice: where batch processing would beat streaming, and how the pipeline itself fails (dropped samples, growing backlog, cardinality blow-up) and how you would detect each.
• Work the webhook fan-out exercise to practise per-tenant isolation, jittered backoff and a delivery guarantee stated in one sentence.

Deliverable: Two design write-ups, each naming the partition key, the replication choice, the failure path and the metrics you would alert on.

Day 5 — Registry, storage control plane and the data layer

• Design the multi-tenant container registry for parallel image pulls: layer caching and deduplication, tenant isolation, and behaviour when the origin is slow.
• Walk through a distributed storage control plane: separating control from data plane, replication, failure recovery and the consistency guarantee you offer.
• Work the concurrent-run quota drill and the online migration exercise to practise the database parts of these designs.

Deliverable: A registry and a storage control-plane design, each with a failure scenario traced end to end, plus your quota and migration answers.

Day 6 — Project deep-dive deck

• Choose one production system and build the deck the reported project walkthrough asks for: topology, data flow, data-store choices against the alternatives, the scaling bottleneck and what you would redesign today.
• Add an incident slide covering how the system handled upstream latency or node crashes and how you isolated the cause, and prepare the reported question on profiling an application that interacts with Linux kernel components or hardware accelerators.
• Present it aloud to someone who interrupts with 'why not X?' and write down every question you could not answer cleanly.

Deliverable: A finished deck plus a list of the follow-up questions you had to fix, each with its revised answer.

Day 7 — Behavioral stories and a final-evaluation mock

• Write stories for the reported behavioral prompts: an ambiguous problem with little documentation, explaining an architecture to non-engineers, technical debt versus features under a deadline, and yourself beyond the résumé.
• Use the metered-usage incident drill as a model for your own incident story, and walk through the hourly-rollup timezone debugging checklist aloud.
• Run one mock that combines a design prompt from days 4-5, the day-6 deck and two behavioral stories, then fix the weakest segment.

Deliverable: Four behavioral stories, each with a decision you owned and its outcome, plus notes from the combined mock.

Scope is read from what you decided

Candidates report behavioral conversations with engineering leaders as part of the final evaluation. The reported prompts cover ambiguity, cross-functional communication, technical debt under a deadline, and production incidents. Choose stories where you made the decision, give each one a concrete outcome, and name one thing you would change.

1. Tell me about a time you had to solve a highly ambiguous technical problem with little documentation or prior precedent.
2. How do you approach prioritizing technical debt versus shipping critical platform features when under aggressive deadlines?
3. Describe yourself beyond what is listed on your resume: what drives your approach to engineering quality and teamwork?
4. Explain how your past system handled unexpected upstream network latency or node crashes during a major production incident.
5. How would you prioritize observability improvements across several engineering teams?

Describe a situation where you had to convey a complex technical architecture to cross-functional stakeholders outside of software engineering.

Medium · behavioural and engineering judgement

Approach

1. Pick a story where you made the decision, not one where you watched it.
2. Give the blast radius: what could have broken, and what you measured.
3. State the situation in two sentences and spend the rest on the reasoning.

Follow-up

• What would you do differently if you ran that again?
• What did you decide not to do, and why?

Describe yourself beyond what is listed on your resume—what drives your approach to engineering quality and teamwork?

Medium · behavioural and engineering judgement

Approach

1. Pick a story where you made the decision, not one where you watched it.
2. State the situation in two sentences and spend the rest on the reasoning.
3. Give the blast radius: what could have broken, and what you measured.

Follow-up

• What would you do differently if you ran that again?
• How did you know your change caused the improvement?

Own the incident where invoices undercounted metered usage

Medium · incident response · at-least-once · billing correction · postmortem

A metering consumer acknowledged each batch before committing the fold into usage_rollup_hourly. A rolling deploy restarted consumers mid-batch for two hours; roughly 1.4M usage_event rows were acknowledged and never folded, and 61 invoices sealed against the resulting rollups before anyone noticed. Take the owner's role. Describe an incident of comparable blast radius you owned: how it surfaced, the query that sized the loss, what you stopped first, and how the money was corrected. Give a wall-clock timeline and one thing you got wrong while it was still live.

Approach

1. Open with the invariant that broke and the direction of the error, because they determine everything else: acknowledging before committing makes the consumer at-most-once, so this loses events rather than duplicating them, and loss raises no error anywhere. A listener who hears 'we lost revenue silently' knows immediately why detection took two hours.
2. Size it with a stated reconciliation rather than an adjective: sum(quantity) from usage_event grouped by (tenant_id, sku, hour of occurred_at) over the window, against usage_rollup_hourly.quantity_sum on the same keys, filtered to environment='production' because staging and sandbox are metered but not billed. Then bisect by hour and tenant until single cells explain the gap. Say how long that ran and whether a replica could serve it while the incident was live.
3. Separate mitigation from fix and say which came first. Mitigation is holding the sealing job, because a sealed row is frozen by design and every minute of sealing converts a recoverable rollup into an invoice correction. The fix is moving the acknowledgement after the commit, which re-introduces duplicates that the dedup check on (tenant_id, idempotency_key) must now absorb.
4. State the correction path in the domain's own terms: sealed periods are never edited, so each affected tenant gets an adjustment line on the next invoice with kind='adjustment' and voided_by_line_id pointing at the line it reverses, priced against the same rate tier and carrying the watermark it priced against. That is four separate numbers — tenants affected, minor units, the cycle the adjustment lands in, and when customers were told.
5. Close on one prevention control with its cost, not five: a per-hour reconciliation comparing raw sum to rollup sum that pages above a threshold. Name the threshold and the false-page rate you accepted, because a detector nobody will keep staffed is not prevention.
6. Name a mistake you made inside the response window — the wrong first hypothesis, a mitigation that made it worse — rather than a design mistake from six months earlier. That is the part candidates rehearse away and interviewers weight heavily.

Follow-up

• Your fix moves the acknowledgement after the commit. What breaks now, and what absorbs it?
• One undercharged tenant has since churned. Do you bill them, and who decides?
• How would you have caught this in ten minutes instead of two hours, and what would that detector cost you in pages per week?