cswork / libraryOpen practice workspace

Interview guides · Nooks

Nooks Software Engineer Interview Guide

Nooks Software Engineer interview guide: reported BFS, integration design and troubleshooting questions, round-by-round prep and a 7-day plan.

The 60-second brief

Nooks builds the AI Sales Assistant Platform (ASAP), which automates busywork for sales teams. The Software Engineer work described in this guide's source material spans backend infrastructure, real-time voice AI, and integrations that keep data in sync with CRMs and sales engagement platforms. That means webhooks, OAuth, retries, and third-party rate limits come up again and again in the reported questions.

This guide covers the three stages candidates report: a recruiter conversation, technical screens (live coding or algorithmic problem-solving), and comprehensive rounds (system design, practical architecture and deep dives into past projects). It also covers the question categories reported for the role: graph traversal and data structure design, integration and real-time system design, practical troubleshooting of call workflows, and behavioral deep dives. Which team the loop is for (core infrastructure, voice AI or core product) changes the emphasis, so confirm that first.

Focus areas

  • Evolve APIs without breaking pinned SDK clients
  • Make every write idempotent under client retries
  • Build at-least-once pipelines with explicit deduplication horizons

Overview

Nooks builds the AI Sales Assistant Platform (ASAP) for sales teams. In the source material, the Software Engineer role covers backend infrastructure, real-time voice AI pipelines, and integrations that move tasks, prospects and account data between the product and external CRMs and sales engagement platforms. The engineering problems are the ones those integrations create: webhooks from systems you do not control, OAuth tokens that expire, strict external rate limits, retries, and keeping data consistent when payloads arrive late or out of order.

The reported questions follow the same themes. On the coding side: implementing BFS and explaining why it is slow in certain cases, speeding up a BFS with parallelization, and building a REST endpoint around a data structure. On the design side: an integration framework that handles billions of data points under retries and rate limits, monitoring and error handling for external webhooks, and configuring a high-volume dialing robot for profit and throughput. Candidates also report practical troubleshooting scenarios built around recorded support calls and simulated call-resolution workflows.

Use this guide in two layers. The reported questions and the question bank show which topics to cover. The practice questions on this page are original drills on nearby themes (webhook signature verification, usage rollups, API versioning, delivery guarantees). They give you depth on the integration and pipeline material, and three of them have worked exercises you can check your answers against.

The Nooks interview process

Nooks candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Conversation

Candidates describe this as an initial conversation to align on background, interest and logistics. Lead with the project closest to this role's reported work (an integration with a third-party API, a data pipeline with retries, or a latency-sensitive service) so the conversation starts from your strongest fit. Use the call to collect what you need for planning. The source material notes that emphasis varies across core infrastructure, voice AI and core product teams, so ask which team the loop is for and what format the coding stage uses.

What to demonstrate

  • Background fit: whether your recent work maps onto backend services, third-party integrations, data pipelines or real-time systems
  • Interest: whether you can say concretely why this product and this kind of engineering work, rather than giving a generic answer
  • Logistics: your availability and which team the loop is for

How to prepare

  • Write a two-sentence version of each headline project with no internal codenames: the constraint, what you changed, what happened. Say it to someone outside engineering and fix whatever they repeat back wrong
  • Prepare a specific reason for applying that is tied to the product: an assistant for sales teams that depends on CRM sync, dialing and voice
  • Ask which team the loop is for and whether coding is live or take-home, and write the answers down before planning the rest of your preparation

Technical Screens

Candidates describe the technical screens as live coding or algorithmic problem-solving. The reported algorithms and data structures questions for this role centre on graph traversal and data structure design: implement BFS for a toy problem and explain why it is slow in certain scenarios, speed up a BFS with parallelization, and build a REST endpoint that constructs and manages a specific data structure. The question bank adds an O(1) insert/delete/random set, maximum contiguous subarray, a BFS web crawler, and a question on handling misleading coding prompts. Sources do not say which of these appears at which stage, so prepare the whole category for this one. Two of the reported BFS questions ask for more than a working traversal: why it is slow, and how to parallelise it.

What to demonstrate

  • Traversal correctness under live conditions: visited-set handling, queue choice, termination on cycles and disconnected input
  • Complexity reasoning you can defend: why BFS is O(V+E), and which implementation choices make a naive version slower than that
  • Whether you can extend a correct answer, for example by parallelising a traversal frontier or exposing a data structure through REST endpoints with clear semantics
  • How you handle an ambiguous or misleading prompt: the requirements you confirm before writing code

How to prepare

  • Implement BFS, then introduce three slowdowns on purpose (marking visited at dequeue instead of enqueue, popping from the front of a list, rebuilding adjacency per step), time each on a large input, and explain each in one sentence
  • Sketch a level-synchronous parallel BFS: process the frontier in parallel, build the next frontier, and say how workers avoid claiming the same node and when the synchronization cost outweighs the gain
  • Build the O(1) insert/delete/random set (array plus index map, swap-with-last on delete), then define REST endpoints for it, including status codes for a duplicate insert and a missing delete
  • Before coding, restate each prompt with one concrete example and ask about input size, duplicates and edge cases out loud

Comprehensive Rounds

Candidates describe the comprehensive rounds as covering system design, practical architecture, and deep dives into past technical projects. The reported design questions for the role are about integrations and real-time systems: an integration framework for billions of data points with retries and strict rate limits, real-time monitoring and error handling for external webhooks, and configuring a high-volume dialing robot for profit and throughput. The bank adds CRM and dialer sync, at-least-once webhook processing, OAuth tokens across a distributed system, low-latency audio streaming, and observability for voice routing. Candidates also report troubleshooting scenarios (a recorded support call, a simulated call-resolution workflow) and discussions of race conditions, OAuth flows and rate limits. None of these are tied to a specific round in the sources, so prepare them as categories.

What to demonstrate

  • Design under third-party constraints: rate-limit budgets per external account, retries with backoff and jitter, idempotent writes, and webhooks that arrive late, duplicated or out of order
  • Operational reasoning: what you would measure and alert on, and what the system does when an external dependency or a latency budget fails
  • Structured diagnosis in troubleshooting scenarios: hypotheses drawn from the symptom and the evidence that would tell them apart
  • Depth on your own projects: the binding constraint, the alternative you rejected, and what you measured before and after

How to prepare

  • Design a CRM sync end to end: a sync cursor per connected account, a shared rate-limit budget per external account, retries only for transient errors, dead-lettering for permanent ones, idempotency keys, and an OAuth refresh that does not race across workers
  • Work webhooks from both sides: ingestion with at-least-once delivery and your own dedupe, then the outbound contract in the practice question on delivery guarantees and replay
  • For the dialing robot question, write the objective and constraints before any architecture, then name the metric you would watch after a configuration change
  • Rehearse one of your own incidents as a diagnosis: symptom, hypotheses, the cheapest check that split them, the fix, and what now catches it earlier

Where candidates lose points

Writing a correct BFS and having nothing concrete to say when asked why it is slow or how to parallelise it

The reported coding questions ask for both. Know the specific causes. Marking a node visited when you dequeue it instead of when you enqueue it lets the same node enter the queue many times. Popping from the front of a list costs O(n) per pop. Rebuilding neighbours on every step repeats work. And a very wide frontier can dwarf a nearby target, which is where bidirectional search helps. For parallelization, describe level-synchronous frontiers, how workers avoid claiming the same node (atomic test-and-set, or per-worker next-frontiers merged with dedupe), and when the synchronization overhead makes it slower on a small graph.

Designing a CRM integration as direct API calls wrapped in a generic retry

External rate limits are shared across all your workers, so enforce a budget per external account in one place rather than letting each worker find the limit by getting 429s. Retry only transient failures (timeouts, 429, 5xx) with capped exponential backoff and jitter, and honour Retry-After. Send permanent validation failures to a dead-letter queue that someone can see and act on. Make writes idempotent so a retry cannot create a duplicate record, sync incrementally with cursors, and state what happens when both sides changed the same record.

Assuming external webhooks arrive once, promptly and in order

Persist the raw payload before acknowledging, acknowledge quickly and process asynchronously, and dedupe on the sender's event id. For out-of-order payloads, compare a version or source timestamp against what you have stored and discard older updates instead of overwriting newer state. Then name the monitoring: ingestion lag, failure rate per source, dead-letter depth, and an alert that fires when a source goes quiet, not only when it errors.

Naming a likely cause in the call-troubleshooting scenario without a way to confirm it

Candidates report scenarios such as listening to a recorded support call and diagnosing the underlying technical issue. First restate the customer's complaint as a technical symptom. Then list hypotheses by layer (client, audio or telephony path, backend service, CRM integration) and, for each, name the evidence that would confirm or rule it out and ask for it. In an unfamiliar workflow, find out who owns each step before assigning the failure to one of them.

Answering the dialing robot optimization question with infrastructure before defining what is being optimised

The reported prompt asks about optimising profit and throughput. Say what each of those means in the setup you are given, list the levers and the constraints, and only then propose a configuration. Close by saying how you would check that a change helped: which baseline you would compare against, and which guardrail metric would show that more throughput cost you profit.

Questions & practice

Find peak concurrent sandbox usage from run intervals

Medium · sweep-line · intervals · concurrency-caps · null-semantics

Given up to 5 million job_run rows for one tenant over one day, with run_id, started_at, finished_at, status and wall_clock_limit_seconds, report the maximum number of sandboxes running at once, the earliest instant that maximum is reached, and the first run_id that would breach a per-tenant cap of C. started_at is null while a run is queued; finished_at is null both for runs still executing and for runs in status lost. Treat a run as occupying [started_at, finished_at). Give the complexity and state how you handle each null.

Approach

  1. Turn each run into two sweep events, (started_at, +1) and (end, -1), then sort the 2n events by timestamp with -1 ordered before +1 at equal timestamps. That tie-break is what makes the interval half-open, so a run finishing at 10:00:00 and one starting at 10:00:00 never overlap.
  2. Decide each null out loud before sweeping, because each choice moves the answer. A null started_at means queued and contributes nothing. A null finished_at with status running or leased is clipped to the window end. Status lost has no observed end at all, so clip it at started_at + wall_clock_limit_seconds on the grounds that the supervisor owns the timeout, and record that you did. The table's check (finished_at is null or started_at is not null) guarantees you never see an end without a start.
  3. Sweep once, maintaining a running counter, the maximum, and the timestamp at which the maximum was first attained (update peak_at only on a strict increase, or you will report the last such instant instead of the earliest). Capture the first run_id whose +1 takes the counter to C+1 during the same sweep rather than in a second pass.
  4. Complexity: O(n log n) dominated by the sort, O(n) space. If rows already arrive ordered by started_at, a min-heap of end times gives O(n log k) time and O(k) space with k the peak concurrency, which is the better shape when the rows come from an index scan on (tenant_id, started_at).
  5. If second resolution is acceptable, counting-sort the endpoints into an 86,400-slot delta array and prefix-sum it: O(n + T) time and O(T) space, which beats the comparison sort at 5 million rows. It answers only at second granularity, so state which resolution the cap is defined in.

Worked solution (20 min)

  1. Write the null policy as three lines of prose first, one per case, and keep them beside the output.
  2. Emit 2n endpoint tuples (timestamp, delta, run_id) and sort on the key (timestamp, delta) so -1 precedes +1.
  3. Sweep, tracking cur, peak, peak_at updated only on a strict increase, and the first run_id whose +1 takes cur to C+1.
  4. Build a fixture with two runs where one ends exactly when the next starts, three genuinely overlapping runs, one run with a null finished_at and status running, and one with status lost and a 300-second wall_clock_limit_seconds.
  5. Re-run with every timestamp shifted by a constant and confirm the peak is unchanged while peak_at shifts by the same constant.

Expected result: Peak is 3, from the overlapping trio. The exact-handoff pair yields a peak of 1, not 2. peak_at is the start instant of the third overlapping run. The lost run occupies exactly [started_at, started_at + 300s) under the stated policy.

Verify it yourself

  • Setting peak_at on >= rather than > moves the reported instant to the last time the peak was attained. Flip it and watch the fixture change.
  • Sorting by timestamp alone, without the delta tie-break, raises the reported peak on the handoff fixture.
  • The deltas over the whole sweep sum to zero once every open-ended run has been clipped.

Follow-up

  • Now report peak concurrency per tenant for 10,000 tenants from one globally sorted stream. What changes about memory and about the sort?
  • The cap has to be enforced at dispatch rather than reported afterwards. What does the admission check look like, and where does it race?
  • How would you answer 'peak concurrency within any 5-minute window' without re-sorting?

Parse and verify a timestamped multi-signature webhook header

Easy · parsing · hmac · constant-time-compare · replay-protection

An inbound webhook carries a signature header of at most 1 KiB shaped t=<unix seconds>,v1=<64 hex chars>, with up to five v1 values during secret rotation and possibly unknown scheme keys. You hold the raw request body bytes and the currently active signing secrets. Write the parser and the verifier: accept when any active secret reproduces a signature and the timestamp is within a five-minute tolerance in either direction, reject otherwise. Single left-to-right pass over the header, no regular expression. State what is inside the MAC and why.

Approach

  1. Parse in one scan: split on ,, then on the first = only, since a value may itself contain = under a future scheme. Accept t exactly once and treat a second t as a reject rather than last-wins. Push every v1 onto a short list and ignore any other key, so a v2 can be introduced later without breaking this verifier.
  2. Say what is signed: HMAC-SHA256 over the exact byte string <t>.<raw body bytes>, yielding 32 bytes or 64 hex characters. The timestamp sits inside the MAC because otherwise an attacker replays yesterday's body with its still-valid signature and only has to edit the header timestamp.
  3. Hash the bytes as received. Verifying against a re-serialised JSON body is the usual defect: key order, whitespace and number formatting all change the bytes while the parsed objects compare equal, so signatures fail for honest senders and the popular 'fix' is to stop checking.
  4. Compare in constant time over fixed-length digests. Decode the hex to 32 bytes, accumulate acc |= a[i] ^ b[i] across the whole length, and test acc == 0 at the end. Evaluate every candidate without an early exit; at five candidates that is five HMACs over the body, linear in body size and negligible beside the network.
  5. Apply the tolerance as a two-sided bound, rejecting when |now - t| > 300 seconds. A sender whose clock runs ahead of yours is an ordinary case, and an unbounded future timestamp is a free replay window.
  6. Complexity: O(L) over the header producing k candidates, plus k HMACs at O(|body|) each. Space is O(k) beyond the body itself. Do the cheap rejections, including the tolerance check, before any cryptography runs.

Follow-up

  • The body is 40 MB. What changes about where you verify, and what can you do before the whole body has arrived?
  • A customer reports that signatures fail for exactly the requests whose body contains a non-ASCII character. What is your first hypothesis?
  • How do you rotate the signing secret with no failed deliveries, and how long do both secrets stay live?

Fold a deduplicated usage stream into hourly rollups

Easy · aggregation · deduplication · watermarks · exact-arithmetic

You are given one day of usage_event rows, up to 250 million, each carrying event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at and ingested_at. Produce usage_rollup_hourly cells keyed (tenant_id, workspace_id, sku, hour_start) with quantity_sum, event_count and source_max_ingested_at. An event counts once per (tenant_id, idempotency_key). The rollup grain has no environment column, so state your filter. One pass. Give your time and space bounds, and say what the deduplication actually costs in memory.

Approach

  1. Bucket on occurred_at, never ingested_at: hour_start = date_trunc('hour', occurred_at at time zone 'UTC'). The two columns answer different questions. occurred_at says which hour the customer is billed for; ingested_at says how current the fold is. Using the second for the first makes late data invisible instead of correctable.
  2. The fold is trivial and the deduplication is the entire cost, so price it before designing anything clever. An exact set over (tenant_id, idempotency_key) at 250M entries, stored as a 16-byte 128-bit hash in an open-addressed table at 0.7 load factor, needs about 357M slots at 16 bytes each, roughly 5.7 GB. The fix is partitioning by hash(tenant_id) % P so each shard holds 1/P of the set and no tenant's keys straddle shards.
  3. Rule out a Bloom filter as a replacement, in the right direction: a false positive reports 'already seen' for an event never seen, so you drop a real event and lose revenue with no error raised. It is usable only as a negative pre-filter in front of the exact set, where a miss is conclusive and a hit must fall through to the real lookup.
  4. Accumulate in scaled integers, not binary floating point. numeric(20,6) admits values below 10^14, so one event scaled to micro-units can reach 10^20, past int64's 9.22 x 10^18; use a 128-bit or arbitrary-precision accumulator unless you first bound the per-event maximum. binary64 represents integers exactly only to 2^53, about 9.01 x 10^15, and cannot represent 0.1 at all, so two runs that sum in different orders disagree.
  5. Carry source_max_ingested_at = max(ingested_at) over the events folded into each cell, and count event_count over accepted, post-dedup events. Without that watermark there is no way to prove later what a number did and did not include, which is the first question any reconciliation asks.
  6. State the environment filter explicitly, because the rollup grain cannot record it. A fold that quietly includes staging bills non-production traffic; one that quietly excludes it loses a cost signal. Production-only is the billing answer, and either way it belongs in the job name and the output metadata. Complexity: O(n) time, O(distinct dedup keys) space, dominated by the dedup set rather than by the cells.

Follow-up

  • A producer retries at 23:59:59 and the retry lands at 00:00:01. The unique index on the daily-partitioned table must include the partition key. What gets double-counted, and what is the smallest change that fixes it?
  • The consumer acknowledges its batch before committing the fold. Which failure loses revenue now, and which arrangement duplicates instead?
  • What makes a re-run over the same day produce byte-identical rollups?

Migrate a live partitioned event table without blocking ingest

Hard · online migration · ddl locks · expand and contract · partitioning

usage_event is range-partitioned daily on ingested_at, holds roughly 250M rows per day across 400 live partitions, and is written at 10-40k rows/second. Two changes are required: quantity must move from double precision to numeric(20,6), and a new environment column must become NOT NULL with a default of 'production'. Ingest cannot stop. Give the ordered plan, naming for each step the lock it takes, what that lock blocks, and roughly how long it is held. Identify the one step that cannot be rolled back cleanly once traffic depends on it.

Approach

  1. Classify the two changes before planning anything. Adding a column with a non-volatile default has been metadata-only since PostgreSQL 11, so it is cheap. Changing double precision to numeric is not binary-coercible, so alter column ... type rewrites every partition under ACCESS EXCLUSIVE and rebuilds its indexes; on this volume that is hours of blocked ingest and is simply not an option, which is why the plan is expand-and-contract rather than one statement.
  2. Expand: add quantity_numeric numeric(20,6) and environment with its default on the parent. Both are catalogue-only but both take a brief ACCESS EXCLUSIVE that cascades to partitions, so run each with lock_timeout set to a second or two and retry on failure. A queued ACCESS EXCLUSIVE request blocks every reader behind it, which is how a metadata-only change turns into an outage.
  3. Dual-write: deploy producer code that populates both columns on every insert, and leave it running before anything reads the new column. This is the step that cannot be reverted cleanly. Once readers depend on quantity_numeric, reverting the writer leaves rows with a null there, and the gap is only discoverable by re-reading the old column, which the readers have stopped doing.
  4. Backfill older partitions in batches keyed on the primary key, oldest first, committing every few thousand rows with a pause between batches, and skipping the partition still receiving writes until it rotates. Each batch is an ordinary UPDATE taking row locks only. The cost is bloat and WAL rather than blocking, so watch dead tuples and let autovacuum keep pace instead of wrapping 400 partitions in one transaction.
  5. Make NOT NULL cheap with the three-step form: add constraint ... check (environment is not null) not valid (brief ACCESS EXCLUSIVE, no scan), then validate constraint (SHARE UPDATE EXCLUSIVE, scans while reads and writes continue), then set not null, which from PostgreSQL 12 uses the validated check and skips its own full scan. Do this per partition, then on the parent.
  6. Switch and contract: move reads to the new column behind a flag, verify over a full period that both columns agree on freshly written rows, drop the old column (metadata-only), and only then remove the dual-write. Any index on the new column goes on with CREATE INDEX CONCURRENTLY per partition, since CIC is not supported on a partitioned parent: create the parent index with ONLY, build each child concurrently, then ALTER INDEX ... ATTACH PARTITION until the parent index becomes valid.

Worked solution (45 min)

  1. On a scratch cluster, build 10 partitions of 2M rows each and run a writer at a few thousand inserts/second.
  2. Run the naive type change and measure how long writes stall and how far ingest lag grows before killing it.
  3. Run the expand step with lock_timeout = '2s' while the writer runs, and observe a clean lock timeout and retry instead of a pile-up of blocked readers.
  4. Backfill in 5k-row batches and chart dead tuples and WAL generated per batch.
  5. Run the not-valid, validate, set-not-null sequence and confirm from pg_stat_activity and timings that nothing held an exclusive lock through a full scan.
  6. Add an index with CIC per partition plus ATTACH PARTITION and confirm the parent index reports valid only after the last attach.

Expected result: An ordered expand, dual-write, backfill, switch, contract plan with the lock named per step: metadata-only ADD COLUMN under a brief ACCESS EXCLUSIVE taken with lock_timeout; batched UPDATE backfill under row locks; CHECK NOT VALID, then VALIDATE under SHARE UPDATE EXCLUSIVE, then SET NOT NULL; CREATE INDEX CONCURRENTLY per partition with ATTACH PARTITION; and the dual-write-to-read switch identified as the step that cannot be reverted cleanly.

Verify it yourself

  • No step in the plan holds ACCESS EXCLUSIVE for longer than the configured lock_timeout.
  • The writer's error rate stays at zero through the rehearsal apart from the deliberate lock timeout.
  • After VALIDATE, SET NOT NULL completes in milliseconds on a table where the naive form scanned every row.
  • Freshly written rows agree on both columns for a full verification period before the old column is dropped.

Follow-up

  • A CREATE INDEX CONCURRENTLY fails halfway through the partition list. What state is the table in, how do you detect it, and what do you run?
  • The producer computes quantity itself. What happens to a request already in flight when the dual-write deploy lands, and does it matter?
  • Give two queries that prove the backfill is complete: one cheap enough to run every minute, one authoritative.

Explain why the metering dashboard scans every daily partition

Medium · indexing · partition pruning · sargability · tenant skew

usage_event is range-partitioned daily on ingested_at and holds tenant_id, workspace_id, environment, sku, quantity numeric(20,6), occurred_at and ingested_at. The only relevant index is on (occurred_at). A dashboard runs select sku, sum(quantity) from usage_event where tenant_id = $1 and date_trunc('hour', occurred_at) >= $2 and environment = 'production' group by sku, and EXPLAIN shows a sequential scan of every partition. Give each distinct reason, rewrite the predicate so an index can serve it, propose the index, and state the write cost its column order adds.

Approach

  1. Separate the three causes rather than blaming one. First, date_trunc('hour', occurred_at) wraps the column, so the predicate is not sargable against a btree on the bare column. Second, pruning keys off ingested_at while the query constrains occurred_at, so no partition can be excluded. Third, even made sargable, (occurred_at) is not tenant-leading, so for one tenant among thousands the scan reads the whole time range and discards nearly all of it.
  2. Rewrite the bound carefully, because the obvious rewrite is only conditionally equivalent. date_trunc('hour', x) >= $2 equals x >= $2 only when $2 is already hour-aligned; for an arbitrary $2 it means x >= date_trunc('hour', $2) + interval '1 hour'. Normalise the parameter in the caller and leave the column bare.
  3. Restore pruning with a second, redundant predicate on the partition key: ingested_at >= $2 - interval '<late-data horizon>'. State both sides of it. It prunes to a handful of partitions, and it silently omits any event whose ingest lagged past that horizon, which is precisely what a producer replay produces. Either document the horizon as a stated bound, or partition on occurred_at and move the problem into the dedup window instead.
  4. Propose (tenant_id, occurred_at) include (sku, quantity) per partition. A partial index where environment = 'production' mostly saves size rather than selectivity, since production dominates the three environments; take it if non-production is a meaningful share and skip it otherwise.
  5. Price the write path honestly. At roughly 250M rows/day each extra index is another insert plus WAL per row, and a tenant-leading key scatters inserts across one hot leaf per active tenant instead of appending to a single rightmost leaf, so page dirtying and random I/O both rise. An INCLUDE payload widens every leaf entry and enlarges the index accordingly.
  6. Add the index-only-scan caveat before someone reports it as a regression: on a freshly appended table the visibility map is not yet set for recent pages, so the INCLUDE columns still cost heap fetches until autovacuum has been through, and the newest hour is exactly the data the dashboard reads.

Follow-up

  • CREATE INDEX CONCURRENTLY is not supported on a partitioned parent. Give the sequence that gets this index onto 400 existing partitions without blocking ingest.
  • One tenant holds 200 times the median row count and the dashboard still times out for them with the index in place. What changes?
  • Should this read hit usage_rollup_hourly instead? State what that costs in freshness and what the watermark lets you promise.

Ship three breaking-looking changes without breaking pinned SDKs

Hard · versioning · backward compatibility · deprecation

GET /v1/runs/{id} returns status from the set queued, leased, running, succeeded, failed, timed_out, cancelled. You must add the terminal state lost, add an optional billable_seconds field, and start rejecting name values over 64 characters that were previously accepted. Clients are generated SDKs pinned inside customer build pipelines you cannot redeploy or reliably contact, some with strict deserialisers. Classify each change as compatible or breaking and justify it mechanically, then give the rollout: the versioning scheme, the transform old clients receive, the telemetry, and the sunset process.

Approach

  1. Classify against a deserialiser rather than against intuition. A new enum value breaks a generated client that validates the enum or switches exhaustively; a new optional field breaks one generated with additionalProperties false; tightening validation breaks every caller already sending an 80-character name even though no field moved. All three are breaking for some real client, and that is the answer, not a technicality.
  2. Choose the versioning axis on maintenance cost. A path version forks the handler, and each subsequent change multiplies the fork. A date-pinned version header, defaulted per credential at first use and pinned thereafter, keeps one internal model and a chain of ordered response transforms, so the handler stays single and the transforms compose.
  3. Write the downgrade honestly. Older versions have no lost, so the transform maps it to the least-wrong existing value and the changelog says which; mapping it to failed asserts an outcome nobody observed, so pair it with a reason field the old version already exposes rather than inventing a field, and keep the mapping in one documented table instead of a conditional inside the serialiser.
  4. Sequence the validation tightening separately from the payload changes: first measure how many tenants send names over 64 characters, then warn for a stated window with a Deprecation header and a non-fatal warning code, then reject only for versions dated after the change, never retroactively for a pinned client.
  5. Deprecate from evidence, not from announcement: per-version, per-tenant, per-credential request counts from the gateway, a Sunset header carrying the date, direct contact for the handful of accounts that make up most of the tail, and short scheduled brownouts before the cutoff so the clients nobody can reach fail on your calendar rather than during their release.

Worked solution (40 min)

  1. Write the three-row classification table with the client-side mechanism that breaks in each case, not just a verdict.
  2. Pick the versioning scheme and write the header, the default-and-pin rule for a new credential, and where the transform chain sits relative to the handler.
  3. Write the lost mapping for pre-change versions as an explicit table entry plus the changelog sentence a customer reads.
  4. Write the validation timeline with four dated steps, and mark which one is visible to clients that never change their version.
  5. Write the deprecation plan: the telemetry query by version and tenant, the headers, the brownout schedule, and the fallback for the largest holdout.

Expected result: All three changes classified breaking with the mechanism named, a date-pinned version header with one internal model and ordered downgrade transforms, a documented lost mapping, validation tightened only for versions after the change, and a sunset driven by per-version traffic with brownouts.

Verify it yourself

  • No change is called additive without naming the client behaviour that makes it safe.
  • A client that never changes its version header sees no behaviour change from any of the three.
  • The plan names a measurement that tells you when sunset is safe.
  • The lost mapping is stated once, in a table, not implied by serialiser branches.

Follow-up

  • Which step of your rollout cannot be rolled back cleanly once a client has depended on it?
  • A tenant pinned to a sunset version breaks their build the morning after the cutoff. What in your plan already lets you resolve it within an hour?
  • Your transform chain is now six versions deep. What stops it becoming the fork you were avoiding?

Write the delivery guarantee and replay API for outbound events

Medium · webhooks · at-least-once · ordering

The webhook-delivery service fans events to customer endpoints, tracked in webhook_delivery (subscription_id, tenant_id, event_id, redelivery_seq, status, attempt_count, next_attempt_at, lease_token, last_response_code, payload_digest). Customers are asking for exactly-once and in-order delivery. Write the contract you can actually honour: the guarantee, the headers that make it usable, which customer response codes are retryable, the retry schedule and terminal condition, what happens to an endpoint that has been down for a day, and the shape of the redelivery endpoint. State plainly what you are not promising and what the customer must do instead.

Approach

  1. Refuse exactly-once with the mechanism rather than with policy: the customer's acknowledgement can be lost after they have already committed, so the sender cannot distinguish unprocessed from processed-but-unacknowledged and must retry. The deliverable is at-least-once with a stable event identifier and the customer documented as the deduplicating party.
  2. Price ordering instead of promising it. With parallel attempts per subscription, a retried event overtakes a newer one, so in-order delivery requires a single in-flight attempt per subscription, which turns any slow endpoint into head-of-line blocking for that subscription's whole queue. Offer it per subscription with that cost written down.
  3. Define the response contract from the customer's side: 2xx is accepted, 408, 429 and 5xx are retryable, 410 disables the subscription, and every other 4xx is permanent and terminal. Publish the read timeout and tell the customer to acknowledge first and process asynchronously, since their processing time otherwise consumes your worker occupancy.
  4. Publish the schedule and its end: capped exponential backoff with full jitter, sleeping uniformly in [0, min(cap, base * 2^attempt)] so a mass failure does not re-synchronise the herd, terminal after a stated attempt count or age, plus a per-endpoint circuit breaker that records further deliveries as dropped_circuit_open and notifies the owner rather than burning shared worker capacity.
  5. Shape redelivery as an insert, not a reset: POST to a redeliveries collection with event ids or a time window creates rows at redelivery_seq + 1 carrying the same bytes, which payload_digest lets you prove, leaving the original terminal rows intact as the record of what happened.
  6. Compare the event's tenant against the subscription's tenant at enqueue and again immediately before signing, because cross-tenant delivery originates in an enqueue path that took the subscription from one lookup and the payload from another, not in the worker.

Follow-up

  • A customer wants everything they missed during their six-hour outage. Do the dropped_circuit_open rows let you answer that, and what retention bound does the answer depend on?
  • You offer the ordered mode and one customer's endpoint slows to two seconds per request. What do their delivery metrics look like, and what do you owe them in the docs?

Invoice detail latency triples after an ORM relationship refactor

Easy · n+1 query · orm · latency regression

An invoice detail endpoint returned in 40 ms at p99 last week. After a refactor replaced a hand-written join with ORM relationship access it returns in 1.4 s, and the regression grows with the number of invoice_line_item rows on the invoice. Database CPU rose, but no statement in the slow-query log exceeds 3 ms. You have request traces with per-span SQL, the ORM statement log, and a staging copy of the data. Produce an ordered diagnostic checklist, the measurement that confirms the cause before any code change, and the fix.

Approach

  1. Count statements per request before reading any statement duration. A slow-query log hides this class by construction, because every individual query is fast and only their number is wrong; take one trace and count SQL spans.
  2. Establish proportionality rather than asserting it: sample invoices with 5, 20, 60 and 200 line items and plot statements per request against line count. A straight line of slope 1 through an intercept of one or two identifies a lazy relationship load, and no index or cache would move that line.
  3. Locate the emitting attribute access in the refactored code and check whether the same shape repeats one level deeper, for instance a tax or adjustment collection hanging off each line, which turns the cost quadratic.
  4. Fix with a bounded statement count: either one join that fetches invoice and lines together, or two statements where the second is WHERE invoice_id = $1 AND tenant_id = $2. Keep tenant_id in the predicate so the read stays tenant-scoped even though invoice_id already implies it.
  5. Choose between the two deliberately: the join duplicates the wide parent row across N children on the wire, the two-statement form avoids that for one extra round trip. Prefer the join for narrow parents and the split for wide ones.
  6. Pin it with a per-request statement-count assertion in a test that varies line count, because a latency assertion passes on a small fixture and would not have caught this.

Follow-up

  • The endpoint now also needs per-line tax rows. Show the shape that keeps statement count constant instead of reintroducing the same defect one level down.
  • How does this change if a transaction-pooling proxy sits between the service and the database, so each statement may land on a different backend session?
  • The same page paginates invoices with LIMIT and OFFSET. Why is that a second, independent defect, and what replaces it?

Suggested practice checklist

Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.

Day 1 — Recruiter conversation and project inventory

  • Write one page on each of three projects: the constraint, the option you rejected, the before-and-after measurement, and what went wrong. Put a third-party integration, a pipeline with retries, or a latency-sensitive service first if you have one.
  • Cut each project to two sentences with no internal names and say them aloud to someone outside engineering.
  • List your recruiter questions: which team the loop is for (core infrastructure, voice AI or core product) and whether coding is live or take-home.

Deliverable: Three project pages, three two-sentence summaries, and a written list of recruiter questions.

Day 2 — Graph traversal coding: BFS, its slow paths, and parallel BFS

  • Implement BFS on a grid and on an adjacency list from memory, then solve the bank's BFS web crawler with a depth limit and a visited set keyed on normalised URLs.
  • Introduce three slowdowns on purpose (visited marked at dequeue, pop from the front of a list, adjacency rebuilt per step), time each on a large input, and write one sentence explaining each.
  • Sketch a level-synchronous parallel BFS and write down how workers avoid claiming the same node and when parallelism does not pay.

Deliverable: A working BFS and crawler, a timing table for the slow variants, and a written parallel BFS sketch.

Day 3 — Data structures behind an API

  • Build the O(1) insert/delete/random set with swap-with-last deletion, then solve maximum contiguous subarray with Kadane's algorithm and handle the all-negative input.
  • Wrap the set in REST endpoints: resources, methods, status codes for a duplicate insert and a missing delete, and behaviour under concurrent requests.
  • Work the worked exercise on peak concurrent sandbox usage and check your tie-break and null handling against its expected result.
  • Take one ambiguous prompt, restate it with an example, and list the questions you would ask before coding.

Deliverable: Two coded solutions, an endpoint spec for the set, and the sweep-line exercise checked against its fixture.

Day 4 — Integration design: rate limits, retries and OAuth

  • Design the CRM and AI dialer sync framework: per-account sync cursors, a shared rate-limit budget per external account, retries for transient errors only, dead-lettering for permanent ones, and idempotent writes.
  • Add OAuth token handling: refresh ahead of expiry, one refresher per connection so workers do not race, encrypted storage, and the path when a refresh token is revoked.
  • Warm up with the bank's crawler-throttling question, then carry its per-host limiting over to per-account limits in the sync design.

Deliverable: A one-page integration design with a failure table: error type, retry or not, and where the work goes next.

Day 5 — Webhooks, real-time voice and the dialing question

  • Design at-least-once webhook ingestion: persist before acknowledging, dedupe on event id, process asynchronously, and handle out-of-order payloads with a version or timestamp rule.
  • Work the practice question on outbound delivery guarantees and replay, and write down what you would not promise and why.
  • For the voice questions in the bank (low-latency audio streaming, observability for voice routing, LLM transcription and summaries), write a latency budget by hop and the metrics and alerts that would catch a spike.
  • Answer the dialing robot optimization question, stating the objective and constraints before any configuration.

Deliverable: Two webhook designs, a per-hop latency budget with alerts, and a written objective and guardrail for the dialing question.

Day 6 — Troubleshooting and debugging

  • Rehearse the reported troubleshooting format on a real bug report of yours: restate the symptom technically, list hypotheses by layer, and name the evidence that separates them.
  • Work the debugging practice question on the invoice endpoint latency regression, counting statements per request before reading any durations.
  • Write up one race condition you have met in an integration (double sync, concurrent token refresh, concurrent webhook processing) and the fix.

Deliverable: Two written diagnoses with hypotheses and deciding evidence, plus one race-condition write-up.

Day 7 — Behavioral deep dives and a back-to-back mock

  • Prepare answers on the reported behavioral themes (a complex technical challenge, fast iteration vs reliability, mentoring or technical leadership) and the bank's production-outage question.
  • Run one coding question and one design question back to back with a partner, then a project deep dive where they ask why four times in a row.
  • Reduce the week to one page of rules you can state without reading them.

Deliverable: Four behavioral answers, mock notes on where the second round suffered, and a one-page card of rules.

Bounding a vague ticket before writing code

The reported behavioral questions for this role are about your own engineering history: complex technical challenges you handled, how you balance fast iteration against reliability, and mentoring or technical leadership. Answer with specifics a follow-up cannot knock over: the constraint, the option you rejected, what you measured, and which part was yours rather than the team's.

  1. Walk through your past work at a high level, focusing on the most complex technical challenge you handled and what you changed because of it.
  2. Describe a time you had to choose between shipping a product change quickly and protecting long-term system reliability. What did you decide, and what did it cost?
  3. Describe a situation where you mentored teammates or drove technical direction on a growing team. What changed in how they worked?
  4. Tell me about a production outage you owned: how you isolated the root cause, how you communicated during it, and what you changed afterwards.
  5. Describe an integration with a third-party API that failed in production. What broke, how did you find out, and what did you add so it would be caught sooner?

Argue against failing open when the control plane is unreachable

Hard · revocation · cache staleness · fail-open vs fail-closed · influence

The gateway caches credential-to-context decisions with a sixty-second TTL. A design proposal says that when control-plane reads fail, pods should keep serving from expired entries indefinitely so a control-plane outage never becomes a product outage. You believe that converts every revocation into an unbounded one. Describe a design you argued against while it was still a live proposal: what you measured or modelled to make the case, what you conceded, who decided, and what happened afterwards. Say what would have changed your mind before the decision, not after it.

Approach

  1. Reframe it from a values argument into a bounded-staleness argument. Both sides already accept the cache; the disagreement is only about the ceiling on how long a revoked credential keeps authorising. Put a number on the table — serve stale for up to fifteen minutes, then fail closed — and make the other side argue against a number rather than against a principle.
  2. Bring arithmetic rather than adjectives: the rate of revocations with revoked_reason in ('suspected_leak','auth_version_bump'), the observed distribution of control-plane unavailability, and the product of the two, which is expected requests served by revoked credentials per outage-hour. At 30k requests/second the unbounded version is not a subtle exposure and the number says so.
  3. Concede the strong half of the opposing case first, because that is what buys you the room: failing closed turns one service's outage into a total outage across three regions, and a control plane doing tens of writes per second is not engineered to the gateway's availability target. A proposal you have not steelmanned reads as reflex.
  4. Propose the asymmetry that usually resolves this: stale entitlements cost bounded money (a quota fifteen minutes out of date over-serves by a computable amount), while a stale revocation costs unbounded access. Split the cached decision by what it authorises, give the two halves different staleness ceilings, and let the entitlement half fail open while the revocation half fails closed.
  5. State the propagation dependency plainly, since it is the part that is missed: validity is also derived from the principal's auth_version, so password reset and sign-out-everywhere flow through this same cache. A design that bounds staleness for explicit revocation and not for auth_version bumps has only fixed half of it.
  6. Say who decided, and what you did afterwards in either outcome: write the decision down with its number and a review date, and instrument the exposure you were worried about so the next round of the argument is settled by data instead of by seniority.

Follow-up

  • Publish-subscribe invalidation is lossy under a partition, and a TTL is the only hard bound. What TTL do you pick, and what does it cost you at 30k requests/second?
  • The key was revoked because it was found in a public repository. Does your answer change, and where does that urgency live in the design?
  • You lost the argument and six weeks later the failure you predicted happens. What do you say in the review, and what do you not say?

Disclose a cross-tenant webhook delivery to affected customers

Medium · cross-tenant leak · disclosure · blast radius · authorisation checks

An enqueue path took the subscription from one lookup and the payload from another. For nineteen minutes, webhook_delivery rows were created whose tenant_id did not match the subscription's tenant, and eleven payloads were signed and sent to four endpoints belonging to other customers. You hold payload_digest, delivery timestamps and response codes. Describe how you handle a disclosure of this kind: what the records prove, what they cannot prove, what you say before you know everything, the one code change that closes it, and which parts you personally drove.

Approach

  1. Bound the population before saying anything externally. The affected set is deliveries in the window where the event's tenant and the subscription's tenant differ; the ones that actually left are those with delivered_at set and a 2xx in last_response_code. Attempted and delivered are two different counts and a disclosure has to use the right one in the right sentence.
  2. Separate what the records prove from what they do not, and say both halves rather than the flattering one. They prove which payloads were signed, where they went, and — through payload_digest — exactly which bytes. They do not prove what the receiving system did with them, and they do not bound the window more precisely than your deploy timestamps do.
  3. Communicate on the facts you hold, with the scope stated as an upper bound: 'at most eleven payloads, four recipient endpoints, these fields, this window' is more useful and more honest than waiting a day for certainty. The field list matters more than the event count, because a customer cannot assess exposure from 'an event'.
  4. Name the code change precisely, because this class never originates in the delivery worker. Compare the event's tenant against the subscription's tenant at enqueue and again immediately before the payload is signed, and make the second comparison drop the delivery rather than log a warning. Say why one check is insufficient: the enqueue check protects against the bug you know about, the pre-signing check protects the boundary itself.
  5. Run the history question in parallel and say so: a query over historical deliveries for the same mismatch tells you whether this was nineteen minutes or a year, and you would rather find the second case yourself than have a customer find it after your disclosure.
  6. Split the response into workstreams with owners — recipients asked to delete, affected customers notified, the check landed with a test, history swept — and say which you personally drove and which you handed off. Claiming all four is not credible and claiming none is not ownership.

Follow-up

  • The historical sweep finds two more instances from last year. What changes in what you have already told people?
  • Who approves the wording, and what do you do when you are asked to soften the scope?
  • A customer asks you to prove a redelivery contained the same bytes as the original. What do you show them?

Reverse a webhook ordering decision after measuring its cost

Medium · reversing decisions · head-of-line blocking · at-least-once · api contracts

You argued for strict per-subscription ordering in webhook-delivery, which means one in-flight attempt per subscription. It shipped. Three months later a single unresponsive endpoint holds one subscription's queue at a six-hour backlog, and two customers report events arriving out of order anyway once their own retries are counted. Describe a decision you reversed: what you originally optimised for, the measurement that changed your mind, what the reversal cost in engineering time and customer change, and how you told the people who had already built on the original guarantee.

Approach

  1. State the original decision as a trade you made knowingly. Ordering across a network requires a single in-flight attempt per subscription, and its price is head-of-line blocking whenever one endpoint is slow. 'We priced it wrong' is a much stronger opening than 'we did not realise', and it is usually the true one.
  2. Bring the measurement that flipped it, not the anecdote: backlog age at the ninety-ninth percentile per subscription, the share of subscriptions where one slow endpoint gated an otherwise healthy queue, and the delivery throughput lost to serialisation. A reversal justified by complaints is indistinguishable from a reversal justified by fatigue.
  3. Name what you learned about the guarantee itself, which is the engineering content of this story. At-least-once delivery means a retried event already arrives after newer ones and the consumer already must be idempotent, so a guarantee the customer has to defend against anyway was never worth what it cost to provide.
  4. Describe the migration, because reversing a published contract is the hard half and the part candidates skip. Parallel attempts behind a per-subscription flag, a monotonically increasing sequence number added to the envelope so order-sensitive consumers can sort or discard, documentation that states at-least-once and unordered in those words, and a deprecation measured in quarters because the client is a pinned SDK inside a build pipeline you cannot see or redeploy.
  5. Give the cost in the two currencies that matter: engineer-weeks, and how many customers had to change code. Then say who you told before it shipped rather than in a changelog afterwards, and which large customer you left on the old behaviour and for how long.
  6. Close with the signal you now weight differently, stated as something you would do earlier next time: measuring the blocking cost on the slowest decile of endpoints before committing to the guarantee, rather than after a customer noticed.

Follow-up

  • A customer insists they need ordering. What do you offer them that is not global serialisation?
  • How did you choose the deprecation window given that you cannot see or redeploy the clients?
  • What would have to be true for you to reverse back?