CloudKitchens Software Engineer Interview Guide

The 60-second brief

CloudKitchens builds software for ghost kitchens, delivery-only restaurants and food logistics. Software Engineers work on order processing, real-time fulfillment, streaming menu management and multi-channel delivery management, including the Otter platform. Orders come in from delivery apps such as DoorDash and UberEats and move to kitchen display systems, then onto pickup shelves, then to couriers. The reported interview questions follow that same flow: parsing menus, simulating shelves and couriers, and designing the services behind them.

This guide covers the two stages candidates report for the CloudKitchens Software Engineer role, the Recruiter Call and the Technical Interviews, plus the five question categories behind them: object-oriented design and stream parsing, concurrency and real-time simulation, top-k and stream algorithms, system design including a past-project deep dive, and behavioral. The SQL, billing-reconciliation and webhook-delivery drills further down are general backend practice. They are not reported CloudKitchens questions.

Focus areas

• Scope every query and cache key by tenant
• Build at-least-once pipelines with explicit deduplication horizons
• Make every write idempotent under client retries

Overview

A Software Engineer at CloudKitchens builds the systems that take an order from a delivery app to a courier's hands. The role covers order processing, real-time fulfillment, streaming menu management and multi-channel delivery management platforms such as Otter. Orders arrive from platforms like DoorDash and UberEats, then show up on kitchen display systems, sit on pickup shelves and leave with couriers. The work involves order value decaying over time, allocating shelf space, updating shared state safely across threads and parsing menus for many tenants.

The reported questions come from that domain. Candidates describe parsing a streamed restaurant menu into an object model and writing it back out, running a kitchen with hot, cold, ambient and overflow shelves while orders decay and couriers arrive, finding the top k items in kitchen logs or telemetry streams, and designing services such as cart synchronization for a delivery aggregator or an S3-like blob store. They also describe walking an interviewer through the architecture of a system they built before.

Candidate reports put most of the weight on code that runs. The recurring advice is to write thread-safe, compilable code with tests, parse real input instead of hardcoding it, and be ready to defend your design choices in a later conversation. Java, Go and Kotlin are the languages reports mention most often for the concurrency work. Confirm the language and tooling rules with your recruiter before you start the practical work.

The CloudKitchens interview process

CloudKitchens candidates report 2 rounds · ≈ 2-4 weeks. The stages below are what candidates describe, not a published process.

Recruiter Call

Candidates describe the Recruiter Call as a background and role conversation. It covers how your experience lines up with the role, what the team expects and whether the fit is right. Use it to learn the format of the technical stage, because candidate reports describe more than one version. Some report a take-home assignment. Others report a live coding project. The technical screen can be an object-oriented design problem or a timed coding question. Knowing which one you face changes how you spend your prep week.

What to demonstrate

• Whether your background aligns with the Software Engineer role, which candidates report as the main purpose of this call
• Operational culture fit, which candidates report this call also covers
• Whether your expectations for the role match what the recruiter describes

How to prepare

• Ask which format the practical coding work takes (take-home or live), which languages are accepted, and whether you will review that code with engineers in a later session
• Ask for the written policy on AI coding tools and auto-completion so you practise under the same conditions
• Prepare a two-minute summary of the most complex system you built end to end, naming the services you owned and the concurrency or streaming problems you solved, because reports say a later round goes deep on one past project
• Have one or two examples of thread-safe or streaming code you shipped, with the concurrency problem each one solved

Technical Interviews

Candidates describe this stage as a series of interviews that can include coding and system design. Reports describe it in stages. It starts with a technical screen, often an object-oriented design problem such as parsing a streamed menu, or a timed coding question. Next comes either a take-home or a 3-hour live coding project. There you build a multi-threaded simulation, such as kitchen order fulfillment with courier dispatch, that ingests JSON orders, keeps state thread-safe and ships with unit tests. After that, reports describe a virtual onsite. It includes a code review where you defend your simulation, an architecture presentation or system design round, a CS fundamentals and algorithms round, and a behavioral conversation with an engineering manager.

What to demonstrate

• Whether your code is thread-safe and compiles: reports mention your choice of data structures, class hierarchy, synchronization primitives and unit tests
• Whether you can model a messy domain, such as nested menu categories, items and options or temperature-controlled shelves, as clean objects and round-trip it through a stream format
• Whether your complexity analysis is correct for stream problems like top-k with a bounded heap or a moving window, and whether you handle edge cases
• Whether you can defend end-to-end data flow, storage choices, trade-offs and failure modes, both for your own past project and for the code you just submitted

How to prepare

• Build the kitchen fulfillment simulation once from scratch in the language you will use: a JSON order feed, shelves with decay and overflow, courier dispatch, step-by-step logging, clean shutdown and unit tests
• Implement a menu stream parser and its serializer, including blank-line separators, a None terminator and malformed lines, and test that parse followed by serialize reproduces the input
• Drill top-k with a hash map and min-heap, the streaming add_data/find_top_k version, and top-k in a moving window
• Prepare one past-project deep dive with a high-level diagram, data schemas, API contracts, the compromises you made and how the system fails

Where candidates lose points

Spawning raw threads in the fulfillment simulation and never shutting them down cleanly

Put the work on a thread pool or executor and use bounded queues between the order producer and the kitchen and courier consumers. Guard shelf state with one clearly named lock or a concurrent structure, not ad hoc synchronization spread across classes. Define when the program ends: every order has been consumed and each one is either picked up or discarded. Then shut the pool down and join it. The bank question about terminating after consuming all orders targets exactly this.

Handing in the simulation with hardcoded orders, no tests, or code that does not compile

Parse the actual JSON order input instead of building orders in code, and write unit tests for shelf placement, moving orders to and from overflow, value decay and courier pickup. Reports say a working, tested solution with a few features cut rates higher than a bigger one that does not run. Get a thin version compiling and passing tests early, then add features. List what you cut so you can explain it in the code review that follows.

Writing a menu stream parser that only handles clean input

Before coding, restate the input contract: line delimiters, blank-line separators between items, how categories, entrees and options nest, and the None termination token. Decide what happens to a malformed or unexpected line (skip and log, or fail) and say so out loud. If the prompt asks for serialization, test that serializing the parsed object graph reproduces the original stream.

Sorting the whole frequency map for every top-k query, or rebuilding the heap on every update

For a one-off query over a log, count with a hash map and keep a min-heap of size k, which gives O(n log k) instead of O(n log n). For the add_data and find_top_k class, say whether counts only ever increase and choose between an indexed heap with O(log n) updates and computing on demand, based on how often each call happens. For top-k in a moving window of size m, explain how you remove elements that fall out of the window (for example, lazy deletion) before you write the code.

Presenting a past project you can only describe at the diagram level

Choose a system you designed and built yourself. Be ready to go below the boxes to data schemas, API contracts, the threading model, the compromise you would change today and how the system behaves when each dependency fails. Use the same approach for your simulation code: before the review, write down its weakest point and what you would change with more time, so you raise it before the interviewer does.

Questions & practice

Implement a stream parsing function that processes raw text strings until reaching a stream termination token (None), handling malformed or unexpected lines gracefully.

Medium · data structures and algorithms

Approach

1. Restate the input: its shape, its size, and what is guaranteed about it.
2. Walk one small example through your approach before writing the whole thing.
3. State the target complexity and say which constraint rules the naive version out.

Follow-up

• What is the worst case, and how likely is it on real data?
• How does this change if the input no longer fits in memory?

Parse a large kitchen log file and calculate the top $k$ most popular food items using a hash map and a min-heap.

Medium · data structures and algorithms

Approach

1. Restate the input: its shape, its size, and what is guaranteed about it.
2. State the target complexity and say which constraint rules the naive version out.
3. Name the brute-force solution and its complexity before improving on it.

Follow-up

• How does this change if the input no longer fits in memory?
• Which test case would catch an off-by-one here?

Design a data stream processing class with add_data and find_top_k functions, optimizing time and space complexity for real-time telemetry.

Medium · data structures and algorithms

Approach

1. State the target complexity and say which constraint rules the naive version out.
2. Walk one small example through your approach before writing the whole thing.
3. Restate the input: its shape, its size, and what is guaranteed about it.

Follow-up

• What is the worst case, and how likely is it on real data?
• How does this change if the input no longer fits in memory?

Design a concurrent producer-consumer pipeline that handles order preparation, courier dispatching, pickup events, and timing bounds safely across threads.

Medium · languages, concurrency and fundamentals

Approach

1. Say what the runtime actually does before reasoning about the code.
2. Reach for the cheapest primitive that closes the race, not the broadest lock.
3. Identify the window where an invariant is briefly untrue.

Follow-up

• Where could this allocate more than you expect?
• What happens if two callers reach this at the same time?

Implement a thread-safe reentrant lock from scratch and explain how thread pools prevent thread starvation under heavy concurrent workloads.

Medium · languages, concurrency and fundamentals

Approach

1. Say what the runtime actually does before reasoning about the code.
2. Identify the window where an invariant is briefly untrue.
3. Name what is shared across threads and what owns each piece of state.

Follow-up

• Where could this allocate more than you expect?
• How would you prove the race exists rather than suspect it?

Locate a billing reconciliation gap without rescanning ninety million events

Hard · reconciliation · dimensional-bisection · watermarks · hypothesis-testing

A tenant's sealed invoice total is 0.4% below the sum of its raw usage_event rows for the period. That tenant has 90 million events over 30 days in a table partitioned daily on ingested_at, and its rollups carry source_max_ingested_at, revision and sealed_at. Recomputing all 30 days from raw is correct, and you are not going to do it. Give the procedure that locates the divergent (workspace, sku, hour) cell, the cost of each probe, and the one query you run before any of it.

Approach

1. Run the free query first. Sum raw quantity for the period restricted to ingested_at <= source_max_ingested_at of the sealed rollups, and compare that against the unrestricted sum. The rollup stores the watermark precisely so this can be answered without a scan. If the whole 0.4% sits above the watermark, nothing is broken: it is late data, it becomes an adjustment line, and the investigation ends in one query.
2. Only if the gap survives that test do you bisect, and you bisect by dimension rather than by rows. Compare 30 per-day totals, then inside the offending day compare the 6 SKUs, then the workspaces, then the 24 hours. That is roughly 30 + 6 + W + 24 grouped probes, each an indexed range scan over one daily partition for one tenant, against O(N) per attempt for the naive re-fold.
3. Quantify why naive is not merely slow but unusable mid-incident: at a generous 200,000 rows/second sequential, 90 million rows is about 7.5 minutes per attempt, you will want ten attempts, and every one competes for I/O on the same partitions live ingest is writing. The diagnostic worsens the backlog it is diagnosing.
4. Before fetching each comparison, state what it would look like under each hypothesis. Two adjacent hours off by equal and opposite amounts is occurred_at versus ingested_at bucketing. A whole day offset by exactly N hours is a timezone applied at the wrong layer. A gap confined to one SKU in one workspace is an environment filter. The same (tenant_id, idempotency_key) present in two ingested_day partitions is the dedup horizon losing a retry that crossed midnight.
5. Make the next bisection cheap by storing the aggregate you keep recomputing. A per-(tenant_id, ingested_day) count and quantity checksum turns step two from thirty probes into one read, and it is the same number the reconciliation job already produces.
6. Whatever you find, the sealed period does not change value. The correction is an adjustment line pointing at the line it reverses, carrying its own source_rollup_watermark, because the original invoice is the evidence of what the customer was charged.

Worked solution (35 min)

1. Reproduce the shape locally: generate 2 million events for one synthetic tenant over 5 days, fold them, then inject three defects, namely 0.2% of events bucketed by ingested_at, a handful of duplicate idempotency_key values whose retries cross midnight, and a block of events ingested after the seal.
2. Write the watermark-bounded query first and record how much of the gap it explains on its own.
3. Bisect by day, then SKU, then hour, recording the probe count and the rows each probe touches.
4. For each located cell, write down the predicted signature before querying it, then check whether the data matches the prediction.
5. Time a full re-fold of the 2 million rows and extrapolate to 90 million.

Expected result: The watermark-bounded query accounts for the post-seal block entirely and removes it from the investigation. The ingested_at bucketing appears as adjacent hours off by equal and opposite amounts. The midnight-crossing duplicates appear as one (tenant_id, idempotency_key) present in two ingested_day partitions. Bisection reaches each cell in fewer than 70 probes.

Verify it yourself

• The three injected defects sum to the original gap with no unexplained remainder.
• Each probe's row count is bounded by one day-partition for one tenant, not by the period.
• Removing the source_max_ingested_at bound from the first query makes a gap appear, and grow, while ingest runs, even after every defect is removed.

Follow-up

• The gap is 0.4% in one direction on one day and 0.4% the other way the next day. What does that shape rule in, and what does it rule out?
• How do you distinguish a duplicate from a restatement, given revision and recomputed_at on the rollup?
• Ingest is still running while you investigate. What makes your two numbers comparable at all?

Model credential revocation so history survives the delete

Easy · soft delete · partial indexes · check constraints · audit history

tenant_api_key stores key_id, tenant_id, workspace_id, name, key_prefix, secret_hash, scopes text[], status (active, revoked, expired, compromised), auth_version, created_at, expires_at, last_used_at, revoked_at, revoked_reason. Rotation inserts a new row and revocation never deletes, because an incident review asks which credential served a request last quarter. Write the constraints that enforce: a label is unique only among a tenant's live keys, revoked_at and status can never disagree, and scopes is never empty. Then write the authentication lookup predicate, and name one column in this table that must stay out of it.

Approach

1. Reach for a partial unique index rather than a plain UNIQUE: create unique index on tenant_api_key (tenant_id, name) where revoked_at is null. Any number of revoked rows may share a label, the live namespace stays unique per tenant, and the revoked majority is not in the index at all, so it stays small on a table that only grows.
2. Tie the nullable timestamp to the enum so the two cannot drift: check ((revoked_at is not null) = (status in ('revoked','compromised'))) and check ((revoked_at is null) = (revoked_reason is null)). A revocation that records no reason is the one an incident review cannot use.
3. Write the emptiness check as check (cardinality(scopes) > 0), not array_length(scopes, 1) > 0. array_length returns NULL for an empty array, a CHECK constraint passes when its expression is NULL, so the array_length version accepts exactly the value it was written to reject.
4. Make the lookup a single index probe with every liveness condition inside it: where secret_hash = $1 and revoked_at is null and (expires_at is null or expires_at > now()) and auth_version = $2, backed by a unique index on secret_hash. Nothing is filtered in application code, so there is no path that forgets a clause.
5. Keep last_used_at out of that predicate. It is written asynchronously and is allowed to lag by a minute, so it is a usage signal; feeding it into an authorisation decision makes the decision depend on a write that may be late, batched away or lost.
6. Flag the modelling smell while you are here: expired is derivable from expires_at < now(), so storing it as a status obliges a job to keep it true and guarantees the column is wrong between the expiry instant and that job's next run. Derive it in the predicate; keep the stored status for states that are decisions rather than clock readings.

Worked solution (20 min)

1. Create the table with all three constraints on a scratch database and insert two revoked rows sharing (tenant_id, name); the partial index should accept both.
2. Insert a second live row with that same name and confirm the violation names the partial index.
3. Run update tenant_api_key set revoked_at = now() leaving status = 'active' and confirm the CHECK rejects it; then try insert ... scopes = '{}' against both the cardinality and the array_length forms and note that only one rejects it.
4. Run explain (analyze, buffers) on the lookup predicate for a live key and confirm an index scan on secret_hash with rows removed by filter equal to zero.

Expected result: A partial unique index on (tenant_id, name) where revoked_at is null; check ((revoked_at is not null) = (status in ('revoked','compromised'))); check ((revoked_at is null) = (revoked_reason is null)); check (cardinality(scopes) > 0); a single-probe lookup on secret_hash plus revoked_at, expires_at and auth_version; last_used_at excluded, with a note that 'expired' should be derived rather than stored.

Verify it yourself

• Two revoked keys with the same name insert cleanly; a second live one does not.
• The array_length form of the emptiness check accepts an empty array and the cardinality form rejects it.
• No statement anywhere in the design deletes or updates a revoked row.
• Revoking a key touches exactly one row and takes effect on the next lookup with no application-side filtering.

Follow-up

• Rotation issues a replacement while the old key stays live for a 30-day overlap. What does the uniqueness rule become, and what does the UI show to tell two same-named keys apart?
• A password reset bumps the principal's auth_version. No row in this table changed. How does the next request fail, and what query counts how many keys that bump just killed?
• A key turns up in a public repository. Which columns let you find it, and what do you write to the row?

Enforce a concurrent-run quota that survives simultaneous requests

Medium · isolation levels · write skew · quotas · lock contention

A plan allows at most 20 concurrently running rows in job_run per tenant. The table holds run_id, tenant_id, workspace_id, status (queued, leased, running, succeeded, failed, timed_out, cancelled, lost), lease_token, leased_until, started_at and finished_at. Today the service runs select count(*) from job_run where tenant_id = $1 and status = 'running', compares the result to 20, then inserts. Under load a tenant exceeds the cap by exactly the number of concurrent requests. Name the anomaly, say which isolation levels do and do not prevent it, and give a version that holds, as SQL.

Approach

1. Name it: write skew. Each transaction reads a predicate (the count of running rows), neither modifies what the other read, and both then insert rows that jointly violate an invariant no single row expresses. Read committed permits it. So does repeatable read, because snapshot isolation's first-updater-wins check fires only on conflicting row updates, and these are inserts touching disjoint rows.
2. Enumerate the fixes with their real costs. SERIALIZABLE works: PostgreSQL's SSI tracks the predicate read and aborts one transaction with SQLSTATE 40001, which obliges the caller to retry and makes the abort rate rise with contention on a hot tenant. Folding the predicate into the write as insert ... select ... where (select count(*) ...) < 20 narrows the race to the statement's snapshot but does not close it under read committed.
3. Give the version that holds at read committed: serialise on a row both transactions must touch. update tenant_concurrency set running = running + 1 where tenant_id = $1 and running < 20 returning running updates zero rows when the cap is reached, and zero rows is the rejection. This works because at read committed a blocked UPDATE re-evaluates its WHERE clause against the newly committed row; at repeatable read the same statement raises a serialisation error instead, so the isolation level changes the calling contract.
4. State the cost you just bought. That row is now a per-tenant serialisation point, so admission throughput for the tenant is bounded by one divided by the lock hold time; at a 2 ms hold that is roughly 500 admissions/second. Keep the critical section to the single UPDATE, with no network call or scheduling decision inside the transaction, and decrement in the same transaction that writes the terminal status.
5. Close the leak the status enum implies: a run can end as lost, so a crashed worker otherwise consumes a slot forever. Reconcile on a schedule against status = 'running' and leased_until < now(), and treat the counter as a fast path over job_run, which stays the system of record.

Follow-up

• Write the retry loop for the SERIALIZABLE version. What does the caller see when it keeps aborting, and what bounds the retries?
• Two regions each keep a counter. What is the effective cap, and what does admission do when the counter store is unreachable?
• The cap changes mid-flight on a plan upgrade. Do running jobs get killed, and what does the counter row look like during the change?

Design a multi-tenant food delivery aggregation engine that synchronizes shopping cart states, supports offline user location recommendations, and updates menu items in real time.

Medium · system design and architecture

Approach

1. Name the read and write paths separately; they rarely have the same bottleneck.
2. Name the failure you are designing for, then the recovery path.
3. Fix the scope first: who calls this, how often, and what they do when it fails.

Follow-up

• What breaks first when traffic grows ten times?
• How does this behave when that dependency is down for an hour?

Architect a high-throughput, highly available blob storage service similar to Amazon S3.

Medium · system design and architecture

Approach

1. State the consistency you need, and where you are willing to be stale.
2. Fix the scope first: who calls this, how often, and what they do when it fails.
3. Name the read and write paths separately; they rarely have the same bottleneck.

Follow-up

• What breaks first when traffic grows ten times?
• What would you drop to keep the system up under load?

Conduct an incident response troubleshooting exercise: analyze unfamiliar multi-service documentation within a limited timeframe to diagnose architecture bottlenecks and propose operational fixes.

Medium · system design and architecture

Approach

1. Choose a partition key and say what query it makes expensive.
2. Name the failure you are designing for, then the recovery path.
3. Fix the scope first: who calls this, how often, and what they do when it fails.

Follow-up

• What would you drop to keep the system up under load?
• What breaks first when traffic grows ten times?

Implement bidirectional stream conversion methods that take a structured menu object graph and serialize it back into its original input stream format.

Medium

Approach

1. Work from the requirement backwards to the design.
2. State your assumptions explicitly before working the problem.
3. Clarify what is being asked and what a complete answer contains.

Follow-up

• What assumption would you test first?
• How would you know your answer was wrong?

Write the delivery guarantee and replay API for outbound events

Medium · webhooks · at-least-once · ordering

The webhook-delivery service fans events to customer endpoints, tracked in webhook_delivery (subscription_id, tenant_id, event_id, redelivery_seq, status, attempt_count, next_attempt_at, lease_token, last_response_code, payload_digest). Customers are asking for exactly-once and in-order delivery. Write the contract you can actually honour: the guarantee, the headers that make it usable, which customer response codes are retryable, the retry schedule and terminal condition, what happens to an endpoint that has been down for a day, and the shape of the redelivery endpoint. State plainly what you are not promising and what the customer must do instead.

Approach

1. Refuse exactly-once with the mechanism rather than with policy: the customer's acknowledgement can be lost after they have already committed, so the sender cannot distinguish unprocessed from processed-but-unacknowledged and must retry. The deliverable is at-least-once with a stable event identifier and the customer documented as the deduplicating party.
2. Price ordering instead of promising it. With parallel attempts per subscription, a retried event overtakes a newer one, so in-order delivery requires a single in-flight attempt per subscription, which turns any slow endpoint into head-of-line blocking for that subscription's whole queue. Offer it per subscription with that cost written down.
3. Define the response contract from the customer's side: 2xx is accepted, 408, 429 and 5xx are retryable, 410 disables the subscription, and every other 4xx is permanent and terminal. Publish the read timeout and tell the customer to acknowledge first and process asynchronously, since their processing time otherwise consumes your worker occupancy.
4. Publish the schedule and its end: capped exponential backoff with full jitter, sleeping uniformly in [0, min(cap, base * 2^attempt)] so a mass failure does not re-synchronise the herd, terminal after a stated attempt count or age, plus a per-endpoint circuit breaker that records further deliveries as dropped_circuit_open and notifies the owner rather than burning shared worker capacity.
5. Shape redelivery as an insert, not a reset: POST to a redeliveries collection with event ids or a time window creates rows at redelivery_seq + 1 carrying the same bytes, which payload_digest lets you prove, leaving the original terminal rows intact as the record of what happened.
6. Compare the event's tenant against the subscription's tenant at enqueue and again immediately before signing, because cross-tenant delivery originates in an enqueue path that took the subscription from one lookup and the payload from another, not in the worker.

Worked solution (30 min)

1. Write the guarantee sentence and the one-sentence reason exactly-once is not available over HTTP.
2. List the headers a customer needs to build their own dedup table: event id, delivery id, attempt number, subscription id, signature and timestamp.
3. Write the response-code table with four classes and the action for each, including the 410 case.
4. Write the retry schedule with concrete numbers, the terminal condition, and the breaker's threshold and its effect on the delivery row's status.
5. Specify the redelivery request and response, and say which columns change and which do not.

Expected result: At-least-once with a stable event id, ordering offered only as an opt-in single-in-flight mode with head-of-line blocking stated, a four-class response table, a jittered capped schedule with a terminal state, and redelivery implemented as new rows at an incremented sequence carrying identical bytes.

Verify it yourself

• The customer can deduplicate using headers alone, without parsing the payload.
• Nothing in the contract requires the sender to know whether the customer committed.
• A day-long outage has a defined end state rather than unbounded retries.
• Redelivery leaves the original delivery rows readable as history.

Follow-up

• A customer wants everything they missed during their six-hour outage. Do the dropped_circuit_open rows let you answer that, and what retention bound does the answer depend on?
• You offer the ordered mode and one customer's endpoint slows to two seconds per request. What do their delivery metrics look like, and what do you owe them in the docs?

A rare job-run overwrite that logging makes disappear

Hard · fencing tokens · leases · concurrency

About one job run in fifty thousand records billable_seconds matching no observed sandbox lifetime, and a few rows show worker_id changing after finished_at was already set. It does not reproduce: debug logging around the terminal write made it vanish for two weeks before it returned. Runs last from 200 ms to 30 minutes, the lease is 60 seconds and is renewed while a run executes. Give an ordered checklist, the mechanism, and a fix that makes the illegal write impossible rather than merely rarer.

Approach

1. Mine the evidence instead of chasing a repro: select rows where updated_at is later than finished_at, or where a terminal status was written twice, and join them to attempt history to recover both writer identities. The defect has already happened tens of times and the rows are the recording.
2. State the signature before measuring it. If the cause is a lease that expired while the original worker was stalled, affected runs should cluster where the gap between the last renewal and the terminal write exceeds the lease, and should correlate with worker pause metrics rather than with workload shape.
3. Read the disappearance honestly. Logging inside the window changed the timing and lowered the probability; it is evidence about how narrow the window is, not a fix. Reproduce by widening the window on purpose, shortening the lease and injecting a pause between sandbox exit and the terminal write, rather than by adding more instrumentation.
4. Name the mechanism precisely: the lease expires during a stall such as a long garbage-collection pause or a brief partition, the run is re-dispatched, and the original worker then wakes and writes its terminal state over the new attempt's row. A lease alone cannot stop this, because the check and the write are separated by the stall.
5. Fix by fencing the write itself: UPDATE job_run SET status = $2, finished_at = $3, billable_seconds = $4 WHERE run_id = $1 AND status = 'running' AND lease_token = $5, with zero rows affected interpreted as having been fenced rather than as success. The token lives on the row so the store arbitrates, not the worker's memory.
6. Keep the state machine honest: a retry inserts a new row pointing at parent_run_id rather than resetting the old one, and a run whose worker vanished terminates as lost with billable_seconds null, because recording failure asserts an outcome nobody observed and then bills and retries on that assertion.

Follow-up

• The supervisor also emits a usage event on completion. What does the fenced worker do about the event it already emitted, and how does metering absorb it?
• Lease renewal is itself a network call. What happens when a renewal times out, and how does the worker decide whether it still holds the lease?
• Why is lengthening the lease past the longest legitimate run the wrong lever, and what breaks if you do it anyway?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Recruiter call and setup

• Write a two-minute summary of your background focused on backend services, concurrency and streaming work, and list the questions for the Recruiter Call: the format of the practical project, accepted languages, the AI tool policy and whether you will defend your code in a later session
• Choose your language (reports mention Java, Go or Kotlin for the concurrency work) and set up a project skeleton with a test runner and a JSON library you know
• Turn off AI completion in your editor for the rest of the week

Deliverable: A one-page call prep sheet and a project template that compiles, with one passing test.

Day 2 — Object-oriented design and menu stream parsing

• Model a restaurant menu with categories, entree items and options, with clear parent-child links, and justify each class boundary in one line
• Implement a parser that reads a streamed menu line by line, separates items by blank lines, stops at a None terminator and handles malformed lines in a rule you state
• Write the reverse serializer and a test that parse followed by serialize reproduces the input
• Sketch a receipt system that updates totals as items are added and applies combo discounts by backtracking over combinations

Deliverable: A tested menu parser and serializer with a round-trip test, plus a written note on the malformed-line rule.

Day 3 — Concurrency primitives

• Implement a bounded producer-consumer queue with a lock and condition variables, then replace it with your language's standard concurrent queue and compare the two
• Implement a reentrant lock from scratch that tracks the owning thread and a hold count, and write the test that shows a thread can re-acquire it
• Write a paragraph on how a fixed-size thread pool with a bounded queue avoids unbounded thread creation and starvation, and what happens when the queue fills
• Work the producer-consumer pipeline question on this page: name each piece of shared state and the one component that owns it

Deliverable: A tested bounded queue and reentrant lock, plus written notes on thread-pool sizing and saturation behaviour.

Day 4 — Build the kitchen fulfillment simulation

• Ingest JSON orders at a fixed rate and place each one on a hot, cold, ambient or overflow shelf, moving orders between shelves when one fills up
• Apply value decay over time and discard orders whose value reaches zero
• Dispatch couriers on a thread pool, log every state change (received, placed, moved, picked up, discarded) and stop cleanly once all orders are consumed
• Write unit tests for placement, overflow moves, decay and pickup, then list the features you would cut if the clock ran out

Deliverable: A running, tested simulation with a README that explains the concurrency design and what was cut.

Day 5 — Algorithms over data streams

• Solve top-k popular items from a kitchen log with a hash map and a min-heap of size k, stating O(n log k) time
• Build a class with add_data and find_top_k, and decide between an indexed heap with O(log n) updates and computing on demand from the call ratio
• Solve top-k largest in a moving window of size m using a heap with lazy deletion, and test windows at the start and end of the stream
• Warm up with Two Sum using a hash map and say the complexity out loud before coding

Deliverable: Three top-k solutions with stated complexity and edge-case tests.

Day 6 — System design

• Design a kitchen ordering system: order states, shelf and courier events, and what happens when a courier never arrives
• Design cart synchronization for a multi-tenant food delivery aggregator that also pushes real-time menu updates, naming the conflict rule for concurrent cart edits
• Outline an S3-like blob store: metadata versus data paths, replication, and the failure you design for
• Read the webhook delivery worked exercise on this page for a reusable at-least-once and retry pattern, then apply it to order ingestion from delivery platforms

Deliverable: Three design one-pagers, each with a data model, a read path, a write path and one named failure mode with its recovery.

Day 7 — Project deep dive, code review and behavioral

• Prepare the past-project deep dive: high-level diagram, data schemas, API contracts, threading model, compromises and failure modes
• Have a friend review your Day 4 simulation as a skeptical code reviewer, and write answers to the three hardest questions they raise
• Prepare stories on shifting constraints mid-project, a technical disagreement with a senior peer, and a project in trouble that you recovered
• Practise an incident troubleshooting exercise: read unfamiliar multi-service docs on a timer and write down the bottleneck and the fix

Deliverable: Deep-dive diagrams, written code-review answers and three behavioral stories with measurable outcomes.

Know where your numbers came from

Reported behavioral questions ask about shifting constraints, disagreements with senior engineers over architecture, and the most complex system you have built. Prepare stories with concrete technical detail: the constraint that changed, the options you weighed, what you decided and how you knew it worked. Reports describe this conversation as happening with an engineering manager, so be ready to explain your own part separately from the team's.

1. Describe a project where the technical constraints shifted midway through development and you had to change direction quickly.
2. Walk through the most complex software system you have built end to end, the key architectural trade-offs, and what you would redesign today.
3. Tell me about a project that was in trouble. What did you own, and how did you get it back on track?
4. Describe a time the requirements changed after you had started building. How did you keep the result correct?

How do you resolve technical disagreements with senior engineering peers when designing system architecture?

Medium · behavioural and engineering judgement

Approach

1. Close with what you would do differently, concretely.
2. Name the disagreement and how you resolved it with evidence.
3. Give the blast radius: what could have broken, and what you measured.

Follow-up

• How did you know your change caused the improvement?
• What did you decide not to do, and why?

Disclose a cross-tenant webhook delivery to affected customers

Medium · cross-tenant leak · disclosure · blast radius · authorisation checks

An enqueue path took the subscription from one lookup and the payload from another. For nineteen minutes, webhook_delivery rows were created whose tenant_id did not match the subscription's tenant, and eleven payloads were signed and sent to four endpoints belonging to other customers. You hold payload_digest, delivery timestamps and response codes. Describe how you handle a disclosure of this kind: what the records prove, what they cannot prove, what you say before you know everything, the one code change that closes it, and which parts you personally drove.

Approach

1. Bound the population before saying anything externally. The affected set is deliveries in the window where the event's tenant and the subscription's tenant differ; the ones that actually left are those with delivered_at set and a 2xx in last_response_code. Attempted and delivered are two different counts and a disclosure has to use the right one in the right sentence.
2. Separate what the records prove from what they do not, and say both halves rather than the flattering one. They prove which payloads were signed, where they went, and — through payload_digest — exactly which bytes. They do not prove what the receiving system did with them, and they do not bound the window more precisely than your deploy timestamps do.
3. Communicate on the facts you hold, with the scope stated as an upper bound: 'at most eleven payloads, four recipient endpoints, these fields, this window' is more useful and more honest than waiting a day for certainty. The field list matters more than the event count, because a customer cannot assess exposure from 'an event'.
4. Name the code change precisely, because this class never originates in the delivery worker. Compare the event's tenant against the subscription's tenant at enqueue and again immediately before the payload is signed, and make the second comparison drop the delivery rather than log a warning. Say why one check is insufficient: the enqueue check protects against the bug you know about, the pre-signing check protects the boundary itself.
5. Run the history question in parallel and say so: a query over historical deliveries for the same mismatch tells you whether this was nineteen minutes or a year, and you would rather find the second case yourself than have a customer find it after your disclosure.
6. Split the response into workstreams with owners — recipients asked to delete, affected customers notified, the check landed with a test, history swept — and say which you personally drove and which you handed off. Claiming all four is not credible and claiming none is not ownership.

Follow-up

• The historical sweep finds two more instances from last year. What changes in what you have already told people?
• Who approves the wording, and what do you do when you are asked to soften the scope?
• A customer asks you to prove a redelivery contained the same bytes as the original. What do you show them?

Argue against failing open when the control plane is unreachable

Hard · revocation · cache staleness · fail-open vs fail-closed · influence

The gateway caches credential-to-context decisions with a sixty-second TTL. A design proposal says that when control-plane reads fail, pods should keep serving from expired entries indefinitely so a control-plane outage never becomes a product outage. You believe that converts every revocation into an unbounded one. Describe a design you argued against while it was still a live proposal: what you measured or modelled to make the case, what you conceded, who decided, and what happened afterwards. Say what would have changed your mind before the decision, not after it.

Approach

1. Reframe it from a values argument into a bounded-staleness argument. Both sides already accept the cache; the disagreement is only about the ceiling on how long a revoked credential keeps authorising. Put a number on the table — serve stale for up to fifteen minutes, then fail closed — and make the other side argue against a number rather than against a principle.
2. Bring arithmetic rather than adjectives: the rate of revocations with revoked_reason in ('suspected_leak','auth_version_bump'), the observed distribution of control-plane unavailability, and the product of the two, which is expected requests served by revoked credentials per outage-hour. At 30k requests/second the unbounded version is not a subtle exposure and the number says so.
3. Concede the strong half of the opposing case first, because that is what buys you the room: failing closed turns one service's outage into a total outage across three regions, and a control plane doing tens of writes per second is not engineered to the gateway's availability target. A proposal you have not steelmanned reads as reflex.
4. Propose the asymmetry that usually resolves this: stale entitlements cost bounded money (a quota fifteen minutes out of date over-serves by a computable amount), while a stale revocation costs unbounded access. Split the cached decision by what it authorises, give the two halves different staleness ceilings, and let the entitlement half fail open while the revocation half fails closed.
5. State the propagation dependency plainly, since it is the part that is missed: validity is also derived from the principal's auth_version, so password reset and sign-out-everywhere flow through this same cache. A design that bounds staleness for explicit revocation and not for auth_version bumps has only fixed half of it.
6. Say who decided, and what you did afterwards in either outcome: write the decision down with its number and a review date, and instrument the exposure you were worried about so the next round of the argument is settled by data instead of by seniority.

Follow-up

• Publish-subscribe invalidation is lossy under a partition, and a TTL is the only hard bound. What TTL do you pick, and what does it cost you at 30k requests/second?
• The key was revoked because it was found in a public repository. Does your answer change, and where does that urgency live in the design?
• You lost the argument and six weeks later the failure you predicted happens. What do you say in the review, and what do you not say?