Interview guides · Engine

Engine Software Engineer Interview Questions & Guide 2026

Engine software engineer prep on operations: decide what you would need to see to know a release is going wrong.

The 60-second brief

As a Software Engineer at Engine, you are at the core of building, scaling, and optimizing high-performance technical systems and infrastructure. Your work directly drives the foundational capabilities that empower teams and customers to interact with complex data, reliable distributed architectures, and modern application layers. Whether you are developing robust backend services, refining query execution paths, or engineering resilient system components, your technical contributions dictate how smoothly products operate at scale.

Most rounds test how you handle an under-specified problem more than what you recall. A wrong first approach you correct out loud is recoverable; a constraint you assumed silently and were never given is the expensive mistake.

Focus areas

Evolve APIs without breaking pinned SDK clients
Make every write idempotent under client retries
Scope every query and cache key by tenant

Overview

This role carries strategic importance for Engine, as the stability and performance of its core software dictate the company's overall business reliability. You will tackle complex engineering challenges involving unmanaged memory environments, database transactions, concurrency control, and distributed systems. The solutions you design must handle massive workloads while maintaining strict fault tolerance, low latency, and uncompromised data integrity.

Succeeding in this position requires a blend of rigorous computer science fundamentals, deep systems-level thinking, and a passion for craftsmanship. You will collaborate closely with product managers, QA specialists, and cross-functional engineering teams to translate ambitious business goals into production-ready code. Expect an intellectually stimulating environment where high standards, rapid iteration, and technical ownership are foundational to everyday engineering.

The Engine interview process

Engine candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening Call

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Technical Interviews

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Behavioral Interviews

Many of these questions are about something that went wrong, and the grading sits mostly in the hours after you knew. Who found out first, whether that was you or an alert or a user, how long it took you to say it out loud, and whether the people who needed the news got it while they could still act on it. Engineers under-tell this part because it feels like confessing. The pattern it is looking for is the opposite: the quiet fix, an incident absorbed without telling anyone, after which nothing changed and the same failure is still available.

What to demonstrate

How the problem was found, and whether that route was one you had built or one that happened to you, since a user reporting it first means your instrumentation did not cover that failure
Whether time-to-detect and time-to-tell are separate numbers in your account and whether you know both, because a fast fix that nobody heard about until the retro is a different answer from a slow one that was announced immediately
Whether the resolution left something durable behind, a check that fires or a default that changed, rather than depending on people remembering to be careful
Whether you can say what the failure cost without either inflating it or waving it away

How to prepare

Reconstruct one incident you were part of as a timeline with clock times: first bad request, first signal, first person who knew, first message outside the team, mitigation, permanent fix. The gaps between those entries are what gets asked about
Look up the configuration of the signal that caught it, including its evaluation window and threshold. An alert defined on a five-minute aggregate cannot fire until the condition holds across that window, which puts a floor under time-to-detect that has nothing to do with how severe the failure was. Be able to say what that floor was and whether anyone had chosen it deliberately
Prepare one story where you escalated early and the severity turned out to be smaller than you thought, including what it cost the people you pulled in. Without it, every answer you give about raising alarms is unfalsifiable

Where candidates lose points

Treating a timed-out write as a failed write

A timeout says the response did not arrive, not that the work did not happen; the server may well have committed and then lost the connection. Retrying a non-idempotent create after a timeout is the standard way to end up with two of something, and those duplicates land precisely when the system is already degraded and least able to absorb them. The discipline is to treat a timeout as unknown: either the write carries an idempotency key so the retry is safe by construction, or the client re-reads authoritative state before deciding what to do, and the interface says unknown rather than showing a failure that invites a second click.

Checking a quota with a select and then writing

Under read-committed isolation, two concurrent transactions both observe a count below the limit and both insert, so the limit is exceeded by exactly the concurrency. Repeatable read does not rescue it either: it provides a stable snapshot, and this is write skew, which snapshot isolation permits by design. The options are serialisable isolation, which detects the conflict and aborts one transaction with a serialisation failure and therefore obliges the caller to retry; a single statement with the predicate inside the write; or a constraint that makes the surplus insert fail outright. The reason this pattern survives review is that it is correct in every test that runs one request at a time.

Writing code before the input contract is pinned down

Before the first line, state the types, the size bounds, whether duplicates, negatives or an empty input are possible, whether the input is sorted, whether you may mutate it, and what the function returns when nothing matches. Every one of those answers changes the code, and discovering one at minute twenty costs a rewrite you no longer have time for.

Arguing past a hint

When the interviewer asks what happens for a particular input or floats a different data structure, stop and take it seriously; it is almost always a correction rather than idle curiosity. Talking over it converts a recoverable wrong turn into a data point about how you handle review.

Questions & practice

Format a complex list and array output based on dynamic input data and custom business formatting rules.

Medium · data structures and algorithms

Approach

State the target complexity and say which constraint rules the naive version out.
Name the brute-force solution and its complexity before improving on it.
Restate the input: its shape, its size, and what is guaranteed about it.

Follow-up

Which test case would catch an off-by-one here?
How does this change if the input no longer fits in memory?

Write clean, well-tested code in an interactive live-coding environment focusing on edge cases and optimal time complexity.

Medium · data structures and algorithms

Approach

Restate the input: its shape, its size, and what is guaranteed about it.
State the target complexity and say which constraint rules the naive version out.
Name the brute-force solution and its complexity before improving on it.

Follow-up

Which test case would catch an off-by-one here?
What is the worst case, and how likely is it on real data?

Design a small React application that integrates with external endpoints like GitHub's API to render real-time scheduling data.

Medium · data structures and algorithms

Approach

State the target complexity and say which constraint rules the naive version out.
Choose the data structure from the access pattern, not from familiarity.
Restate the input: its shape, its size, and what is guaranteed about it.

Follow-up

What is the worst case, and how likely is it on real data?
Which test case would catch an off-by-one here?

Implement a small React and JavaScript application adhering to state management best practices and stateful UI/UX requirements.

Medium · data structures and algorithms

Approach

Walk one small example through your approach before writing the whole thing.
Restate the input: its shape, its size, and what is guaranteed about it.
Choose the data structure from the access pattern, not from familiarity.

Follow-up

What is the worst case, and how likely is it on real data?
Which test case would catch an off-by-one here?

Discuss your familiarity with systems-level frameworks, design patterns, Kubernetes, and containerized deployment models.

Medium · languages, concurrency and fundamentals

Approach

Name what is shared across threads and what owns each piece of state.
Reach for the cheapest primitive that closes the race, not the broadest lock.
Distinguish a value from a reference to it, and say which one you handed out.

Follow-up

What happens if two callers reach this at the same time?
How would you prove the race exists rather than suspect it?

Design a core transaction processing pipeline and explain how you ensure atomicity and durability under heavy concurrent load.

Medium · languages, concurrency and fundamentals

Approach

Say what the runtime actually does before reasoning about the code.
Reach for the cheapest primitive that closes the race, not the broadest lock.
Identify the window where an invariant is briefly untrue.

Follow-up

What happens if two callers reach this at the same time?
Where could this allocate more than you expect?

Locate a billing reconciliation gap without rescanning ninety million events

Hard · reconciliation · dimensional-bisection · watermarks · hypothesis-testing

A tenant's sealed invoice total is 0.4% below the sum of its raw usage_event rows for the period. That tenant has 90 million events over 30 days in a table partitioned daily on ingested_at, and its rollups carry source_max_ingested_at, revision and sealed_at. Recomputing all 30 days from raw is correct, and you are not going to do it. Give the procedure that locates the divergent (workspace, sku, hour) cell, the cost of each probe, and the one query you run before any of it.

Approach

Run the free query first. Sum raw quantity for the period restricted to ingested_at <= source_max_ingested_at of the sealed rollups, and compare that against the unrestricted sum. The rollup stores the watermark precisely so this can be answered without a scan. If the whole 0.4% sits above the watermark, nothing is broken: it is late data, it becomes an adjustment line, and the investigation ends in one query.
Only if the gap survives that test do you bisect, and you bisect by dimension rather than by rows. Compare 30 per-day totals, then inside the offending day compare the 6 SKUs, then the workspaces, then the 24 hours. That is roughly 30 + 6 + W + 24 grouped probes, each an indexed range scan over one daily partition for one tenant, against O(N) per attempt for the naive re-fold.
Quantify why naive is not merely slow but unusable mid-incident: at a generous 200,000 rows/second sequential, 90 million rows is about 7.5 minutes per attempt, you will want ten attempts, and every one competes for I/O on the same partitions live ingest is writing. The diagnostic worsens the backlog it is diagnosing.
Before fetching each comparison, state what it would look like under each hypothesis. Two adjacent hours off by equal and opposite amounts is occurred_at versus ingested_at bucketing. A whole day offset by exactly N hours is a timezone applied at the wrong layer. A gap confined to one SKU in one workspace is an environment filter. The same (tenant_id, idempotency_key) present in two ingested_day partitions is the dedup horizon losing a retry that crossed midnight.
Make the next bisection cheap by storing the aggregate you keep recomputing. A per-(tenant_id, ingested_day) count and quantity checksum turns step two from thirty probes into one read, and it is the same number the reconciliation job already produces.
Whatever you find, the sealed period does not change value. The correction is an adjustment line pointing at the line it reverses, carrying its own source_rollup_watermark, because the original invoice is the evidence of what the customer was charged.

Worked solution (35 min)

Reproduce the shape locally: generate 2 million events for one synthetic tenant over 5 days, fold them, then inject three defects, namely 0.2% of events bucketed by ingested_at, a handful of duplicate idempotency_key values whose retries cross midnight, and a block of events ingested after the seal.
Write the watermark-bounded query first and record how much of the gap it explains on its own.
Bisect by day, then SKU, then hour, recording the probe count and the rows each probe touches.
For each located cell, write down the predicted signature before querying it, then check whether the data matches the prediction.
Time a full re-fold of the 2 million rows and extrapolate to 90 million.

Expected result: The watermark-bounded query accounts for the post-seal block entirely and removes it from the investigation. The ingested_at bucketing appears as adjacent hours off by equal and opposite amounts. The midnight-crossing duplicates appear as one (tenant_id, idempotency_key) present in two ingested_day partitions. Bisection reaches each cell in fewer than 70 probes.

Verify it yourself

The three injected defects sum to the original gap with no unexplained remainder.
Each probe's row count is bounded by one day-partition for one tenant, not by the period.
Removing the source_max_ingested_at bound from the first query makes a gap appear, and grow, while ingest runs, even after every defect is removed.

Follow-up

The gap is 0.4% in one direction on one day and 0.4% the other way the next day. What does that shape rule in, and what does it rule out?
How do you distinguish a duplicate from a restatement, given revision and recomputed_at on the rollup?
Ingest is still running while you investigate. What makes your two numbers comparable at all?

Paginate a tenant's delivery export without skipping rows

Easy · keyset pagination · indexing · multi-tenancy

A customer exports webhook_delivery: delivery_id (bigint identity), subscription_id, tenant_id, event_id, status, attempt_count, next_attempt_at, created_at, delivered_at, updated_at. The endpoint runs select ... where tenant_id = $1 order by created_at desc limit 100 offset $2, and customers report rows missing from exports taken while new deliveries are being inserted. Write the replacement query and the index that supports it, paging a tenant's deliveries newest first at constant cost per page. State why updated_at cannot be the cursor column.

Approach

Name the defect precisely. OFFSET is a position in a result set that is recomputed on every request, so a row inserted ahead of the window shifts everything back by one and the next page starts after a row the client never received. Nothing errors and no identifier gap appears, so the loss is silent.
Replace the position with a value predicate over a stable, unique, indexed ordering: where tenant_id = $1 and (created_at, delivery_id) < ($2, $3) order by created_at desc, delivery_id desc limit 100. The row comparison is load-bearing: created_at alone is not unique, so ties straddling a page boundary are dropped or repeated, which is the same bug in a smaller window.
Index (tenant_id, created_at, delivery_id). PostgreSQL scans a btree in either direction, so an all-DESC ORDER BY is served by an ASC index read backwards and no DESC modifiers are needed; they only matter when the ORDER BY mixes directions. Confirm the plan has no Sort node above the index scan, or the LIMIT stops being an early exit.
Price both forms: keyset is one index descent plus 100 adjacent leaf entries per page, constant regardless of depth, while OFFSET still produces and discards every skipped row, so page N costs time proportional to N times the page size and a deep page on a large table goes from milliseconds to seconds.
Rule out updated_at as the cursor from the precondition, not from taste: a cursor column must never change value for a row already paged past. updated_at moves on every delivery attempt, so a row the client already emitted re-enters a later page and is exported twice. created_at and delivery_id are immutable, which is the whole qualification.

Worked solution (20 min)

Load about 50k deliveries for one tenant, then walk them with the OFFSET query while a writer inserts 10 rows/second, collecting every returned delivery_id.
Compare the distinct ids collected against the set of ids that existed when the walk started, and record the shortfall.
Repeat the walk with the keyset query and confirm every pre-existing id is returned exactly once.
Run explain (analyze, buffers) on page 1 and page 500 of each form and compare shared buffer hits.

Expected result: A keyset query using the row comparison (created_at, delivery_id) < ($2, $3) with order by created_at desc, delivery_id desc limit 100, backed by (tenant_id, created_at, delivery_id) scanned backwards; plus the statement that OFFSET drops rows under concurrent insert and costs time proportional to the offset, and that updated_at is disqualified because it mutates on retry.

Verify it yourself

Buffer counts for page 500 are the same order of magnitude as page 1.
Every id present before the walk is returned exactly once when only inserts run concurrently.
The query plan shows no Sort node above the index scan.
Two rows sharing a created_at value to the microsecond are both returned across the page boundary.

Follow-up

The client wants a snapshot as of one instant rather than a live tail. Compare a repeatable-read transaction held open, an added created_at <= $snapshot bound, and a materialised export table.
A retention job deletes deliveries older than 90 days. What does a client mid-walk see, and does keyset pagination help at all?
The customer wants to resume an export from yesterday's last cursor. What must be true of the cursor for that to be safe?

Rebuild an hourly rollup with deduplication and late-arrival accounting

Hard · window functions · deduplication · event time vs ingest time · watermarks

From usage_event (event_id, tenant_id, workspace_id, environment, sku, quantity numeric(20,6), idempotency_key, occurred_at, ingested_at), produce the values usage_rollup_hourly should hold for one tenant over one day: per (workspace_id, sku, hour_start) the deduplicated quantity_sum, event_count and source_max_ingested_at, bucketed by occurred_at. Duplicates share (tenant_id, idempotency_key). Also report, per hour, the running total across the day and the share of quantity that arrived more than two hours after the hour began. Write the query, and state which duplicates a daily unique index cannot catch.

Approach

Deduplicate in its own CTE before any aggregation, because a SUM cannot be un-summed: row_number() over (partition by tenant_id, idempotency_key order by ingested_at, event_id) = 1. Include the tiebreaker. Without it the surviving row is non-deterministic when two duplicates share an ingested_at, and a rollup described as deterministically recomputable then disagrees with itself between runs.
Bucket on occurred_at and nothing else, and pin the timezone explicitly. date_trunc('hour', timestamptz) truncates in the session's TimeZone setting, so the same query run by a session set to a non-UTC zone buckets differently; use the three-argument date_trunc('hour', occurred_at, 'UTC') on PostgreSQL 16 or later, or date_trunc('hour', occurred_at at time zone 'UTC') at time zone 'UTC' before that. Filter environment = 'production' explicitly, since metering covers three environments and billing covers one.
Aggregate to the grain with sum(quantity), count(*) and max(ingested_at). The last is not decoration: it is the watermark the row consumed up to, and without it there is no way to prove afterwards what a number did and did not include.
Compute the late share inside the dedup-and-aggregate step as a conditional aggregate, sum(quantity) filter (where ingested_at > hour_start + interval '2 hours'), then divide by the hour's total. Compute the running total as a window over the already aggregated rows: sum(quantity_sum) over (partition by workspace_id, sku order by hour_start rows between unbounded preceding and current row). Running either over raw rows puts the duplicates back.
Answer the index question exactly. The unique constraint is on (ingested_day, tenant_id, idempotency_key), because a unique index on a partitioned table must contain the partition key. It therefore deduplicates only within one ingest day and admits a duplicate whose retry crosses midnight or whose replay runs a week later. That is why this CTE dedups across the whole window being recomputed, and why the dedup horizon is a correctness parameter rather than a retention cost.
Keep the numeric type all the way through. quantity is numeric so the sums are exact; a cast to double precision anywhere in this pipeline reintroduces drift that surfaces only as a few unreconcilable cents per tenant per month, long after the query is out of anyone's mind.

Follow-up

A dispute forces the same recompute over 40 days for one tenant. What changes about the dedup CTE's memory use and the chosen plan, and what would you do about it?
Two runs a minute apart return different quantity_sum values for an hour that is already closed. Give two mechanisms that produce that, and the single query that distinguishes them.
Express the same rollup incrementally so it does not re-scan the day each time the watermark advances. What does the incremental version stop being able to answer?

Ship three breaking-looking changes without breaking pinned SDKs

Hard · versioning · backward compatibility · deprecation

GET /v1/runs/{id} returns status from the set queued, leased, running, succeeded, failed, timed_out, cancelled. You must add the terminal state lost, add an optional billable_seconds field, and start rejecting name values over 64 characters that were previously accepted. Clients are generated SDKs pinned inside customer build pipelines you cannot redeploy or reliably contact, some with strict deserialisers. Classify each change as compatible or breaking and justify it mechanically, then give the rollout: the versioning scheme, the transform old clients receive, the telemetry, and the sunset process.

Approach

Classify against a deserialiser rather than against intuition. A new enum value breaks a generated client that validates the enum or switches exhaustively; a new optional field breaks one generated with additionalProperties false; tightening validation breaks every caller already sending an 80-character name even though no field moved. All three are breaking for some real client, and that is the answer, not a technicality.
Choose the versioning axis on maintenance cost. A path version forks the handler, and each subsequent change multiplies the fork. A date-pinned version header, defaulted per credential at first use and pinned thereafter, keeps one internal model and a chain of ordered response transforms, so the handler stays single and the transforms compose.
Write the downgrade honestly. Older versions have no lost, so the transform maps it to the least-wrong existing value and the changelog says which; mapping it to failed asserts an outcome nobody observed, so pair it with a reason field the old version already exposes rather than inventing a field, and keep the mapping in one documented table instead of a conditional inside the serialiser.
Sequence the validation tightening separately from the payload changes: first measure how many tenants send names over 64 characters, then warn for a stated window with a Deprecation header and a non-fatal warning code, then reject only for versions dated after the change, never retroactively for a pinned client.
Deprecate from evidence, not from announcement: per-version, per-tenant, per-credential request counts from the gateway, a Sunset header carrying the date, direct contact for the handful of accounts that make up most of the tail, and short scheduled brownouts before the cutoff so the clients nobody can reach fail on your calendar rather than during their release.

Follow-up

Which step of your rollout cannot be rolled back cleanly once a client has depended on it?
A tenant pinned to a sunset version breaks their build the morning after the cutoff. What in your plan already lets you resolve it within an hour?
Your transform chain is now six versions deep. What stops it becoming the fork you were avoiding?

Specify webhook signature verification a customer can implement

Easy · webhooks · hmac · replay protection

The webhook-delivery service signs each payload before POSTing it to a customer endpoint. Write the signature specification a customer implements in their own language: the header format, exactly which bytes are signed, the algorithm, how replay is bounded, and how a signing secret rotates without a delivery gap. Then write the verification steps the customer performs, in order, including what they compare and what they return on failure. Constraint: most customers reach for their web framework's parsed JSON body by default. Deliverable: the spec section plus reference pseudocode.

Approach

Sign the concatenation of the timestamp and the raw body, t + "." + body, and emit a header of the form t=<unix seconds>,v1=<hex>. The timestamp has to be inside the MAC, or an attacker re-stamps a captured body and the tolerance window buys nothing.
Require the raw request bytes. A framework that parses JSON and re-serialises it changes key order, whitespace and number formatting, so the spec must tell the customer to capture the body before the parser runs and give the middleware note for each common framework.
Use HMAC-SHA256, not sha256(secret || body): SHA-256 is a Merkle-Damgard construction, so the naive form admits length extension. Require a constant-time comparison as well, since a short-circuiting byte compare leaks the expected prefix under repeated probing.
Bound replay in two layers: reject when |now - t| exceeds a stated tolerance such as 300 seconds, then deduplicate on the event identifier header. The tolerance is what makes the customer's dedup store finite rather than unbounded.
Rotate by allowing two live secrets and emitting both signatures in one header (v1=<old>,v1=<new>); the customer accepts if any candidate matches, so neither side needs an instantaneous cutover. A failed verification returns 400 and the body is not processed.

Worked solution (15 min)

Write the header grammar and one real example line with a plausible timestamp and hex digest.
Write the signed string construction explicitly as a byte concatenation, and add the sentence telling the customer where in their framework to obtain the raw body.
Write the five verification steps in order: extract t and candidates, check the tolerance, recompute the HMAC over t + '.' + raw body, compare in constant time against each candidate, then deduplicate on the event identifier.
Add the rotation paragraph: two active secrets, both signatures sent, overlap window stated in the dashboard.
State the failure response and the fact that the payload is not processed, plus what the sender does with that 400.

Expected result: A spec where the signed input is the timestamp joined to the exact raw bytes, the algorithm is HMAC-SHA256 compared in constant time, replay is bounded by a stated tolerance plus event-id deduplication, and rotation is an overlap of two accepted signatures rather than a cutover.

Verify it yourself

The timestamp is inside the signed string, not merely sent alongside it.
The spec names raw bytes and says where the customer gets them before parsing.
Rotation requires no coordinated instant on either side.
The reason for HMAC over sha256(secret || body) is stated as length extension, not as a preference.

Follow-up

A customer's verification passes locally and fails in production behind a proxy that re-encodes the response body. Where do you look first?
Why sign with a per-endpoint secret rather than the tenant's API key?

Metering partition crash-loops and the sealing watermark freezes

Medium · poison message · at-least-once · dead-letter queue

One metering-ingest partition has stopped advancing. Lag grows linearly, the consumer restarts about every 40 seconds, and the same offset appears in every startup log while other partitions stay healthy. Events are committed in batches of a few thousand and the acknowledgement follows the commit. Sealing is six hours away and source_max_ingested_at for that partition's tenants is frozen. Give an ordered checklist, a containment action available within minutes, and the durable fix, saying what each does to exactly-once accounting.

Approach

Distinguish a poison record from a capacity problem in one measurement: compare the offset and the exception across restarts. An identical pair every time is deterministic failure on one record, whereas a throughput problem still advances the offset between crashes.
Read the record from a separate consumer group so the bytes can be inspected without perturbing the stuck consumer, then classify the defect: schema violation, a quantity failing the non-negative check, a null workspace, an unmappable SKU enum, or a payload past a size limit. That classification decides whether this is a producer bug or a missing consumer guard.
Account for batch granularity before acting. With commits of a few thousand, one bad record fails thousands of good ones, so the blast radius is the batch. Halve the batch around the offset to isolate the record, or move to per-record error isolation so the radius becomes the record.
Contain by diverting that record to a dead-letter store with its raw bytes and offset, then resume. This is safe here precisely because the acknowledgement follows the commit: the good records from the failed batch are re-consumed and absorbed by the uniqueness check on (tenant_id, idempotency_key) rather than counted twice.
Make the fix durable with per-record error isolation, a bounded poison counter, and an alert on dead-letter rate rather than on lag alone, since lag only reveals this after the sealing margin has already been eaten.
Check the horizon before replaying anything. The unique index lives on a daily-partitioned table and therefore includes the partition key, so it deduplicates within a day only; a replay landing on a later ingest day needs the separate dedup store or it double-counts into a tenant's bill.

Follow-up

Move the acknowledgement before the commit and describe exactly what is lost and what is duplicated in each of the two crash windows.
Sealing is in six hours and the partition will not drain in time. What do you seal on, and what does the invoice have to record so the difference is explainable later?
A producer replays two weeks of events next month. Which part of your fix stops holding, and what is the dedup horizon you would actually configure?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

What the behavioural round is actually measuring

Every story you tell gets read for blast radius and judgement: what could have broken, who else it touched, what you knew at the moment you decided. Nobody can audit your code in an hour, so they audit your reasoning instead. Pick work where the call was genuinely yours and the consequences were real enough to remember.

Tell me about a recent project you worked on, the technologies you used, and the specific architectural challenges you overcame.

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
State the situation in two sentences and spend the rest on the reasoning.
Give the blast radius: what could have broken, and what you measured.

Follow-up

What did you decide not to do, and why?
How did you know your change caused the improvement?

Why do you believe you are a strong technical and cultural fit for Engine based on your recent career experience?

Medium · behavioural and engineering judgement

Approach

Name the disagreement and how you resolved it with evidence.
Close with what you would do differently, concretely.
Give the blast radius: what could have broken, and what you measured.

Follow-up

How did you know your change caused the improvement?
What did you decide not to do, and why?

Estimate a tenant-leading index migration you have never run

Hard · estimation · online migration · index builds · uncertainty

Someone needs a date. usage_event carries an index on (occurred_at) and needs (tenant_id, occurred_at); the largest tenant holds roughly a hundred times the median tenant's rows, the table is partitioned daily with years of retention, and you have never run a migration on a table this large. Give an estimate you would defend: how you decompose the work, the two or three numbers you would go and measure first, the range and confidence you state, and what you commit to when the person asking needs a single date today.

Approach

Refuse the bare number and then give one anyway, in the form that is actually useful: a range plus the measurement that collapses it. 'Four to eleven days; one afternoon building this index on a restored copy of the largest partition takes that to within a day' is an answer, while 'it depends' is not.
Decompose by failure mode rather than into equal chunks, because that is where estimates go wrong. On a partitioned parent you create the index ON ONLY the parent, build each partition's index with CREATE INDEX CONCURRENTLY, then ALTER INDEX ... ATTACH PARTITION, at which point the parent index becomes valid. CONCURRENTLY does not block writes but scans each partition twice, waits out older transactions, cannot run inside a transaction block, and on failure leaves an invalid index you must drop concurrently and retry.
Name the two unknowns that dominate and price them: build time on one restored partition of realistic size, and whether the planner actually chooses the new index for the skewed tenant, since selectivity for a tenant holding most of the rows is a different question from selectivity for the median tenant. Both are half-day measurements against a replica, and both are cheaper than being wrong by a week.
State the assumptions the range is conditional on, because that is what makes a slip a re-estimate instead of a credibility event: no partition above a stated row count, one concurrent build at a time so it does not compete with ingest for I/O, and an ingest backlog that can absorb the added write amplification while both indexes exist.
Budget the step nobody budgets: verification and the old index's removal. Dropping the old index is fast, but deciding it is safe to drop means confirming no plan still uses it, and that confirmation waits on real traffic across a full weekly cycle rather than on your patience.
Answer the single-date request honestly. Commit to a date for the first checkpoint — the measured build number from the replica — and to re-estimating on that date, and say plainly what you are not committing to yet. A date with a scheduled re-estimate is worth more to the asker than a confident wrong one, and you should say why in those words.

Follow-up

The concurrent build fails half way through the largest partition. What is the state of the database and what do you do next?
Your estimate slips by sixty percent. Which assumption broke, and at what point would you have known?
The person asking needs the date for a customer commitment. Does your answer change?