Interview guides · Baseten

Baseten Software Engineer Interview Questions & Guide 2026

Baseten software engineer prep on operations: decide what you would need to see to know a release is going wrong.

The 60-second brief

As a Software Engineer at Baseten, you play a foundational role in powering mission-critical inference and training for the world's most dynamic AI companies. Operating at the bleeding edge of applied artificial intelligence, you build the scalable infrastructure, ergonomic developer tooling, and flexible platforms that allow organizations to seamlessly transition cutting-edge models from research into production. Your work directly enables teams across industries to deploy, scale, and monitor complex machine learning workloads with high performance and reliability.

Most rounds test how you handle an under-specified problem more than what you recall. A wrong first approach you correct out loud is recoverable; a constraint you assumed silently and were never given is the expensive mistake.

Focus areas

Make every write idempotent under client retries
Scope every query and cache key by tenant
Build at-least-once pipelines with explicit deduplication horizons

Overview

This role sits at the intersection of distributed systems, high-performance computing, and developer experience. Whether you focus on core product features, infrastructure scaling, model APIs, or training systems, you will tackle hard engineering challenges such as multi-node orchestration, GPU utilization, and low-latency serving. The scope of impact is immense; your code drives the systems that handle frontier models, multi-component workflows, and massive developer traffic.

Working at Baseten means joining a fast-paced, high-impact team backed by top-tier investors and fueled by rapid growth. You will collaborate closely with peers who value technical excellence, product taste, and pragmatic problem-solving. Expect an environment where you are trusted to take ownership of complex components from day one, balancing rigorous performance optimization with a deep empathy for developer workflows.

The Baseten interview process

Baseten candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Technical Phone Screen

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Virtual Onsite Loop

Coding rounds mostly set a floor. They decide whether you clear the bar, not where you land on the ladder. Level tends to come out of the design discussion and the ownership stories, so the question worth auditing beforehand is whether the scope you describe matches the scope of the job. Work that stops at your own service, or a story whose hard part was writing the code rather than getting several people to agree on an interface, reads a level below where you think you are interviewing, and that gap is usually resolved downwards.

What to demonstrate

Whether the largest thing you describe owning ran end to end — the decision, the migration path, the rollout, and what you did when it went wrong — or stopped at the change you merged
Whether design answers include what you would not build, what you would defer, and what you would measure before committing, rather than only what the boxes are
Whether a disagreement in a story was settled with something checkable — a benchmark, a prototype, a written proposal — instead of by seniority or by waiting it out
Whether you can say which calls you made alone and which you escalated, and why the line sat where it did

How to prepare

Write your largest piece of owned work as a timeline of decisions — who decided what, when, and what you did when the plan broke — then delete every sentence whose subject is "we" and see how much survives
Take one system you know well and drill the migration answer: how old and new paths run side by side under live traffic, how you compare their outputs, what the rollback is once writes are going to both, and which step you would not automate
Map each line of the ladder in the job posting to a specific thing you have done, find the line you cannot support, and prepare the closest evidence you have plus an honest account of the gap

Where candidates lose points

Treating a timed-out write as a failed write

A timeout says the response did not arrive, not that the work did not happen; the server may well have committed and then lost the connection. Retrying a non-idempotent create after a timeout is the standard way to end up with two of something, and those duplicates land precisely when the system is already degraded and least able to absorb them. The discipline is to treat a timeout as unknown: either the write carries an idempotency key so the retry is safe by construction, or the client re-reads authoritative state before deciding what to do, and the interface says unknown rather than showing a failure that invites a second click.

Paginating a growing table with limit and offset

Two unrelated defects share the idiom. Correctness: rows inserted or deleted between page requests shift the window, so a consumer walking an export skips rows and sees others twice, which for a customer-facing sync is silent data loss rather than an error anyone notices. Cost: the database still produces and discards the skipped rows, so page N costs time proportional to N times the page size and a deep page on a large table degrades from milliseconds to seconds. Keyset pagination over a stable, unique, indexed ordering -- where (created_at, id) < ($1, $2) order by created_at desc, id desc limit $3 -- is constant-cost per page and immune to shifting, on the precondition that the cursor columns never change value for a row, which disqualifies updated_at as a cursor.

Never running a concrete value through the code

Trace one small input and one edge input by hand, index by index, out loud. Re-reading your own code catches design mistakes; walking a real value through it catches the off-by-one, the uninitialised accumulator and the loop that never advances.

Sharing mutable state with no stated owner

Say which thread, request or task owns each mutable structure, and what protects it when the answer is more than one: a lock, a queue that hands ownership across, or an immutable copy per reader. A structure documented as safe for concurrent reads is usually not safe for a concurrent write alongside those reads.

Questions & practice

How would you design a benchmarking framework to measure real-world performance across varying hardware configurations and sequence lengths?

Medium · languages, concurrency and fundamentals

Approach

Distinguish a value from a reference to it, and say which one you handed out.
Reach for the cheapest primitive that closes the race, not the broadest lock.
Identify the window where an invariant is briefly untrue.

Follow-up

Where could this allocate more than you expect?
How would you prove the race exists rather than suspect it?

Parse and verify a timestamped multi-signature webhook header

Easy · parsing · hmac · constant-time-compare · replay-protection

An inbound webhook carries a signature header of at most 1 KiB shaped t=<unix seconds>,v1=<64 hex chars>, with up to five v1 values during secret rotation and possibly unknown scheme keys. You hold the raw request body bytes and the currently active signing secrets. Write the parser and the verifier: accept when any active secret reproduces a signature and the timestamp is within a five-minute tolerance in either direction, reject otherwise. Single left-to-right pass over the header, no regular expression. State what is inside the MAC and why.

Approach

Parse in one scan: split on ,, then on the first = only, since a value may itself contain = under a future scheme. Accept t exactly once and treat a second t as a reject rather than last-wins. Push every v1 onto a short list and ignore any other key, so a v2 can be introduced later without breaking this verifier.
Say what is signed: HMAC-SHA256 over the exact byte string <t>.<raw body bytes>, yielding 32 bytes or 64 hex characters. The timestamp sits inside the MAC because otherwise an attacker replays yesterday's body with its still-valid signature and only has to edit the header timestamp.
Hash the bytes as received. Verifying against a re-serialised JSON body is the usual defect: key order, whitespace and number formatting all change the bytes while the parsed objects compare equal, so signatures fail for honest senders and the popular 'fix' is to stop checking.
Compare in constant time over fixed-length digests. Decode the hex to 32 bytes, accumulate acc |= a[i] ^ b[i] across the whole length, and test acc == 0 at the end. Evaluate every candidate without an early exit; at five candidates that is five HMACs over the body, linear in body size and negligible beside the network.
Apply the tolerance as a two-sided bound, rejecting when |now - t| > 300 seconds. A sender whose clock runs ahead of yours is an ordinary case, and an unbounded future timestamp is a free replay window.
Complexity: O(L) over the header producing k candidates, plus k HMACs at O(|body|) each. Space is O(k) beyond the body itself. Do the cheap rejections, including the tolerance check, before any cryptography runs.

Worked solution (15 min)

Write the grammar on one line before coding: header := field (',' field)*, field := key '=' value, split on the first = only.
Implement the parser to return {t: int, v1: [hex, ...]}, rejecting a missing t, a duplicate t, any v1 that is not 64 hex characters, and a header over 1 KiB, all before any cryptography runs.
Implement the verifier: for each active secret compute HMAC-SHA256(secret, f'{t}.'.encode() + raw_body), compare it in constant time against each parsed v1, and OR the results with no early exit.
Test with a valid signature; the same body with t moved 400 seconds into the past; the same body with t 400 seconds into the future; a header carrying an unknown v2= alongside a valid v1; and a body re-serialised with different JSON key order.

Expected result: The valid case accepts. Both out-of-tolerance cases reject, including the future one. The unknown v2 field is ignored and the v1 still verifies. The re-serialised body fails, which is correct and is exactly why the raw bytes must be retained.

Verify it yourself

Timing the comparison over 10,000 mismatching signatures that differ at the first character versus the last shows no separation you can measure.
Removing t from the MAC input makes an old captured body with its original signature pass. Write that test, watch it pass, then put t back.
Every rejection except the signature mismatch itself returns before an HMAC is computed.

Follow-up

The body is 40 MB. What changes about where you verify, and what can you do before the whole body has arrived?
A customer reports that signatures fail for exactly the requests whose body contains a non-ASCII character. What is your first hypothesis?
How do you rotate the signing secret with no failed deliveries, and how long do both secrets stay live?

Schedule ordered webhook retries with a heap of subscription queues

Medium · heap · scheduling · backoff · head-of-line-blocking

Design the in-memory scheduler for webhook delivery. Up to 20 million rows sit in status pending or failed_retryable across 200,000 subscriptions, each row carrying next_attempt_at and attempt_count, and each endpoint having a circuit breaker. Deliveries for one subscription must be attempted in order, so at most one attempt per subscription may be in flight. Support due(now), complete(delivery, outcome) and insert(delivery) in O(log S), where S is the subscription count rather than the delivery count. Give the backoff formula you schedule retries with.

Approach

Key the global heap by subscription, not by delivery. Each subscription owns a FIFO of its due deliveries in event order; the heap holds one entry per eligible subscription, keyed by its head's next_attempt_at. That is 200,000 heap entries instead of 20 million, and it makes the one-in-flight rule structural rather than a check somebody can forget.
due(now): peek the minimum. If its key is in the future, sleep until then instead of spinning. Otherwise pop it, move the subscription into an in-flight set, and do not re-push it. A subscription absent from the heap cannot be dispatched twice, which is precisely how ordering is preserved.
complete: on success, drop the head and re-push the subscription keyed by its new head, or leave it out when the queue empties. On a retryable failure, increment attempt_count and set next_attempt_at = now + uniform(0, min(cap, base * 2^attempt)), sampled uniformly across the whole interval. That is full jitter; deterministic backoff re-synchronises the herd you just created.
Circuit breaker: park the subscription in a second heap keyed by its half-open time, so an endpoint dead for six hours costs one heap entry and zero attempts rather than consuming worker slots. Admit exactly one probe at half-open and close the breaker only on its success.
Say the price of the ordering guarantee out loud. One in-flight attempt per subscription means an endpoint answering in 10 seconds drains at 0.1 deliveries/second however many workers you run, and its backlog grows until it recovers. If the customer does not need order, allow k in flight and document delivery as unordered; that is the trade, and it is a product decision.
All three operations are O(log S) with O(S) resident heap memory and the queues themselves backed by the store. The database-backed equivalent is a partial index on (subscription_id, next_attempt_at) where status in ('pending','failed_retryable') claimed with FOR UPDATE SKIP LOCKED, and the write-back must be fenced on lease_token so a worker that stalled and resumed cannot overwrite a newer attempt.

Follow-up

One subscription has 4 million queued deliveries. What stops it from starving the other 199,999, and what does your heap look like under that load?
A customer requests redelivery of last Tuesday's events. Where do those rows enter your structure, and what keeps them from reordering live traffic?
The process restarts. How much state do you rebuild, and what stops every subscription from being attempted in the same second?

Paginate a tenant's delivery export without skipping rows

Easy · keyset pagination · indexing · multi-tenancy

A customer exports webhook_delivery: delivery_id (bigint identity), subscription_id, tenant_id, event_id, status, attempt_count, next_attempt_at, created_at, delivered_at, updated_at. The endpoint runs select ... where tenant_id = $1 order by created_at desc limit 100 offset $2, and customers report rows missing from exports taken while new deliveries are being inserted. Write the replacement query and the index that supports it, paging a tenant's deliveries newest first at constant cost per page. State why updated_at cannot be the cursor column.

Approach

Name the defect precisely. OFFSET is a position in a result set that is recomputed on every request, so a row inserted ahead of the window shifts everything back by one and the next page starts after a row the client never received. Nothing errors and no identifier gap appears, so the loss is silent.
Replace the position with a value predicate over a stable, unique, indexed ordering: where tenant_id = $1 and (created_at, delivery_id) < ($2, $3) order by created_at desc, delivery_id desc limit 100. The row comparison is load-bearing: created_at alone is not unique, so ties straddling a page boundary are dropped or repeated, which is the same bug in a smaller window.
Index (tenant_id, created_at, delivery_id). PostgreSQL scans a btree in either direction, so an all-DESC ORDER BY is served by an ASC index read backwards and no DESC modifiers are needed; they only matter when the ORDER BY mixes directions. Confirm the plan has no Sort node above the index scan, or the LIMIT stops being an early exit.
Price both forms: keyset is one index descent plus 100 adjacent leaf entries per page, constant regardless of depth, while OFFSET still produces and discards every skipped row, so page N costs time proportional to N times the page size and a deep page on a large table goes from milliseconds to seconds.
Rule out updated_at as the cursor from the precondition, not from taste: a cursor column must never change value for a row already paged past. updated_at moves on every delivery attempt, so a row the client already emitted re-enters a later page and is exported twice. created_at and delivery_id are immutable, which is the whole qualification.

Follow-up

The client wants a snapshot as of one instant rather than a live tail. Compare a repeatable-read transaction held open, an added created_at <= $snapshot bound, and a materialised export table.
A retention job deletes deliveries older than 90 days. What does a client mid-walk see, and does keyset pagination help at all?
The customer wants to resume an export from yesterday's last cursor. What must be true of the cursor for that to be safe?

Migrate a live partitioned event table without blocking ingest

Hard · online migration · ddl locks · expand and contract · partitioning

usage_event is range-partitioned daily on ingested_at, holds roughly 250M rows per day across 400 live partitions, and is written at 10-40k rows/second. Two changes are required: quantity must move from double precision to numeric(20,6), and a new environment column must become NOT NULL with a default of 'production'. Ingest cannot stop. Give the ordered plan, naming for each step the lock it takes, what that lock blocks, and roughly how long it is held. Identify the one step that cannot be rolled back cleanly once traffic depends on it.

Approach

Classify the two changes before planning anything. Adding a column with a non-volatile default has been metadata-only since PostgreSQL 11, so it is cheap. Changing double precision to numeric is not binary-coercible, so alter column ... type rewrites every partition under ACCESS EXCLUSIVE and rebuilds its indexes; on this volume that is hours of blocked ingest and is simply not an option, which is why the plan is expand-and-contract rather than one statement.
Expand: add quantity_numeric numeric(20,6) and environment with its default on the parent. Both are catalogue-only but both take a brief ACCESS EXCLUSIVE that cascades to partitions, so run each with lock_timeout set to a second or two and retry on failure. A queued ACCESS EXCLUSIVE request blocks every reader behind it, which is how a metadata-only change turns into an outage.
Dual-write: deploy producer code that populates both columns on every insert, and leave it running before anything reads the new column. This is the step that cannot be reverted cleanly. Once readers depend on quantity_numeric, reverting the writer leaves rows with a null there, and the gap is only discoverable by re-reading the old column, which the readers have stopped doing.
Backfill older partitions in batches keyed on the primary key, oldest first, committing every few thousand rows with a pause between batches, and skipping the partition still receiving writes until it rotates. Each batch is an ordinary UPDATE taking row locks only. The cost is bloat and WAL rather than blocking, so watch dead tuples and let autovacuum keep pace instead of wrapping 400 partitions in one transaction.
Make NOT NULL cheap with the three-step form: add constraint ... check (environment is not null) not valid (brief ACCESS EXCLUSIVE, no scan), then validate constraint (SHARE UPDATE EXCLUSIVE, scans while reads and writes continue), then set not null, which from PostgreSQL 12 uses the validated check and skips its own full scan. Do this per partition, then on the parent.
Switch and contract: move reads to the new column behind a flag, verify over a full period that both columns agree on freshly written rows, drop the old column (metadata-only), and only then remove the dual-write. Any index on the new column goes on with CREATE INDEX CONCURRENTLY per partition, since CIC is not supported on a partitioned parent: create the parent index with ONLY, build each child concurrently, then ALTER INDEX ... ATTACH PARTITION until the parent index becomes valid.

Worked solution (45 min)

On a scratch cluster, build 10 partitions of 2M rows each and run a writer at a few thousand inserts/second.
Run the naive type change and measure how long writes stall and how far ingest lag grows before killing it.
Run the expand step with lock_timeout = '2s' while the writer runs, and observe a clean lock timeout and retry instead of a pile-up of blocked readers.
Backfill in 5k-row batches and chart dead tuples and WAL generated per batch.
Run the not-valid, validate, set-not-null sequence and confirm from pg_stat_activity and timings that nothing held an exclusive lock through a full scan.
Add an index with CIC per partition plus ATTACH PARTITION and confirm the parent index reports valid only after the last attach.

Expected result: An ordered expand, dual-write, backfill, switch, contract plan with the lock named per step: metadata-only ADD COLUMN under a brief ACCESS EXCLUSIVE taken with lock_timeout; batched UPDATE backfill under row locks; CHECK NOT VALID, then VALIDATE under SHARE UPDATE EXCLUSIVE, then SET NOT NULL; CREATE INDEX CONCURRENTLY per partition with ATTACH PARTITION; and the dual-write-to-read switch identified as the step that cannot be reverted cleanly.

Verify it yourself

No step in the plan holds ACCESS EXCLUSIVE for longer than the configured lock_timeout.
The writer's error rate stays at zero through the rehearsal apart from the deliberate lock timeout.
After VALIDATE, SET NOT NULL completes in milliseconds on a table where the naive form scanned every row.
Freshly written rows agree on both columns for a full verification period before the old column is dropped.

Follow-up

A CREATE INDEX CONCURRENTLY fails halfway through the partition list. What state is the table in, how do you detect it, and what do you run?
The producer computes quantity itself. What happens to a request already in flight when the dual-write deploy lands, and does it matter?
Give two queries that prove the backfill is complete: one cheap enough to run every minute, one authoritative.

How do you approach rate-limiting, authentication, and usage metering for a high-throughput, low-latency API gateway?

Medium · system design and architecture

Approach

Fix the scope first: who calls this, how often, and what they do when it fails.
State the consistency you need, and where you are willing to be stale.
Name the failure you are designing for, then the recovery path.

Follow-up

What breaks first when traffic grows ten times?
What would you drop to keep the system up under load?

How would you design a multi-cloud capacity management system to handle sudden spikes in inference traffic?

Medium · system design and architecture

Approach

Name the read and write paths separately; they rarely have the same bottleneck.
Name the failure you are designing for, then the recovery path.
Choose a partition key and say what query it makes expensive.

Follow-up

How does this behave when that dependency is down for an hour?
What breaks first when traffic grows ten times?

Explain how you would troubleshoot a memory leak or bottleneck in a distributed training setup using frameworks like PyTorch FSDP or ZeRO.

Medium · system design and architecture

Approach

State the consistency you need, and where you are willing to be stale.
Choose a partition key and say what query it makes expensive.
Fix the scope first: who calls this, how often, and what they do when it fails.

Follow-up

How does this behave when that dependency is down for an hour?
What would you drop to keep the system up under load?

Walk through your approach to building a CLI tool or SDK that developers will find intuitive and reliable.

Medium

Approach

Clarify what is being asked and what a complete answer contains.
Work from the requirement backwards to the design.
Say what you would check first and why it is the highest-information step.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

How do you balance rapid feature delivery with maintaining system reliability and addressing customer issues with urgency?

Medium

Approach

Clarify what is being asked and what a complete answer contains.
State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Metering ingest that survives a six-hour producer replay

Hard · idempotency · dedup horizon · partitioning · watermarks

metering-ingest consumes usage events at-least-once - 250M/day, 10-40k/second at peak - and folds them into usage_rollup_hourly keyed (tenant_id, workspace_id, sku, hour_start). usage_event is partitioned daily on ingested_at with unique (ingested_day, tenant_id, idempotency_key). A producer outage ends in a six-hour replay that re-sends events already ingested, some of whose originals crossed midnight. Design the consumer: partitioning, where the acknowledgement sits relative to the commit, the deduplication horizon and its storage cost, and how the rollup watermark advances. Nothing may be double-counted and nothing may be silently dropped.

Approach

Choose the acknowledgement position deliberately and name what each choice costs. Acknowledging after the fold commits makes the consumer at-least-once: a crash between the two replays the batch and produces duplicates, which are ordinary and absorbable. Acknowledging first makes it at-most-once: a crash between the two drops revenue with no error raised anywhere and no way to detect it later. Take at-least-once and design everything downstream to absorb duplicates.
Put the dedup and the fold in one transaction so there is no window between them. Insert the batch into usage_event with ON CONFLICT DO NOTHING, take the rows actually inserted, and fold only those into usage_rollup_hourly with an upsert on (tenant_id, workspace_id, sku, hour_start) bucketed by occurred_at, not ingested_at. A batch of about 2,000 rows is one round trip and one index probe per event.
Attack the partition-key flaw head on: the unique index includes ingested_day because a unique index on a partitioned table must contain the partition key, so the same (tenant_id, idempotency_key) re-sent after midnight is a different index entry and passes. Deduplicate instead against a store keyed (tenant_id, idempotency_key) with no date component, whose horizon exceeds the producer's maximum retry window plus the longest replay you intend to support. At 14 days that is 250M x 14 = 3.5 billion keys, which is a dedicated key-value store, not a larger index on the same table. The alternative - partitioning usage_event on (tenant_id, occurred_day) so the natural key is stable - fixes dedup but loses pruning on ingest time and makes retention by dropping partitions awkward.
Partition the consumer by hash of tenant_id so one tenant's replay stalls only its own partitions, and give replay traffic a separate lower-priority lane so live ingest keeps its latency. The cost is explicit: that tenant's watermark lags while the replay drains, and everything gated on the watermark waits for it.
Define the watermark as a property of committed work, not of wall-clock time: per partition it is the largest occurred_at such that every event with a smaller occurred_at has committed, and the sealing decision uses the minimum across partitions. Record source_max_ingested_at on every rollup row so any number can prove what it did and did not include, and keep restatement legal only while status = 'open' - after sealed_at the value is frozen and a late event becomes an invoice adjustment instead.

Worked solution (40 min)

Write the consumer loop in pseudocode with the acknowledgement after the commit, then annotate each line with what is lost or duplicated if the process dies exactly there.
Size the dedup store: events/day x horizon_days keys, bytes per key including the tenant prefix, and the resulting memory or disk. Compare that cost against simply extending retention on the partitioned table and say why the latter does not fix the problem.
Take one event ingested at 23:59:58 and replayed at 00:00:04 and work out its fate under (a) the partitioned unique index alone and (b) the separate dedup store.
Write the per-partition watermark formula, then what the seal uses, then what a single stalled partition does to sealing.

Expected result: An at-least-once consumer acknowledging after the commit; deduplication on (tenant_id, idempotency_key) in a store whose horizon of at least 14 days exceeds max retry window plus supported replay length, sized at roughly 3.5 billion keys; tenant-hashed partitions with replay on a separate lane; a per-partition watermark with sealing gated on the minimum across partitions; source_max_ingested_at written on every rollup row.

Verify it yourself

The midnight-crossing retry must be rejected by whatever you chose. If the table's unique index is your only dedup, it is accepted and the invoice is wrong by exactly that event.
You should be able to name a distinct failure for acknowledge-before-commit and for acknowledge-after-commit. If both sound the same, the position was not actually chosen.
quantity_sum for any (tenant, workspace, sku, hour) must be reproducible by re-aggregating usage_event filtered to ingested_at <= source_max_ingested_at. If it is not reproducible, the rollup is an opinion rather than a projection.

Follow-up

The dedup store is lost entirely. What can you still guarantee, and how do you rebuild it from what remains?
A replay delivers events for an hour that is already sealed. Trace exactly what happens to them, row by row.
One partition is stuck on a poison message, so the minimum-across-partitions watermark never advances and no tenant can be sealed. What is your escape hatch and what does it cost in correctness?

Hourly rollups merge one hour and lose another

Easy · timezone · rollups · reconciliation

Reconciliation flags one tenant on one day. Summing usage_event.quantity by hour of occurred_at gives 24 non-empty hours, but usage_rollup_hourly holds 23 rows for that tenant, workspace and SKU, one of which carries roughly the sum of two adjacent hours. Other days reconcile exactly, and the affected date matches a civil-time transition. hour_start is documented as truncated to the hour in UTC. You have both tables, the rollup job source, and its runtime environment. Give an ordered checklist, the mechanism, and the correction path for a day that may already be sealed.

Approach

Bisect by dimension until one cell explains the whole difference: tenant, then day, then SKU, then hour. A defect confined to a single transition date already rules out deduplication and late arrival, both of which are indifferent to which hour an event lands in.
Read the truncation with its precondition stated: date_trunc on a timestamptz value is evaluated in the session TimeZone, not in UTC. If the job connects without pinning that setting, it inherits the server or container default.
Follow that to the collision: in a zone that observes daylight saving, two distinct UTC hours map to the same local wall-clock label at the autumn transition, so both fold into one key under the unique constraint on (tenant_id, workspace_id, sku, hour_start) and their quantities sum into one row. At the spring transition a label never occurs and the row is simply absent.
Confirm from data rather than from reading code: run the same aggregate twice, once with the session pinned to UTC and once with the job host zone, and check that the second reproduces the stored rollup exactly.
Fix at the source by pinning the connection to UTC explicitly, or by truncating on occurred_at AT TIME ZONE 'UTC', rather than relying on a default that differs between a developer machine, CI and production.
Correct according to status, not convenience: an open hour is recomputed with revision incremented, a sealed hour is frozen and the difference becomes an adjustment line on the next invoice with voided_by_line_id pointing at the line it reverses.

Follow-up

The same job also emits a daily figure for a dashboard. Why can a correct hourly rollup still produce a wrong day, and what does the tenant's billing timezone have to do with it?
How would you detect this class automatically rather than waiting for reconciliation, given that it only manifests twice a year per zone?

Suggested practice checklist

Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.

Day 1 — Diagnostic, scored before you study anything

Sit a 110-minute diagnostic in four blocks: forty-five minutes on two coding problems, twenty-five on one design prompt taken to interface and data model, twenty of short-answer fundamentals, and twenty delivering two behavioural answers aloud.
Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct and 0 is stuck, grading the artifact rather than how the attempt felt.
Allocate days two to five in proportion to 3 minus each block's score, write the allocation down, and commit to leaving it alone.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

Split the weakest area into named sub-skills and rate each separately. For coding those are restating the problem, choosing the structure, stating the invariant, turning the invariant into loop bounds, handling empty and single-element input, and accounting for complexity out loud.
Attempt three items positioned just above where the rating drops off, and for each write the first move you failed to make.
Re-attempt one of them from blank four hours later with nothing open.

Deliverable: A sub-skill map with the two blocking sub-skills circled.

Day 3 — Drill the blocking sub-skill by repeating the shape

Do eight short repetitions of the same shape rather than eight different problems, so what gets practised is the pattern and not the puzzle.
State the rule you now hold in one sentence, then test it against a case built to break it, a sliding window over an array containing negative values, or a cache-aside read path whose invalidation message is dropped.
Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on the strongest area

Run the same sub-skill decomposition on the second-largest gap in half the time.
Spend twenty-five timed minutes on the block you scored highest, choosing the hardest item you can still finish rather than a warm-up.
Write whether each area fails you on recall, on setup, or on execution, and set the fix accordingly: repetition for recall, a written checklist for setup, timed work for execution.

Deliverable: A second sub-skill map plus a one-line failure diagnosis for each area.

Day 5 — The gap that is not a skill

Record one technical and one behavioural answer, then count two things in the playback: seconds before your first clarifying question, and sentences you began without knowing where they would end.
Practise saying that you do not know, followed by how you would find out, without letting it soften into a guess, and practise stating a complexity or an estimate before being asked for it.
Redeliver one answer under a hard ninety-second cap, which forces structure ahead of detail.

Deliverable: Two recordings with a counted reduction in time-to-first-question.

Day 6 — Retest under day-one conditions

Sit the same 110-minute structure with new prompts of comparable difficulty and score it on the identical rubric.
For any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
Write down which single block you would still lose the offer on.

Deliverable: A second scored rubric placed beside the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

Run a sixty-minute mock over the two blocks that moved least, with an interviewer briefed to interrupt and change direction mid-answer.
Write the recovery script for going blank: restate the question, state your assumption, name the first thing you would check.
Say every rule from the week aloud without reading it, and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive an interruption.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

The estimate and the day you knew

A slipped date is only a bad story if you sat on it. What matters is what you believed when you gave the number, the signal that told you it was wrong, how many days passed before you said so, and what you cut rather than asking for more time. Scope you defended counts as much as scope you dropped.

Tell me about a time you had to debug a complex system failure across multiple layers, from runtime internals to infrastructure.

Medium · behavioural and engineering judgement

Approach

Give the blast radius: what could have broken, and what you measured.
Pick a story where you made the decision, not one where you watched it.
Close with what you would do differently, concretely.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?

Estimate a tenant-leading index migration you have never run

Hard · estimation · online migration · index builds · uncertainty

Someone needs a date. usage_event carries an index on (occurred_at) and needs (tenant_id, occurred_at); the largest tenant holds roughly a hundred times the median tenant's rows, the table is partitioned daily with years of retention, and you have never run a migration on a table this large. Give an estimate you would defend: how you decompose the work, the two or three numbers you would go and measure first, the range and confidence you state, and what you commit to when the person asking needs a single date today.

Approach

Refuse the bare number and then give one anyway, in the form that is actually useful: a range plus the measurement that collapses it. 'Four to eleven days; one afternoon building this index on a restored copy of the largest partition takes that to within a day' is an answer, while 'it depends' is not.
Decompose by failure mode rather than into equal chunks, because that is where estimates go wrong. On a partitioned parent you create the index ON ONLY the parent, build each partition's index with CREATE INDEX CONCURRENTLY, then ALTER INDEX ... ATTACH PARTITION, at which point the parent index becomes valid. CONCURRENTLY does not block writes but scans each partition twice, waits out older transactions, cannot run inside a transaction block, and on failure leaves an invalid index you must drop concurrently and retry.
Name the two unknowns that dominate and price them: build time on one restored partition of realistic size, and whether the planner actually chooses the new index for the skewed tenant, since selectivity for a tenant holding most of the rows is a different question from selectivity for the median tenant. Both are half-day measurements against a replica, and both are cheaper than being wrong by a week.
State the assumptions the range is conditional on, because that is what makes a slip a re-estimate instead of a credibility event: no partition above a stated row count, one concurrent build at a time so it does not compete with ingest for I/O, and an ingest backlog that can absorb the added write amplification while both indexes exist.
Budget the step nobody budgets: verification and the old index's removal. Dropping the old index is fast, but deciding it is safe to drop means confirming no plan still uses it, and that confirmation waits on real traffic across a full weekly cycle rather than on your patience.
Answer the single-date request honestly. Commit to a date for the first checkpoint — the measured build number from the replica — and to re-estimating on that date, and say plainly what you are not committing to yet. A date with a scheduled re-estimate is worth more to the asker than a confident wrong one, and you should say why in those words.

Follow-up

The concurrent build fails half way through the largest partition. What is the state of the database and what do you do next?
Your estimate slips by sixty percent. Which assumption broke, and at what point would you have known?
The person asking needs the date for a customer commitment. Does your answer change?

Argue against failing open when the control plane is unreachable

Hard · revocation · cache staleness · fail-open vs fail-closed · influence

The gateway caches credential-to-context decisions with a sixty-second TTL. A design proposal says that when control-plane reads fail, pods should keep serving from expired entries indefinitely so a control-plane outage never becomes a product outage. You believe that converts every revocation into an unbounded one. Describe a design you argued against while it was still a live proposal: what you measured or modelled to make the case, what you conceded, who decided, and what happened afterwards. Say what would have changed your mind before the decision, not after it.

Approach

Reframe it from a values argument into a bounded-staleness argument. Both sides already accept the cache; the disagreement is only about the ceiling on how long a revoked credential keeps authorising. Put a number on the table — serve stale for up to fifteen minutes, then fail closed — and make the other side argue against a number rather than against a principle.
Bring arithmetic rather than adjectives: the rate of revocations with revoked_reason in ('suspected_leak','auth_version_bump'), the observed distribution of control-plane unavailability, and the product of the two, which is expected requests served by revoked credentials per outage-hour. At 30k requests/second the unbounded version is not a subtle exposure and the number says so.
Concede the strong half of the opposing case first, because that is what buys you the room: failing closed turns one service's outage into a total outage across three regions, and a control plane doing tens of writes per second is not engineered to the gateway's availability target. A proposal you have not steelmanned reads as reflex.
Propose the asymmetry that usually resolves this: stale entitlements cost bounded money (a quota fifteen minutes out of date over-serves by a computable amount), while a stale revocation costs unbounded access. Split the cached decision by what it authorises, give the two halves different staleness ceilings, and let the entitlement half fail open while the revocation half fails closed.
State the propagation dependency plainly, since it is the part that is missed: validity is also derived from the principal's auth_version, so password reset and sign-out-everywhere flow through this same cache. A design that bounds staleness for explicit revocation and not for auth_version bumps has only fixed half of it.
Say who decided, and what you did afterwards in either outcome: write the decision down with its number and a review date, and instrument the exposure you were worried about so the next round of the argument is settled by data instead of by seniority.

Follow-up

Publish-subscribe invalidation is lossy under a partition, and a TTL is the only hard bound. What TTL do you pick, and what does it cost you at 30k requests/second?
The key was revoked because it was found in a public repository. Does your answer change, and where does that urgency live in the design?
You lost the argument and six weeks later the failure you predicted happens. What do you say in the review, and what do you not say?