Interview guides · Allen Institute

Allen Institute Software Engineer Interview Questions & Guide 2026

Testing practice for Allen Institute software engineer candidates: the boundaries worth covering, and why flakiness is a real defect.

The 60-second brief

As a Software Engineer at the Allen Institute, you operate at the intersection of cutting-edge scientific research and high-scale software engineering. This role is pivotal in building the computational infrastructure that powers large-scale biological, neuroscience, and AI research. You are not just writing code; you are enabling scientists to process massive datasets, visualize complex biological structures, and accelerate discoveries that have global implications.

Seniority moves the scope further than the words in the title do. An earlier-career loop mostly checks that you implement something correctly and can reason about its cost, while a senior loop checks that you can pick between two defensible designs and say what you gave up.

Focus areas

Make every write idempotent under retry
Bound every outbound call with a timeout
Paginate large result sets with keyset cursors

Overview

The work is characterized by high technical complexity and a focus on long-term scientific impact. Whether you are developing robust data pipelines, designing scalable architectures for research repositories, or creating intuitive tools for data analysis, your contributions directly influence the Institute’s ability to tackle some of the most challenging problems in modern science. You will collaborate with a diverse group of engineers, researchers, and project leads, requiring a blend of rigorous engineering discipline and the agility to work in a dynamic, research-driven environment.

Where to focus your preparation

Treat the sections below as preparation areas and confirm the format with your recruiter.

Preparation focus

No round sequence has been reported for this company, so work the categories below and confirm the format with your recruiter.

What to demonstrate

Breadth across SQL, experimentation and product reasoning
Ability to state assumptions before choosing a method

How to prepare

Drill the practice exercises below and time yourself
Prepare three quantified stories about decisions you drove

Where candidates lose points

Assuming an isolation level prevents the anomaly you actually have

Isolation levels are named by the SQL standard but implemented differently, so any claim about one is only true of a named engine. PostgreSQL defaults to READ COMMITTED, where every statement takes a fresh snapshot, so two statements inside one transaction can legitimately disagree about the same row. Its REPEATABLE READ is snapshot isolation: it removes non-repeatable and phantom reads but permits write skew, where two transactions each read a set, each conclude their own write is safe, both commit, and the combined result violates a constraint that no single row expresses. Only SERIALIZABLE closes that, and it closes it by aborting a transaction with a serialization failure (SQLSTATE 40001), which means the guarantee is theoretical unless the application has a retry loop. InnoDB's REPEATABLE READ is a different mechanism again - plain SELECTs read a consistent snapshot while locking reads and writes see the latest committed row - so a read-modify-write inside one transaction can act on a value that the transaction's own earlier SELECT never returned.

Running a schema change as though the lock lasts as long as the statement

In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.

A queue or buffer with no bound

Every producer-consumer boundary needs a capacity and a policy for reaching it: block the producer, shed load, or drop the oldest entry. Unbounded buffering converts a temporary slowdown into memory exhaustion and hides the backpressure signal that would have revealed the consumer was falling behind.

Comparing floating-point values for equality, or holding money in them

Binary floating point cannot represent 0.1 exactly, so repeated addition drifts and an equality check fails on values that are mathematically equal. Store currency as integer minor units or a decimal type, and compare floats against a tolerance you chose for a stated reason.

Questions & practice

Merge partitioned event streams into one ordered feed with bounded lateness

Hard · k-way merge · watermarks · out-of-order streams

The read-model service consumes 64 log partitions carrying about 4,000 events per second in total. Each partition is ordered within itself, but partitions drift by up to 30 seconds, and the activity feed must present a tenant's events in occurred_at order. Produce the merge. State its complexity, the buffer it requires in events and in bytes, what happens when one partition is idle, and what you do with an event that arrives after you have already emitted its position. Payloads average 1 KB.

Approach

Merge with a min-heap over the 64 partition heads keyed on (occurred_at, event_id): O(log P) per event and O(n log P) overall. The tie-break on event_id is what makes the output deterministic when two partitions carry the same millisecond, which matters because the feed is paginated and a non-deterministic order reorders pages under the reader.
Emitting the heap head is only correct once every partition has produced everything up to that timestamp, so the emit condition is a watermark: the minimum across partitions of the highest occurred_at seen, less the allowed lateness. Events are held until the watermark passes them, which is what turns individually ordered streams into a jointly ordered one.
Size the buffer from the lateness rather than guessing: 4,000 events per second times 30 seconds is 120,000 buffered events, and at 1 KB each about 120 MB of heap. That number is the real price of the ordering guarantee and belongs in front of whoever asked for it.
Handle the idle partition explicitly, because it fails the feed rather than corrupting it: a partition with no traffic never advances its own maximum, so the watermark freezes and output stops entirely. Either every partition emits a periodic idle marker carrying the broker's current time, or the watermark falls back to wall clock for a partition silent beyond a threshold.
Choose the late-event policy from what the projection is keyed on. The projection upserts on (aggregate_id, aggregate_version) and discards a version it has already applied, so a late event is safe to apply out of order and correctness never depended on the merge at all. Apply it, recompute the affected feed page, and count lateness so the 30-second budget can be re-derived from data rather than folklore.
Say what the merge does not buy: ordering is guaranteed within one aggregate by the log's partitioning, and no watermark makes the cross-aggregate order authoritative. Two events from different aggregates in the same millisecond have no true order, so the feed's order is a presentation choice that must be stable rather than correct.

Follow-up

The lateness budget is raised to five minutes. What is the new buffer, and what besides memory changes?
The consumer restarts. Where does it resume from, and what does the feed look like for the first 30 seconds?
One partition is ten minutes behind because its producer is slow. Do you stall the feed or emit without it?

Collapse a redelivered event batch into per-aggregate high-water marks

Easy · hashing · at-least-once · aggregation

You drain a batch of up to 5,000,000 events, each (aggregate_id BIGINT, aggregate_version INT, event_type, payload). The log guarantees order within one aggregate only; the batch merges 64 partitions, and a relay failover has redelivered a range, so an older version for an aggregate can appear after a newer one. Given a map of last_applied_version per aggregate, produce the events worth applying, at most one per (aggregate_id, version), plus the count discarded. Target O(n) time. State the memory for 2,000,000 distinct aggregates and what you do when it does not fit.

Approach

One pass, one hash map from aggregate_id to the highest version kept, and a discard counter. An event whose version is at or below last_applied_version for its aggregate is dropped without further work, which is the whole reason the event carries its version rather than a delta. O(n) expected time, O(d) space in distinct aggregates.
Keep the maximum, never the last occurrence. The redelivered range means the final appearance of an aggregate in the batch can be an older version than one seen earlier in the same batch, so last-wins applies stale state over newer state and the projection regresses with no error anywhere.
Cost the memory instead of calling it large: an 8-byte key plus a 4-byte version is 12 bytes of payload, and an open-addressed table held at a 0.7 load factor costs roughly 17 bytes per entry before per-slot metadata, so 2,000,000 aggregates is tens of megabytes in a native layout and several times that in a runtime that boxes both key and value.
If the distinct set exceeds memory, partition on hash(aggregate_id) mod P and reduce each partition independently. Every event for one aggregate hashes to the same partition, so the per-partition result is exact and the merge is concatenation rather than a second reduction.
Reject sorting the batch by (aggregate_id, version) as the default. It is O(n log n) and buys nothing, because max is associative and commutative and needs no ordering; sorting earns its cost only when the downstream consumer must receive the events in order rather than a per-aggregate winner.
Separate the two mechanisms out loud: in-batch deduplication does not make the consumer idempotent, because the same event redelivered tomorrow arrives in a different batch entirely. The projection write itself still has to be keyed on (aggregate_id, version).

Worked solution (20 min)

Write the pass: look up last_applied_version, skip if the event's version is not greater, otherwise upsert into the keep-map only when the incoming version exceeds the version already held, incrementing the discard counter on every skip.
Hand-trace one aggregate whose events arrive as v5, v3, v4, v5 with last_applied_version = 2, and confirm the output holds v5 once while the counter reads 3.
Compute the table footprint for 2,000,000 entries at 12 bytes of payload and a 0.7 load factor, then state the multiplier for a runtime that boxes keys and values.
Add the hash-partitioning fallback and say in one sentence why the per-partition results need no cross-partition merge logic.

Expected result: A single-pass O(n) reduction keyed on aggregate_id that keeps the maximum version rather than the last occurrence, O(d) space with the byte cost stated for 2,000,000 aggregates, a hash-partitioning fallback keyed on aggregate_id, and an explicit statement that batch-local deduplication does not replace a projection write keyed on (aggregate_id, version).

Verify it yourself

Shuffling the batch order changes nothing in the output, which is the test that separates max-wins from last-wins.
Feeding the same batch twice produces an identical keep-set and only doubles the discard counter.
Every emitted event has a version strictly greater than the supplied last_applied_version for its aggregate.

Follow-up

The payload is a patch rather than a snapshot, so applying only the highest version loses the intermediate changes. What changes in your reduction?
How do you detect that version 7 arrived while version 6 was never delivered, and what should the consumer do about the gap?
Two events for one aggregate carry the same version with different payloads. Which one is wrong, and how would you find out?

Canonicalise a request body into a stable idempotency fingerprint

Medium · parsing · canonicalisation · hashing

idempotency_key.request_fingerprint is a SHA-256 over the method, path and canonicalised body, and a retry whose fingerprint differs must be rejected with 422 rather than served the stored response. Write the canonicaliser. Bodies are JSON up to 256 KB nested at most 32 levels; clients vary key order, whitespace and unicode escaping, and some send 64-bit ids as JSON numbers. Produce a deterministic byte string such that semantically identical bodies match and any semantic difference does not. State your complexity and name two normalisations you refuse to perform.

Approach

Parse once into a tree, then re-serialise under fixed rules: object keys sorted, array order preserved, one escaping convention, no insignificant whitespace. Parsing is O(n) and sorting keys is O(k log k) per object, so O(n log n) overall with O(depth) stack, and the 32-level cap is enforced during parsing because hostile nesting is how a canonicaliser becomes a stack overflow.
Sort keys by their UTF-8 bytes and say why the obvious implementation is wrong in some runtimes: a default string comparison that orders by UTF-16 code units places surrogate pairs, meaning code points from U+10000 up, below U+E000 to U+FFFF, which is not UTF-8 byte order, so two services written in different languages disagree on the same document.
Do not re-encode numbers through a double. IEEE-754 binary64 represents integers exactly only up to 2^53, so normalising a 19-digit id through a float changes it, and 1 against 1.0 cannot be reconciled without deciding whether they are the same value. Preserve the literal token, and require ids as strings at the API boundary if you want them comparable.
Reject duplicate keys rather than picking one. JSON permits them and parsers disagree, most keeping the last, so any choice you make ties the fingerprint to a parser detail that the code handling the request does not necessarily share.
Frame the hash preimage so concatenation cannot collide: delimit or length-prefix the method, path and body, otherwise one request's fields can be rearranged into another request with the same byte stream and the same fingerprint.
Name the refusals and their consequence: no case folding, no dropping of null-valued keys, no Unicode normalisation. Each makes two different requests fingerprint alike, and the resulting failure is the worst one this table has, since the second request is answered with the first one's stored response and its effect never happens.

Follow-up

A client sends the same logical request with an extra field your API ignores. Same key, different fingerprint, so you return 422. Is that the right answer?
Where does the fingerprint get computed relative to request decompression and the body-size limit?
The endpoint takes 1,000 requests per second with 256 KB bodies. What does hashing cost, and does it belong at the edge or in the core service?

Explain why the owner filter ignores the listing index

Medium · composite index · query plans · sort elimination

The only index on resource is (tenant_id, status, updated_at DESC, resource_id DESC). A new endpoint returns one user's resources across all statuses, newest created first: WHERE tenant_id = $1 AND owner_user_id = $2 ORDER BY created_at DESC LIMIT 20. On a tenant with 2M rows it takes 900 ms and EXPLAIN shows a sort above a large scan. Explain precisely why the existing index cannot serve it, give the index that can, and state which of these the new index still will not help: owner_user_id alone across tenants; the same query ordered by updated_at. PostgreSQL 16.

Approach

Separate the two jobs an index does. For filtering, a composite btree is seekable only on a left prefix, so with no predicate on status the scan can at best range over tenant_id and test owner_user_id per row; PostgreSQL 16 has no btree skip scan to jump the unconstrained column.
For ordering, the index is sorted by (status, updated_at) within a tenant and not by created_at, so the LIMIT cannot stop early: every matching row is read and then sorted. That is the 'Sort Method: top-N heapsort' line, and it is why the plan reads 2M rows to answer with 20.
Derive the replacement from the access path — equality, equality, then the ordering column: CREATE INDEX CONCURRENTLY ON resource (tenant_id, owner_user_id, created_at DESC). The scan seeks to the (tenant, owner) range and walks 20 entries in order, so the Sort node disappears along with the row-read.
Treat INCLUDE (title, status) as conditional, not free. An index-only scan still visits the heap for any row whose page is not marked all-visible, so on a table taking 1.2k writes/second the win depends on autovacuum keeping the visibility map current, and the wider index costs more on every insert.
Answer the two negatives explicitly. owner_user_id alone is not a left prefix of the new index, so it degrades to a full scan of the index at best. Ordered by updated_at, the query still seeks on the (tenant, owner) pair but must sort, because only created_at is ordered within that pair.
Measure both sides with EXPLAIN (ANALYZE, BUFFERS) and compare estimated against actual rows at the lowest node — a 2M-versus-200 misestimate there is usually what chose the plan, and adding an index will not fix a statistics problem.

Follow-up

90% of rows are status='active'. Would a partial index WHERE status = 'active' change your answer, and for which of the three queries?
A dashboard runs this for 40 owners in one page load. What changes about the design?
How do you roll this index out on a table taking 1.2k writes/second, and what does it cost on every insert from then on?

Hold a per-tenant active cap against concurrent creates

Hard · write skew · isolation levels · constraints

A tenant on the standard plan may hold at most 50 resources with status='active'. The create handler runs SELECT count(*) FROM resource WHERE tenant_id = $1 AND status = 'active', compares to 50, then inserts. Two creates arrive 3 ms apart on different instances and the tenant lands at 51. Name the anomaly, say whether PostgreSQL 16 READ COMMITTED or REPEATABLE READ prevents it and why, then give an implementation that holds the cap at READ COMMITTED with the exact statements. Finally, say what changes when the cap is 'at most one running export per tenant' on job_run.

Approach

Name it: write skew. The two transactions read an overlapping set and write disjoint rows, so there is no row-level conflict for the engine to detect and each commit is individually legal.
Rule out the levels precisely. READ COMMITTED takes a fresh snapshot per statement and takes no lock on the counted rows, so both see 49. PostgreSQL's REPEATABLE READ is snapshot isolation: it removes non-repeatable reads and phantoms within the snapshot but still admits write skew, because the anomaly is not a re-read of a changed row, it is a read of a set that a concurrent transaction invalidates. Only SERIALIZABLE closes it, by tracking the read dependency and aborting one transaction with SQLSTATE 40001 — a guarantee that exists only if the application re-runs the whole transaction from the read.
Convert the set predicate into a single-row conflict: keep tenant.active_resource_count and run UPDATE tenant SET active_resource_count = active_resource_count + 1 WHERE tenant_id = $1 AND active_resource_count < 50 in the same transaction as the INSERT. Zero affected rows is the cap, returned as 409. The row lock serialises the decision at any isolation level, and contention is bounded to one tenant's row — which is also the fair-scheduling unit, unlike a global counter that would convoy every tenant behind one row.
State the cost you just took on: a counter is a second source of truth that can drift, so every path that changes status must adjust it inside the same transaction, and a periodic reconciliation has to exist, with resource_revision as the authority for what the count should have been.
For the job case the invariant is expressible per row, so let the database hold it: a partial unique index on job_run (tenant_id, job_type) WHERE status IN ('queued','running') makes a second running export unwritable and the loser takes 23505, mapped to 409. That is strictly better than a counter — no drift, no reconciliation — and it is available only because the cap is one rather than fifty.
Add the retry discipline each route demands: under SERIALIZABLE both 40001 and deadlock 40P01 are retryable and the retry must re-execute the read, while under READ COMMITTED with the counter nothing retries, because the conflict is reported to the caller rather than raised as an error.

Worked solution (35 min)

Reproduce with two sessions that both count 49, both insert and both commit, at READ COMMITTED and then at REPEATABLE READ; record the final active count for each.
Repeat both sessions at SERIALIZABLE and record which SQLSTATE the loser receives and at which statement it is raised.
Implement the counter form and run a 20-way concurrent create against a tenant sitting at 45 active resources.
Implement the partial unique index for the job case and race 20 enqueues of the same export.

Expected result: READ COMMITTED and REPEATABLE READ both finish at 51. SERIALIZABLE finishes at 50 with one transaction aborted at 40001 — usually at commit, though SSI may raise it at an earlier statement. The counter form finishes at exactly 50: of the twenty creates, the five that take the tenant from 45 to 50 return 201 and the remaining fifteen match zero rows and return 409, with no error raised. The job index admits one running export and rejects the rest with 23505.

Verify it yourself

count(*) WHERE status='active' never exceeds the cap across 50 randomised concurrent runs.
Remove the active_resource_count < 50 predicate and the same test must start failing; if it still passes, the race is not being exercised.
At SERIALIZABLE, confirm every participating session is SERIALIZABLE — one session at a lower level and the guarantee is gone.
After each run, the counter equals the real count of active rows.

Follow-up

A resource moves from archived back to active. Which statements change, and what breaks if the counter update and the status change land in different transactions?
The cap becomes plan-dependent and a plan can change mid-month. Where does the number 50 live, and who reads it?
How do you detect after the fact that the counter drifted, without locking the table?

How would you build a specific component or system from scratch?

Medium

Approach

Work from the requirement backwards to the design.
State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Design a cache system that can recover from sudden shut-offs.

Medium

Approach

Work from the requirement backwards to the design.
Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

These questions assess your ability to bridge theoretical knowledge with practical implementation, often focusing on data structures or specific domain challenges.

Medium

Approach

Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.
Work from the requirement backwards to the design.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Evolve the resource contract without breaking integrations you cannot upgrade

Hard · versioning · backward compatibility · deprecation

GET /v1/resources/{id} returns status from a four-value enum ('draft','active','archived','deleted'), a numeric version, and the body inline. Consumers are a browser app you deploy and roughly 300 server-side integrations, some untouched for two years, that switch exhaustively on status and parse ids as JSON numbers. You must add a 'pending_review' status, move bodies over 256 KB to a body_ref pointer, and expose per-field change history from resource_revision. Specify the compatibility policy, the wire changes, how both generations are served, and the evidence that lets you remove the old shape.

Approach

Write the policy first and date it: fields are added, never retyped or repurposed; consumers ignore unknown fields; an unknown enum value maps to a documented fallback; nothing is removed until telemetry shows no caller reads it. Then say the uncomfortable part out loud - v1 shipped without the unknown-value rule, so 300 running integrations have no fallback, and no server change can install one into code that is already deployed.
That single fact forces per-request negotiation rather than a server-side default. Keep one internal model and select a serialiser from an explicit version in the request, and default a caller that sends nothing to the oldest supported version. Defaulting to the newest is the change that breaks every integration that never asked for anything, on the day you ship.
Downgrade 'pending_review' for old callers to the nearest state they already handle, 'draft', and state the loss explicitly: those integrations cannot see review state and will treat the resource as editable. If that is unacceptable for one integration, the remedy is moving it to the new version, not a cleverer projection - there is no mapping that invents a state the client has no code for.
Make the body change additive. Old callers keep body inline; the new shape adds body_ref and a size field, and resources over the limit are served to old callers by resolving the pointer server-side or by refusing with a documented code, chosen once and published. Never repurpose body to carry the pointer: a client that renders it shows a storage key to a user, and that failure is silent, where a missing field would have been loud. While you are here, serialise BIGINT ids as strings in the new shape - a browser parsing JSON numbers gets IEEE-754 doubles, exact only to 2^53 - and treat that as its own breaking change requiring the same negotiation, not a quiet fix.
Add change history as a separate sub-resource, GET /v1/resources/{id}/revisions, keyset-paginated over (resource_id, version) rather than as an array inside the resource. A field added to a hot response is paid for by every caller including those that never read it, and an unbounded array inside a cached object destroys the size assumptions the cache was configured with.
Retire on evidence rather than on a date alone: count requests per negotiated version per credential, publish a Sunset header (RFC 8594) with the removal date and a link to the migration, contact the credentials still on the old version, then answer 410 Gone once it is removed. Keep each version's serialiser under snapshot tests so a refactor cannot change v1's bytes by accident.

Worked solution (40 min)

Write and date the compatibility policy as five rules, then mark which of them v1 callers cannot honour.
Choose the negotiation mechanism and the default for an unversioned request, and justify the default in one sentence.
Write the downgrade table: new state to old state, new body shape to old body shape, and what is lost in each direction.
Specify the revisions sub-resource: its URL, its cursor, and why it is not a field on the resource.
Write the sunset plan: the per-version per-credential metric, the header, the lead time, and the terminal status code.

Expected result: A dated additive policy, version negotiation over one internal model with the oldest version as the default, a downgrade table with its losses named, history as a paginated sub-resource, and a removal gated on per-credential version telemetry rather than a calendar entry.

Verify it yourself

A two-year-old integration receives a resource that is really in 'pending_review' and neither crashes nor renders a blank state.
Deleting the legacy body field from the internal model breaks exactly one serialiser and no business logic.
Every v1 response is byte-stable under snapshot tests after an unrelated refactor.

Follow-up

An old integration submits a status transition while the resource is really in 'pending_review'. What does the write path accept, and what does it reject?
Two years on you want to delete the v1 serialiser. What evidence makes that safe, and who must be contacted before it happens?
How would you test against a two-year-old integration rather than against today's source?

Exports duplicate a row range about once a week

Hard · leases · clock skew · idempotency · fencing

Roughly once a week an export writes a file containing a duplicated range of rows. The affected job_run rows show attempt = 1, status = succeeded, one started_at, and a lease_owner naming a different host from the one whose logs show the job starting. Leases last 30 seconds and are heartbeated every 10 from inside the handler; lease_expires_at is computed on the worker and compared against the database's now(). Find the mechanism, and give a fix that holds even if you cannot fix the clocks.

Approach

Start from the fact that eliminates the obvious answer. attempt = 1 means no retry was recorded, so this is not a re-run after failure; two workers ran the same row concurrently and the takeover path never touched the counter. lease_owner naming a host other than the one that started the job is the same statement from the other side.
Enumerate the mechanisms that cause a premature takeover, then find the signal that separates them. Either the lease genuinely expired because the heartbeat did not fire, which is what happens when the heartbeat runs on the handler's own thread and the handler makes a long blocking call, or it only appeared expired because two clocks disagree, since lease_expires_at is written from the worker's clock and evaluated against the database's. The discriminator is the distribution: incidents clustered on the longest exports indict the heartbeat, incidents clustered on one host indict skew. Measure both, and measure each host's offset against the database directly.
Read the reclaim query precisely. In PostgreSQL now() is transaction start time, not statement time, so a reclaimer holding a long transaction compares against an older timestamp than expected; clock_timestamp() is the statement-time function. This is worth ruling in or out before you redesign anything, because it changes which rows look expired.
Remove the second clock rather than trying to synchronise it. Issue and extend the lease in the database, with lease_expires_at = now() + interval '30 seconds' in both the claim and the heartbeat, so exactly one clock is ever compared and worker skew stops mattering to this predicate.
Accept that a lease can still expire under a slow worker, because a lease cannot distinguish slow from dead, and fence the work. Carry a monotonically increasing lease generation and make every write the handler performs conditional on still holding it, as UPDATE ... WHERE job_run_id = $1 AND lease_owner = $2 AND lease_generation = $3, so a displaced worker's writes affect zero rows and it aborts instead of duplicating.
Make the handler's writes idempotent independently of all that: give each exported chunk a natural key of (job_run_id, batch_start) with a unique constraint so a second copy conflicts rather than appends, move the heartbeat off the handler's thread, and increment attempt on takeover so the event is visible in a metric.

Follow-up

The displaced worker has already streamed half the file to object storage. What makes that side effect safe to repeat?
You now count takeovers. What alert fires on that counter, and at what threshold?
What breaks if you simply raise the lease to five minutes?

Suggested practice checklist

For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.

Day 1 — Measure before reasoning

Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.

Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.

Day 2 — References, copies, and the bugs they produce

Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.

Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.

Day 3 — Types, once, in a language that checks them

Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.

Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.

Day 4 — Concurrency, starting with what actually runs at the same time

Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.

Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.

Day 5 — Debugging as a procedure rather than an instinct

Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.

Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.

Day 6 — Tests that catch the bug you are about to write

Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.

Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.

Day 7 — Debug something broken, out loud

Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.

Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.

Know where your numbers came from

Bring the two or three numbers the story rests on and know how they were collected. A p99 whose timer starts inside your handler excludes the time a request spent queued, so it can sit flat while users wait longer. Give the window, the percentile and what the measurement left out, or drop the number.

These questions evaluate your communication style, past project impact, and how you navigate team dynamics or ambiguity.

Medium · behavioural and engineering judgement

Approach

Close with what you would do differently, concretely.
State the situation in two sentences and spend the rest on the reasoning.
Name the disagreement and how you resolved it with evidence.

Follow-up

What would you do differently if you ran that again?
How did you know your change caused the improvement?

Ship under a deadline and bound the debt you chose

Medium · pagination · technical debt · tradeoffs

You have four days to ship a tenant-facing listing endpoint. The version you would defend uses keyset pagination over (tenant_id, status, updated_at DESC, resource_id DESC); the version you can finish uses LIMIT/OFFSET with no matching index. Describe a deadline call you actually made of this shape: what you shipped, what you knowingly deferred, how you bounded the damage with a mechanism rather than an intention, and the specific numeric condition that would force the follow-up. Name who you told and where you wrote it down.

Approach

Name the deferred failure precisely instead of calling it slow. OFFSET n makes the database produce and discard n rows, so cost grows with page depth; without an index matching the sort, every matching row is read and sorted before the limit applies; and rows inserted between two page fetches shift across the boundary so items are skipped or repeated with nothing in the response to signal it.
Bound the blast radius with something mechanical rather than a promise: cap maximum page depth, cap page size, restrict the endpoint to one internal caller, or keep it behind a flag. State which failure each cap removes and which it leaves standing.
Attach a number to the trigger and wire it to an alarm: the first tenant crossing N resources, or the endpoint's p99 crossing its share of the 400 ms budget, so the debt announces itself instead of waiting to be remembered.
Write it where the next engineer looks, which is the code and the ticket, not a chat message: what was deferred, why, the cap, and the trigger.
Report what actually happened in your real example, including the case where the trigger never fired and the debt was correctly never repaid.

Follow-up

At what page depth does the offset version breach your latency budget, given your page size and row counts?
What breaks first when you switch to keyset pagination later, and what does a client holding an old page token see?
Who would have overruled you if you had asked for two more days, and did you ask?

Turn a code review disagreement into a decision

Easy · code review · optimistic concurrency · communication

A colleague's change updates a row with UPDATE resource SET version = version + 1 WHERE resource_id = $1 AND version = $2 and treats an affected-row count of zero as a successful no-op. You read that as a silently lost update; they think returning 200 is friendlier to clients than returning a conflict. Describe how you have handled a review disagreement of this shape: what goes in the comment, when you leave the thread, and who decides. Then write the comment you would leave here, in under 80 words.

Approach

Sort the disagreement before writing anything. A silently discarded write is a correctness claim about data; the choice between 409 and 412 is taste. Only the first justifies blocking a merge, and saying which one you are doing is most of the value of the comment.
Make the claim reproducible in the comment itself with an interleaving rather than a principle: A reads version 7, B reads version 7, B commits version 8, A's predicate matches zero rows, A is told it succeeded and A's edit is gone.
Offer the alternative with its cost attached: return 409 carrying the current version and the revision that won, so the client can re-read and re-apply. Note that automatic retry is not the fix, because a retry re-reads the winner's state and reapplies an intent formed against data that no longer exists.
Apply an escalation rule you can state: two round trips on the thread, then a call, and the service's owner decides rather than the reviewer. A reviewer who cannot be overruled is a bottleneck with extra steps.
Close in writing wherever the decision lands, so the next reader finds the reasoning in the code or the ticket instead of in a collapsed review thread.

Follow-up

Where would you put the test that fails if someone reintroduces the swallowed zero rowcount?
The author says clients cannot handle a 409. How do you check whether that is true?
How do you handle the same review comment when the author is more senior than you and in a hurry?