Interview guides · Sandia National Laboratories

Sandia National Laboratories Machine Learning Engineer Interview Questions & Guide 2026

Sandia National Laboratories Machine Learning Engineer interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.

The 60-second brief

The role of a Machine Learning Engineer at Sandia National Laboratories is crucial in leveraging advanced machine learning techniques to solve complex problems in physical and materials sciences. This position is at the intersection of cutting-edge research and practical application, directly impacting the development of innovative solutions that enhance national security and energy initiatives. As a Machine Learning Engineer, you'll contribute to projects that drive significant advancements in scientific understanding and technology.

State every complexity claim with the assumption sitting under it. Hash lookup is O(1) on average and only for a hash that spreads your actual keys; comparison-based sorting cannot beat n log n, though counting or radix sort can when the keys are bounded integers.

Focus areas

Choose indexes from the query's access path
Trace a symptom to a mechanism under load
Bound every outbound call with a timeout

Overview

This role is not only about applying algorithms; it involves a deep understanding of the domain, collaboration with multidisciplinary teams, and the ability to translate theoretical models into real-world applications. You'll work with state-of-the-art tools and methodologies to develop models that can predict behavior, optimize processes, and generate insights that inform decisions at various levels of the organization. The impact of your work will resonate through products and services that enhance the operational capabilities of Sandia National Laboratories and its stakeholders.

Where to focus your preparation

Treat the sections below as preparation areas and confirm the format with your recruiter.

Preparation focus

No round sequence has been reported for this company, so work the categories below and confirm the format with your recruiter.

What to demonstrate

Breadth across SQL, experimentation and product reasoning
Ability to state assumptions before choosing a method

How to prepare

Drill the practice exercises below and time yourself
Prepare three quantified stories about decisions you drove

Where candidates lose points

Assuming an isolation level prevents the anomaly you actually have

Isolation levels are named by the SQL standard but implemented differently, so any claim about one is only true of a named engine. PostgreSQL defaults to READ COMMITTED, where every statement takes a fresh snapshot, so two statements inside one transaction can legitimately disagree about the same row. Its REPEATABLE READ is snapshot isolation: it removes non-repeatable and phantom reads but permits write skew, where two transactions each read a set, each conclude their own write is safe, both commit, and the combined result violates a constraint that no single row expresses. Only SERIALIZABLE closes that, and it closes it by aborting a transaction with a serialization failure (SQLSTATE 40001), which means the guarantee is theoretical unless the application has a retry loop. InnoDB's REPEATABLE READ is a different mechanism again - plain SELECTs read a consistent snapshot while locking reads and writes see the latest committed row - so a read-modify-write inside one transaction can act on a value that the transaction's own earlier SELECT never returned.

Running a schema change as though the lock lasts as long as the statement

In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.

Check-then-act on shared state

Read, decide, write is not safe under concurrency unless the decision and the write are one atomic step: a unique constraint with conflict handling, a compare-and-set, or a row lock held for the whole transaction. Two requests can both pass the existence check before either inserts, which shows up as duplicate rows under load and never in a single-threaded test.

Assuming the bug is in the framework

Suspect your own code first: read the stack trace top to bottom, check which versions are actually installed rather than which ones you believe are, and reproduce in isolation before blaming a library that thousands of people run daily. When the fault really is upstream, you need that minimal reproduction to say so credibly anyway.

Questions & practice

Can you discuss a machine learning project you've completed and the challenges you faced?

Medium · machine learning fundamentals

Approach

State the learning problem: the label, the unit of prediction and how the model is used.
Say how you would validate it, and where leakage could enter the split.
Name the simplest model that could work and what would make you move past it.

Follow-up

What changes if the classes are heavily imbalanced?
How would you know the model is overfitting?

Explain the concept of overfitting and how you would prevent it.

Medium · machine learning fundamentals

Approach

Say how you would validate it, and where leakage could enter the split.
Name the simplest model that could work and what would make you move past it.
State the learning problem: the label, the unit of prediction and how the model is used.

Follow-up

How would you know the model is overfitting?
What changes if the classes are heavily imbalanced?

What tools and libraries do you prefer for building machine learning models?

Medium · machine learning fundamentals

Approach

Say how you would validate it, and where leakage could enter the split.
Name the simplest model that could work and what would make you move past it.
Pick the metric from the cost of each error type, not from habit.

Follow-up

What changes if the classes are heavily imbalanced?
How would you know the model is overfitting?

Track a rolling failure rate per destination for circuit decisions

Easy · sliding window · ring buffer · circuit breaker

The egress service delivers about 1,500 webhooks per second across roughly 40,000 destinations, each call bounded by a 10 second timeout. Maintain, per destination, the failure rate over the trailing 60 seconds so a caller can ask before dispatch whether the circuit should open. Attempts arrive as (destination_id, finished_at_ms, outcome). Requirement: amortised O(1) per attempt, with total memory bounded by the destination count rather than by traffic. Give the structure, its exact memory, and the rule that stops a destination with three attempts from opening a circuit.

Approach

Name the exact-deque version and then reject it as the default. Holding timestamps and advancing a tail pointer past anything older than now minus 60 seconds is a correct two-pointer window at amortised O(1) per attempt, but its memory tracks in-window traffic, so one destination in a retry storm holds hundreds of thousands of entries while thousands of quiet destinations hold none.
Use a ring of 60 one-second buckets per destination, each bucket a pair of counters for attempts and failures. On an attempt, advance the ring by the elapsed whole seconds, zeroing at most min(elapsed, 60) buckets, then increment the head. That is amortised O(1) with a fixed footprint per destination.
State the footprint: 60 buckets times two 4-byte counters is 480 bytes of payload per destination, so 40,000 destinations is roughly 20 to 25 MB with per-entry overhead, bounded by the catalogue rather than by the rate. The cost is granularity, since the oldest bucket ages out in whole seconds, which is far tighter than the decision needs.
Require a minimum sample before the circuit may open. A destination with three attempts and three failures reads as 100 percent and is not evidence; a floor of roughly 20 attempts in the window makes the ratio meaningful, and below that floor use a run of consecutive failures as the trigger instead.
Expire idle destinations, or memory grows with every destination ever seen rather than with the live set. Hold the rings in a bounded LRU keyed on destination_id and treat a miss as no history, which is the correct default for an endpoint that has been silent for a minute.
Keep the half-open probe out of the window arithmetic. After the circuit opens, one probe per interval decides whether to close it, and folding that single success into a window that still holds a 100 percent failure history would reopen the destination on one data point.

Worked solution (20 min)

Define the bucket struct and the advance step: take floor(finished_at_ms / 1000), compare with the ring's current second, zero min(delta, 60) buckets forward, then write into the new head.
Trace a destination that receives 5 attempts, goes silent for 90 seconds, then receives one more, and confirm the rate is computed from one attempt rather than six.
Compute total memory for 40,000 destinations at 60 buckets of two 4-byte counters, and state what changes if the window widens to 300 seconds.
Write the open rule as a single predicate combining the minimum-attempt floor with the rate threshold.

Expected result: A per-destination ring of 60 one-second (attempts, failures) buckets advanced lazily for amortised O(1) cost, roughly 480 bytes of counters per destination and about 20 to 25 MB for 40,000 of them, an LRU bound on live rings, and an open rule requiring both a minimum attempt count in the window and a rate above threshold.

Verify it yourself

After 60 seconds of silence a destination's window holds zero attempts and cannot open a circuit.
Memory measured with one hot destination and 39,999 idle ones is indistinguishable from the reverse, which the deque version would fail.
Replaying the same attempt stream with the clock shifted by an hour yields the same final rate.

Follow-up

The fleet is 30 instances and each sees roughly a thirtieth of a destination's traffic. Where does the rate actually live, and what does a per-instance answer get wrong?
A destination answers in 9.5 seconds and succeeds. It is not failing but it is consuming your per-destination concurrency. What signal should open the circuit here?
How would you make the window survive a process restart, and is it worth the cost?

Write the update path that detects a concurrent edit

Medium · optimistic concurrency · transactional outbox · audit history

resource carries version INT NOT NULL DEFAULT 1. resource_revision holds revision_id, resource_id, version, actor_user_id, change_kind, patch JSONB, request_id, created_at with UNIQUE (resource_id, version). outbox_event holds aggregate_type, aggregate_id, aggregate_version, event_type, payload, status. A PUT carries the version the client read. Write the exact statements for the single transaction that applies the edit, records the revision and enqueues 'resource.updated', and give the handler's branch on zero affected rows. Then say what PostgreSQL 16 does under READ COMMITTED when two of these updates hit one row at once.

Approach

One transaction, three writes, no network call inside it: UPDATE resource SET title = $3, version = version + 1, updated_at = now() WHERE resource_id = $1 AND tenant_id = $4 AND version = $2; then INSERT the resource_revision row at version $2 + 1; then INSERT the outbox_event row at the same aggregate_version. The event goes to a table rather than a broker because no transaction spans both.
Branch on the affected-row count before doing anything else. Zero has three causes — stale version, wrong tenant, row gone — so re-read once and map to 409 carrying the current version, or 404 for an id outside the caller's tenant, which also stops the endpoint confirming that another tenant's id exists.
State the engine behaviour instead of assuming it. Under READ COMMITTED the second UPDATE blocks on the row lock, and when the first commits PostgreSQL re-evaluates the WHERE clause against the newly committed row, so the version predicate now fails and the statement reports zero rows. Under REPEATABLE READ the identical collision raises SQLSTATE 40001 instead, so the handler must fold both shapes into one conflict response.
Keep UNIQUE (resource_id, version) even though the predicate already serialises writers. It is what makes a lost update unwritable if any other path ever reaches the revision table, and it converts a logic bug into 23505 rather than into a silently missing history row.
Refuse to auto-retry the whole PUT. A retry re-reads the winner's state and reapplies an intent formed against data that no longer exists — the silent overwrite the version token was added to detect. Return the conflict; merge field-wise only if the patches are provably disjoint.
Note that now() is the transaction timestamp in PostgreSQL, so resource.updated_at, the revision's created_at and the outbox row share one instant, which is what later makes reconciliation between the three tables unambiguous.

Worked solution (25 min)

Write the three statements plus the rowcount branch and confirm they sit in one BEGIN/COMMIT with no outbound call between them.
Run two clients that both read version 7 and apply their updates 5 ms apart; assert one 200 and one 409.
Assert resource.version = 8, exactly one resource_revision row at version 8, and one outbox_event row at aggregate_version 8.
Repeat at REPEATABLE READ and record the different failure shape (SQLSTATE 40001) the handler must also map to 409.
Delete the version predicate and re-run: both writes commit and the first edit disappears with no error raised anywhere.

Expected result: Exactly one of the two commits. version = 8 with one revision row and one outbox row at version 8, the loser gets 409 carrying version 8, and no state exists in which the resource moved but the revision or the outbox row is missing.

Verify it yourself

COUNT(*) of resource_revision for that resource equals resource.version, with no gap in the version sequence.
Kill the process between the UPDATE and the outbox INSERT — the transaction rolls back and no event describes a change that never happened.
Under a 50-way race, resource.version equals the number of 2xx responses rather than the number of requests.

Follow-up

A client sends the version it read ten minutes ago and the resource has moved three versions. What is in your 409 so it can resolve the conflict without a full re-fetch?
Two editors, two disjoint fields, no overlap. Does your answer still refuse the second write, and should it?
Every write now touches a second hot table. How do you keep the outbox insert and its partial index from becoming the write bottleneck at 1.2k writes/second?

Find version gaps and relay lag with window functions

Medium · window functions · event tables · frame semantics

outbox_event holds event_id, aggregate_type, aggregate_id, aggregate_version, event_type, payload, status ('pending','published','dead'), attempts, created_at, published_at. A projection is missing rows and you must decide whether the relay skipped events or the consumer dropped them. Write three queries over the last seven days: one listing every aggregate_id whose published aggregate_version sequence has a hole, one giving per-day counts with a running total, and one returning the newest published event per aggregate. For each, say where the window function is evaluated relative to WHERE and LIMIT. PostgreSQL 16.

Approach

Gaps: compute lead(aggregate_version) OVER (PARTITION BY aggregate_id ORDER BY aggregate_version) in a subquery, then filter next_version <> aggregate_version + 1 in the outer query. Window functions are evaluated after WHERE, GROUP BY and HAVING and before the outer ORDER BY and LIMIT, so the predicate cannot sit in the same WHERE clause and PostgreSQL 16 has no QUALIFY.
Say what the seven-day filter does to the answer: it truncates every partition, so the first row per aggregate has no predecessor inside the window and a hole spanning the boundary is invisible. Widen the window, or join to resource.version as the authority for the true maximum.
Running total: SELECT date_trunc('day', created_at) AS d, count() AS n, sum(count()) OVER (ORDER BY date_trunc('day', created_at) ROWS UNBOUNDED PRECEDING). An aggregate inside a window call is legal because grouping runs before windowing. The grouping key is unique per row here so ROWS and RANGE agree, but write the frame anyway — over ungrouped rows with tied timestamps the default RANGE frame pulls in every peer row and the total jumps.
Newest per aggregate: DISTINCT ON (aggregate_id) ... ORDER BY aggregate_id, aggregate_version DESC is the cheap PostgreSQL-only form when an index matches that order; row_number() OVER (PARTITION BY aggregate_id ORDER BY aggregate_version DESC) = 1 is the portable form and needs a subquery for the same evaluation-order reason as the gap query.
Interpret rather than report: no gaps plus a normal p95 of published_at - created_at points at the consumer; gaps or a fat lag tail point at the relay; rows still 'pending' with attempts > 0 point at neither, because they never left the database.
Be explicit that the partial index on (created_at, event_id) WHERE status = 'pending' does not serve any of these — they read published rows. Name the index a recurring monitor would need, and say why a query run twice a year may not deserve one.

Follow-up

Relay failover redelivers events. Does a duplicate break the gap query, and how would you detect one from this table alone?
Turn the gap check into a continuous monitor rather than a query someone runs after an incident. What does it watch?
The consumer claims it never received event 4,812,006. What do you look at, in what order?

What are the differences between supervised and unsupervised learning?

Medium

Approach

Work from the requirement backwards to the design.
State your assumptions explicitly before working the problem.
Say what you would check first and why it is the highest-information step.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

Describe how you would approach feature selection for a new dataset.

Medium

Approach

Clarify what is being asked and what a complete answer contains.
Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Specify the outbound webhook contract a customer endpoint can verify

Medium · webhooks · hmac · at-least-once · ordering

Egress delivery sends resource.created, resource.updated and resource.archived to roughly 40k customer endpoints at about 1.5k deliveries/second, with a per-destination concurrency cap of 4 and a 10 second connect-plus-read timeout, at-least-once from the relay. The receiver is code you do not control, often behind a proxy that may re-encode JSON. Specify the payload envelope, how the receiver verifies authenticity and rejects replays, what ordering it may assume, how it deduplicates, which status codes it should return and what each means to us, and the retry, circuit and replay behaviour.

Approach

Send the fact, not a pointer: event_id, event_type, aggregate_type, aggregate_id, aggregate_version, occurred_at and the payload copied from outbox_event. A receiver told only that resource 42 changed has to call back, and the call back returns a newer state than the event describes - that is precisely how a consumer applies changes out of order while believing it is following them.
Sign the exact bytes on the wire: HMAC-SHA256 over the timestamp, a separator, and the raw body, in a header, compared in constant time before the body is parsed. Instruct receivers to verify the raw bytes, never a re-serialised copy, since a proxy that reorders keys or changes whitespace invalidates a signature computed over re-encoded JSON. Reject a timestamp outside a tolerance - five minutes is typical - and keep a short seen-set of event_ids, because a capture replayed inside the tolerance still verifies.
Support key rotation by sending several signature values in one header during the overlap window and publishing that any one that verifies is sufficient. Without that rule a key rotation is a synchronised outage across every destination at once.
State the guarantee as it actually is: at-least-once and unordered, including within a single aggregate_id. The outbox is ordered per aggregate at the source, but the relay holds up to 4 requests in flight per destination and backs off each retry independently, so a version 6 delivery that needs one retry lands after a version 7 that succeeded on its first attempt. Publishing per-aggregate ordering we do not enforce is worse than publishing none, because a receiver that trusts it overwrites new state with old. Receivers deduplicate on event_id and apply conditionally - UPDATE ... WHERE aggregate_version < :incoming, or the same comparison under a row lock on the aggregate - which discards stale redeliveries and stays correct when two deliveries for one aggregate are processed at the same time. A duplicate and an out-of-order delivery then have the same outcome: no-op.
Define the response contract in the receiver's terms. 2xx means durably accepted, and accepting means writing it down - a receiver that does its processing inline inside our 10 second timeout will be retried while it is still working. 4xx is permanent and we stop. 429, 5xx and timeouts are retried with capped exponential backoff and full jitter within the per-destination concurrency cap, and a circuit opens after a run of consecutive failures and half-opens on a schedule, so a permanently dead endpoint costs one probe per interval instead of a growing queue.
Give them a recovery path that does not involve us holding a queue for everyone: a visible parked state, and a replay endpoint keyed by event_id or time range, so a receiver that was down for two hours asks for the gap on its own schedule.

Worked solution (30 min)

Write the envelope fields and mark which are needed for deduplication and which for discarding a stale delivery.
Write the signature string construction and the receiver's verification steps in order, including the constant-time compare and the tolerance check.
Write the receiver's idempotent apply: dedupe on event_id, then the version-conditional write that discards stale and concurrently delivered redeliveries.
Write the status-to-action table from our side: which codes retry, which stop, and what opens the circuit.
Describe the parked state and the replay call, including what the customer must supply.

Expected result: A self-contained signed envelope, verification over raw bytes with a replay window and a seen-set, an at-least-once guarantee that states delivery is unordered even within one aggregate, a version-conditional apply, a status table that distinguishes permanent from transient, and a customer-driven replay path.

Verify it yourself

Delivering the same event twice leaves the receiver's state identical to delivering it once.
Delivering version 7 and then version 6 for one aggregate leaves version 7 applied.
A destination that always times out costs one probe per circuit interval rather than a retry queue.

Follow-up

A customer insists an event never arrived. What do you check, in order, and which table settles the argument?
A receiver cannot store a version and asks us to deliver their events in order. What would the relay have to give up to offer that, and what does one failing event then do to the rest of their stream?

One log partition stops advancing while the others drain

Easy · poison message · head-of-line blocking · consumer lag · projections

Search results for a subset of tenants are hours stale; the rest are current. The projection consumer reports lag of zero on 15 of 16 partitions and 400,000 on one. Its error rate is flat and its CPU is idle. outbox_event has no pending rows older than a second, so the relay has published everything it holds. Identify the mechanism, give the ordered checks, and state what you do in the first ten minutes versus what you change permanently.

Approach

Read the lag distribution first. A slow consumer lags everywhere; zero on fifteen partitions and 400,000 on one is not throughput. Idle CPU on the stuck partition means the consumer is not advancing its offset at all, which points at one message it cannot get past rather than at a rate problem.
Exonerate the producer before touching the consumer. No pending outbox rows older than a second means the relay published, so the event exists in the log. This separates never sent from sent and never applied, which are different code paths and usually different owners.
Read the message at the stuck offset and the handler's log lines for its event_id. A flat error rate with no progress has two explanations and you must distinguish them: the handler is throwing and the retry loop is swallowing it, or the handler is blocking on something and never returning. Idle CPU with no error lines favours the second.
Mitigate before diagnosing further. Move the offending event to a dead-letter store and commit the offset past it. Adding consumers does nothing here, because a partition is consumed by exactly one member of the group, and the blast radius is every aggregate hashed to that partition, not only the aggregate that produced the bad event.
Fix permanently by bounding handler attempts and dead-lettering on exhaustion, so no single message can stop a partition. Then replay the dead-lettered event once the handler is fixed: it carries aggregate_id and aggregate_version, so a consumer that discards versions it has already applied can absorb the replay, and resource_revision is the fallback if the event itself is unusable.

Follow-up

The dead-lettered event carried aggregate_version 7 and the projection had applied 6. What must the replay do differently if 8 and 9 landed in the meantime?
How do you show staleness to the user while the partition is behind, given the API already returns the projection's watermark?
What changes if the message is poison because a previous deploy wrote a payload shape the current code cannot parse?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

Outage answers are graded on instruments

Nobody is scoring your stamina at three in the morning. What carries weight is which signal told you something was wrong, what you measured before touching anything, what you rolled back versus what you fixed forward, and why you picked one. 'We restarted it and it went away' is a story about not knowing.

Unblock an engineer without taking the keyboard

Easy · mentoring · leases · at-least-once

A teammate has spent two days on a job handler that occasionally writes duplicate rows. They are certain the queue is delivering twice by mistake. You suspect a lease expiring under a slow handler, so the job is running concurrently with itself. Describe how you have unblocked someone in this position: what you asked before offering a hypothesis, what you showed them rather than told them, and what you left them owning. Then say what you would do if their theory turned out to be the right one.

Approach

Ask before diagnosing, and ask for things answerable from data they already have: the attempt count on the job rows that produced duplicates, the handler's observed duration against its lease expiry, and whether the duplicate rows share a natural key that a unique constraint could have caught.
Teach the shape rather than the answer. A lease cannot distinguish a dead worker from a slow one, so a handler that outruns its lease is running twice by design, and deploys deliver the other half by killing handlers mid-run on every rollout. Both of their candidate theories produce identical duplicate rows, which is why the evidence has to come from timings rather than from argument.
Hand over a checklist they execute: a natural key on every write the handler performs so the second copy collides rather than appends, the record of intent written before any external effect, a lease heartbeat while running, and the metric that shows it working.
Keep ownership with them deliberately. Pair on the first write, then step back; if you finish it yourself you have closed one ticket and left the same person stuck on the next redelivery.
Close on the systemic gap that let two days pass, which is usually a missing dashboard for attempt counts or an undocumented at-least-once contract, and fix that rather than only the bug.

Follow-up

How would you distinguish a genuine double-delivery from a lease expiry using only the data already stored?
Their handler calls an external endpoint before recording that it did. What do you tell them to change first?
What do you do the third time the same person brings you the same class of bug?

Narrate an outage you owned from page to postmortem

Hard · incident response · blast radius · postmortems

Pick an incident you personally drove, ideally one where writes were affected rather than reads. In six to eight minutes: state the symptom as it first appeared on a dashboard, the blast radius you established before you knew the cause, the mitigation you applied and when, the mechanism you eventually proved, and the follow-up that would prevent a repeat. Bring numbers: error rate, tenants affected, minutes to mitigate, minutes to resolve. If you cannot name what you measured, choose a different incident.

Approach

Open on the signal rather than the cause: which metric at which percentile moved, on which service, at what time, so the listener follows the same evidence you had rather than a conclusion you already reached.
Separate mitigation from diagnosis out loud. State what you did to stop the bleeding (flag off, shed traffic, drain a lease, roll back a deploy) and say plainly that you did it before the mechanism was known, because those are two jobs with different deadlines.
Establish blast radius in countable terms: how many tenants, how many writes, and crucially whether the effect was loss or only delay. An append-only revision table or a pending outbox row means the change survived and the projection was merely behind, which is a repair rather than a data-loss incident.
Prove the mechanism instead of asserting it. Name the trace span that grew, the plan that flipped to a sequential scan, the lease that expired, plus one alternative you ruled out and the signal that stayed flat while you ruled it out.
Close on the durable fix and its cost, distinguishing what landed that week from what needed an expand-and-contract migration across several deploys, and say which of the two you actually finished.

Follow-up

What would you do differently in the first five minutes, given the same dashboard and no more information?
Which follow-up action did you deliberately not take, and why was dropping it the right call?
How did you convince yourself the mitigation was safe to apply while the cause was still unknown?

Ship under a deadline and bound the debt you chose

Medium · pagination · technical debt · tradeoffs

You have four days to ship a tenant-facing listing endpoint. The version you would defend uses keyset pagination over (tenant_id, status, updated_at DESC, resource_id DESC); the version you can finish uses LIMIT/OFFSET with no matching index. Describe a deadline call you actually made of this shape: what you shipped, what you knowingly deferred, how you bounded the damage with a mechanism rather than an intention, and the specific numeric condition that would force the follow-up. Name who you told and where you wrote it down.

Approach

Name the deferred failure precisely instead of calling it slow. OFFSET n makes the database produce and discard n rows, so cost grows with page depth; without an index matching the sort, every matching row is read and sorted before the limit applies; and rows inserted between two page fetches shift across the boundary so items are skipped or repeated with nothing in the response to signal it.
Bound the blast radius with something mechanical rather than a promise: cap maximum page depth, cap page size, restrict the endpoint to one internal caller, or keep it behind a flag. State which failure each cap removes and which it leaves standing.
Attach a number to the trigger and wire it to an alarm: the first tenant crossing N resources, or the endpoint's p99 crossing its share of the 400 ms budget, so the debt announces itself instead of waiting to be remembered.
Write it where the next engineer looks, which is the code and the ticket, not a chat message: what was deferred, why, the cap, and the trigger.
Report what actually happened in your real example, including the case where the trigger never fired and the debt was correctly never repaid.

Follow-up

At what page depth does the offset version breach your latency budget, given your page size and row counts?
What breaks first when you switch to keyset pagination later, and what does a client holding an old page token see?
Who would have overruled you if you had asked for two more days, and did you ask?