Interview guides · MarshBerry

MarshBerry Machine Learning Engineer Interview Questions & Guide 2026

MarshBerry Machine Learning Engineer interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.

The 60-second brief

A Machine Learning Engineer at MarshBerry plays a pivotal role in enhancing the organization’s capabilities in health and fitness through advanced algorithms and machine learning systems. This role is critical as it directly influences the development of features that empower users to understand their health better and improve their performance. As part of a cross-functional team, you will engage in creating innovative solutions that integrate physiological data with intelligent insights, ultimately impacting millions of users.

The title spans product, platform and infrastructure work, and which of those the seat actually is decides whether design or algorithms carries more weight in your preparation. The posting rarely settles it; what the team is on call for usually does.

Focus areas

Choose indexes from the query's access path
Make every write idempotent under retry
Trace a symptom to a mechanism under load

Overview

By leveraging cutting-edge technologies and methodologies, you will not only optimize algorithms for real-time performance but also contribute to the strategic direction of MarshBerry’s product offerings. The complexity of working with both edge and cloud environments adds a layer of excitement and challenge, making this position integral to MarshBerry’s mission of unlocking human performance and enhancing healthspan. Your contributions will be at the forefront of delivering personalized coaching and insights, helping to shape the future of health technology.

The MarshBerry interview process

MarshBerry candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Application Review

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Technical Interviews

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Behavioral Assessments

Many of these questions are about something that went wrong, and the grading sits mostly in the hours after you knew. Who found out first, whether that was you or an alert or a user, how long it took you to say it out loud, and whether the people who needed the news got it while they could still act on it. Engineers under-tell this part because it feels like confessing. The pattern it is looking for is the opposite: the quiet fix, an incident absorbed without telling anyone, after which nothing changed and the same failure is still available.

What to demonstrate

How the problem was found, and whether that route was one you had built or one that happened to you, since a user reporting it first means your instrumentation did not cover that failure
Whether time-to-detect and time-to-tell are separate numbers in your account and whether you know both, because a fast fix that nobody heard about until the retro is a different answer from a slow one that was announced immediately
Whether the resolution left something durable behind, a check that fires or a default that changed, rather than depending on people remembering to be careful
Whether you can say what the failure cost without either inflating it or waving it away

How to prepare

Reconstruct one incident you were part of as a timeline with clock times: first bad request, first signal, first person who knew, first message outside the team, mitigation, permanent fix. The gaps between those entries are what gets asked about
Look up the configuration of the signal that caught it, including its evaluation window and threshold. An alert defined on a five-minute aggregate cannot fire until the condition holds across that window, which puts a floor under time-to-detect that has nothing to do with how severe the failure was. Be able to say what that floor was and whether anyone had chosen it deliberately
Prepare one story where you escalated early and the severity turned out to be smaller than you thought, including what it cost the people you pulled in. Without it, every answer you give about raising alarms is unfalsifiable

Coding Exercises

What to demonstrate

In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Final Interviews

Where the day includes a partner from product, design or data, that conversation is weighted like the technical ones and prepared for least. They are deciding one thing: whether having you in the room makes their decisions cheaper. That means options with costs attached, not implementation detail and not "it depends". An estimate someone can plan against — a range, the assumption that would push it to the high end, and what you would drop to hit the low one — is worth more than a confident single number, which everyone present already knows is wrong.

What to demonstrate

Whether an estimate comes as a range with the assumption most likely to break it, and states what a specific scope cut would actually buy
Whether a technical constraint is handed over as a choice with consequences on their side, rather than as a verdict they have no standing to argue with
Whether you establish what decision is on the table before proposing anything
Whether risk is raised while it can still change the plan, with the trigger that would confirm it, instead of reported afterwards as a slip

How to prepare

Take a project that shipped late and write the two-sentence warning you could have given three weeks earlier, naming what you would have needed decided at that point
Rehearse one estimate out loud until it arrives in three parts: the range, the single assumption that would blow it, and the smallest thing you would cut to protect the date
Rewrite an objection you have actually made — the "we can't do that" version — as two options with their costs, so the choice ends up with the person who owns it

Where candidates lose points

Running a schema change as though the lock lasts as long as the statement

In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.

Choosing an index from the columns a query mentions rather than from how it filters and orders

A composite B-tree index on (a, b, c) can be seeked only as a left prefix: equality on a, then equality on b, then a range or an ordering on c. A query that filters on b alone cannot seek into it at all and at best gets a full scan of the index; a query that filters a and ranges on b gets no benefit from c, because the index is only sorted by c within a fixed (a, b) pair. The practical consequence is that one index per column is close to useless for multi-predicate queries while a single correctly ordered composite index turns a scan into a lookup. The ordering half is what gets missed: if the index cannot satisfy the ORDER BY, the database must read every matching row and sort before the limit can apply, so a LIMIT 20 over a million matching rows still reads a million rows.

A cache with no invalidation story

Say how an entry goes stale, how long you can serve it stale, and what happens when many requests miss the same key at the same instant. One popular key expiring under load sends every concurrent request to the origin together; single-flight coalescing, jittered expiry, or serving stale while revalidating are the standard answers.

Designing for a scale nobody asked for

Ask for request rate, data size, read-to-write ratio and expected growth, then size the simplest option first; one relational instance on current hardware covers a large share of real workloads. Reaching for shards, queues and a cache tier before any number has been quoted reads as pattern-matching rather than judgement.

Questions & practice

Design a machine learning system that analyzes health metrics from wearable devices.

Medium · machine learning fundamentals

Approach

Pick the metric from the cost of each error type, not from habit.
State the learning problem: the label, the unit of prediction and how the model is used.
Name the simplest model that could work and what would make you move past it.

Follow-up

How would you know the model is overfitting?
Where could label leakage enter this setup?

Describe a time when you successfully deployed a machine learning model to production.

Medium · machine learning fundamentals

Approach

Name the simplest model that could work and what would make you move past it.
Pick the metric from the cost of each error type, not from habit.
Say how you would validate it, and where leakage could enter the split.

Follow-up

What changes if the classes are heavily imbalanced?
Where could label leakage enter this setup?

Given a dataset of health metrics, how would you approach feature engineering?

Medium · machine learning fundamentals

Approach

State the learning problem: the label, the unit of prediction and how the model is used.
Say how you would validate it, and where leakage could enter the split.
Pick the metric from the cost of each error type, not from habit.

Follow-up

What changes if the classes are heavily imbalanced?
Where could label leakage enter this setup?

Implement a simple decision tree algorithm from scratch.

Medium · machine learning fundamentals

Approach

Name the simplest model that could work and what would make you move past it.
Pick the metric from the cost of each error type, not from habit.
Say how you would validate it, and where leakage could enter the split.

Follow-up

What changes if the classes are heavily imbalanced?
Where could label leakage enter this setup?

Write a function to calculate the mean and standard deviation of a given list of numbers.

Medium · coding and algorithms

Approach

Name the brute-force solution and its complexity before improving on it.
Choose the data structure from the access pattern, not from familiarity.
State the target complexity and say which constraint rules the naive version out.

Follow-up

What is the worst case, and how likely is it on real data?
Which test case would catch an off-by-one here?

How would you optimize a given algorithm for speed and efficiency?

Medium · coding and algorithms

Approach

State the target complexity and say which constraint rules the naive version out.
Choose the data structure from the access pattern, not from familiarity.
Walk one small example through your approach before writing the whole thing.

Follow-up

How does this change if the input no longer fits in memory?
Which test case would catch an off-by-one here?

Archive a resource graph without breaking live references or recursing

Medium · graph traversal · topological order · tenant isolation

Resources reference other resources within a tenant; for the largest tenant the reference table holds up to 2,000,000 nodes and 8,000,000 edges. Archiving a resource must archive everything reachable from it that nothing outside the set still references, refuse when a live external referrer exists, and terminate when references form cycles, which they legitimately do. Produce the archive order and the refusal list, targeting O(V+E). Say what stops the traversal crossing a tenant boundary, and why recursion is the wrong control structure at this size.

Approach

Load the subgraph with the tenant predicate on both endpoints of the edge, not only on the side you started from. Scoping the left table alone is the classic cross-tenant leak: one mis-entered edge then pulls another tenant's resources into the traversal and, worse, into the archive.
Traverse iteratively with an explicit stack. A 2,000,000-node graph can hold a chain deep enough to exhaust a native stack in the low tens of thousands of frames, and that failure is a process crash rather than an error you can return.
Treat cycles as data rather than corruption: compute strongly connected components with Tarjan in O(V+E) using its own explicit stack, then condense. The condensation is a DAG, so a topological order over it gives the archive order, and every member of a component archives in one transaction because no order within a cycle is valid.
Decide refusals with reverse edges. A candidate is archivable only if every in-edge originates inside the candidate set, so build the transpose or count in-degrees restricted to the visited set, and emit each blocked resource with the id of the external referrer, which is the only part of the answer an operator can act on.
Store the graph as CSR rather than a map of lists: an offsets array of V+1 8-byte entries plus E 8-byte targets is about 80 MB at this size, where boxed adjacency lists cost several times that and lose cache locality on every hop.
Run Kahn over the condensation for the order in O(V+E). If the emitted count is short of the component count the condensation step itself is wrong, since a condensation cannot contain a cycle, which makes the check free.

Worked solution (30 min)

Write the edge-loading query with the tenant predicate on both endpoints and state what it does with a cross-tenant edge.
Implement iterative Tarjan with an explicit stack and confirm on a three-node cycle that it emits one component of size three.
Build the transpose restricted to the visited set and mark every node with an in-edge from outside it as refused, carrying the referrer id.
Run Kahn over the condensation and verify the emitted order against the referrer-before-referenced rule.
Size the CSR arrays for 2,000,000 nodes and 8,000,000 edges and compare against a boxed adjacency map.

Expected result: An iterative O(V+E) traversal over a tenant-scoped CSR subgraph, SCC condensation so cycles archive atomically as one component, a transpose-based refusal list naming the external referrer for each blocked resource, and a Kahn topological order over the condensation, with recursion replaced by an explicit stack because of graph depth rather than style.

Verify it yourself

A two-node cycle with no external referrers archives as one unit, and adding a single external referrer moves both nodes to the refusal list.
No resource appears in the archive order before something that references it, except within its own component.
A counter on edge visits matches the loaded edge count exactly, confirming each edge is traversed once.

Follow-up

The graph is read in one query and the archive writes a minute later. What can change in between, and how do you make the write safe?
The candidate set is 400,000 resources. Is that one transaction, and if not, what does a half-finished archive look like to a reader?
An edge points at a resource in another tenant. Is that a refusal, an error, or an alert?

Keep soft-deleted accounts from blocking re-registration

Easy · soft delete · unique constraints · engine differences

app_user holds user_id, tenant_id, email CITEXT, password_hash (NULL for SSO principals), email_verified_at, auth_version, status ('invited','active','suspended','deactivated'), created_at, updated_at, deleted_at. Two live accounts for one address inside a tenant must be impossible, but an address freed by a soft delete must be reusable, and the same tenant may delete and re-register it repeatedly. Write the uniqueness DDL for PostgreSQL 16, then the equivalent for MySQL 8 where partial indexes do not exist, and say what each permits once three deleted rows already hold that address.

Approach

Start from what is actually unique: not (tenant_id, email), but (tenant_id, email) among live rows. PostgreSQL says that directly — CREATE UNIQUE INDEX app_user_live_email ON app_user (tenant_id, email) WHERE deleted_at IS NULL. A full constraint over the same two columns burns the address permanently the first time someone deletes an account.
Keep case-insensitivity in the type or the index, never in the application: CITEXT as given, or UNIQUE (tenant_id, lower(email)) as an expression index where the extension is unavailable. A case-sensitive unique column is exactly how two accounts for one human appear.
For MySQL 8 the predicate has to move inside the key: add a discriminator column that is a constant 0 while the row is live and is set to user_id on delete, with UNIQUE (tenant_id, email, deleted_marker). Live rows share the constant and still collide; deleted rows differ from each other and stop colliding.
State the NULL variant and its dependency: leaving the marker NULL for deleted rows also works, because a unique index treats NULLs as distinct — true in MySQL, and true in PostgreSQL only under the default NULLS DISTINCT, which PostgreSQL 15 lets you reverse. Check the polarity against the three existing deleted rows: constant-on-live is what preserves the collision you want, and reversing it silently admits duplicate live accounts.
Say what a soft delete must do besides setting deleted_at: increment auth_version so existing tokens stop validating, leave resource.owner_user_id and resource_revision.actor_user_id intact, and accept that the address is retained — erasure is a different requirement answered by scrubbing the column, not by a DELETE that would break those references.

Worked solution (20 min)

Create the PostgreSQL partial unique index, insert a live row, soft delete it, and insert the same address again.
Repeat the delete-and-reinsert cycle three times and confirm three deleted rows coexist with exactly one live row.
Write the MySQL form with the discriminator, then deliberately reverse the polarity so live rows carry NULL, and show two live duplicates commit.
Attempt a second live insert on both engines and map the resulting 23505 / ER_DUP_ENTRY to the 409 the handler should return.

Expected result: PostgreSQL rejects only the second live row and admits unlimited deleted ones. MySQL behaves identically with UNIQUE (tenant_id, email, deleted_marker) where live = 0; with the polarity reversed, two live rows commit and nothing afterwards can tell them apart.

Verify it yourself

Group live rows by (tenant_id, lower(email)) after a randomised insert/delete run — no group exceeds one.
Inserting 'User@x.test' after 'user@x.test' is rejected; if it is accepted, the case handling is in application code rather than in the index.
EXPLAIN a login lookup and confirm the partial index is chosen — it is only usable by queries that carry the deleted_at IS NULL predicate, so a query without it falls back to a scan.

Follow-up

A deleted account re-registers with the same address the next day. Do the old resource rows follow the new user_id, and how does the API keep the two principals apart?
How do you honour an erasure request while resource_revision.actor_user_id still references this table?
What changes if a user may hold membership in two tenants?

Write the update path that detects a concurrent edit

Medium · optimistic concurrency · transactional outbox · audit history

resource carries version INT NOT NULL DEFAULT 1. resource_revision holds revision_id, resource_id, version, actor_user_id, change_kind, patch JSONB, request_id, created_at with UNIQUE (resource_id, version). outbox_event holds aggregate_type, aggregate_id, aggregate_version, event_type, payload, status. A PUT carries the version the client read. Write the exact statements for the single transaction that applies the edit, records the revision and enqueues 'resource.updated', and give the handler's branch on zero affected rows. Then say what PostgreSQL 16 does under READ COMMITTED when two of these updates hit one row at once.

Approach

One transaction, three writes, no network call inside it: UPDATE resource SET title = $3, version = version + 1, updated_at = now() WHERE resource_id = $1 AND tenant_id = $4 AND version = $2; then INSERT the resource_revision row at version $2 + 1; then INSERT the outbox_event row at the same aggregate_version. The event goes to a table rather than a broker because no transaction spans both.
Branch on the affected-row count before doing anything else. Zero has three causes — stale version, wrong tenant, row gone — so re-read once and map to 409 carrying the current version, or 404 for an id outside the caller's tenant, which also stops the endpoint confirming that another tenant's id exists.
State the engine behaviour instead of assuming it. Under READ COMMITTED the second UPDATE blocks on the row lock, and when the first commits PostgreSQL re-evaluates the WHERE clause against the newly committed row, so the version predicate now fails and the statement reports zero rows. Under REPEATABLE READ the identical collision raises SQLSTATE 40001 instead, so the handler must fold both shapes into one conflict response.
Keep UNIQUE (resource_id, version) even though the predicate already serialises writers. It is what makes a lost update unwritable if any other path ever reaches the revision table, and it converts a logic bug into 23505 rather than into a silently missing history row.
Refuse to auto-retry the whole PUT. A retry re-reads the winner's state and reapplies an intent formed against data that no longer exists — the silent overwrite the version token was added to detect. Return the conflict; merge field-wise only if the patches are provably disjoint.
Note that now() is the transaction timestamp in PostgreSQL, so resource.updated_at, the revision's created_at and the outbox row share one instant, which is what later makes reconciliation between the three tables unambiguous.

Follow-up

A client sends the version it read ten minutes ago and the resource has moved three versions. What is in your 409 so it can resolve the conflict without a full re-fetch?
Two editors, two disjoint fields, no overlap. Does your answer still refuse the second write, and should it?
Every write now touches a second hot table. How do you keep the outbox insert and its partial index from becoming the write bottleneck at 1.2k writes/second?

How would you optimize a system for low latency and high throughput in a production environment?

Medium · ml system design and production

Approach

Separate the offline training path from the online serving path.
Name what you would monitor after launch and what triggers a retrain.
Fix the product goal and the online metric before choosing any model.

Follow-up

How would you roll the new model out safely?
What happens when a feature is missing at serving time?

Discuss the considerations for deploying models in both edge and cloud environments.

Medium · ml system design and production

Approach

Name what you would monitor after launch and what triggers a retrain.
Fix the product goal and the online metric before choosing any model.
Separate the offline training path from the online serving path.

Follow-up

How would you roll the new model out safely?
What happens when a feature is missing at serving time?

Explain the differences between supervised and unsupervised learning.

Medium

Approach

Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer contains.
Work from the requirement backwards to the design.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Design the async export contract a client can resume safely

Hard · async operations · idempotency · leases

A tenant asks for a CSV of every resource. The work runs for minutes on the worker fleet through a job_run row carrying a lease, an attempt count and a dedupe_key, far past the edge's 400 ms budget. Callers are a browser that polls and a script that walks away and checks later. Specify what the submit call returns, the operation resource and its states, how a duplicate submit is handled, how a client learns about completion, what cancellation means given that a lease can expire mid-run, and how the result is fetched and when it expires.

Approach

Split the API in two. Submit returns 202 with an operation id and a location to poll, and never blocks on the work. The operation is a real resource with its own lifecycle - queued, running, succeeded, failed, cancelled - plus attempt, a monotonic progress figure, and a terminal error drawn from the same code taxonomy the synchronous endpoints use, so a client needs one error vocabulary rather than two.
Deduplicate at submit using job_run.dedupe_key, unique over (job_type, dedupe_key) while status is 'queued' or 'running': a repeat submit of the same logical export returns 200 with the existing operation instead of 202 with a new one, and the partial index deliberately permits a legitimate re-run once the first has finished. Pair it with the request's idempotency key so an HTTP-level retry of the submit is exact rather than merely similar.
Tell the poller how to poll: Retry-After on the polling response, a minimum interval enforced at the edge, and a documented maximum lifetime after which an operation is reaped. Polling is the contract of record; the webhook is the fast path, and both must lead to the same terminal state, so a client that receives the completion event and then polls anyway sees no contradiction.
Be exact about cancellation. A cancel request records intent; it cannot stop work already executing. The handler reads the flag at checkpoints, and because a lease expires on a clock that cannot distinguish a dead worker from a slow one, a second copy may start after the cancel was recorded - so the handler re-reads the flag immediately after claiming the lease. 'cancelled' becomes terminal only when no lease is outstanding; reporting it earlier shows a client a stopped job while a worker is still writing output.
Make the handler safe to run twice, because the lease guarantees that it will be. Write output to a deterministic object key derived from the operation id so a second copy overwrites its own work instead of appending a second file, and record completion with a conditional update that only the copy holding the current lease can win.
Treat result fetch as a separate authorised read: a short-lived signed URL, the tenant checked when it is issued rather than only when the file was produced, and a documented retention after which the operation remains terminal but the bytes are gone - a state the client must be able to tell apart from a failure.

Worked solution (40 min)

Write the submit request and its two possible responses, 202 for new and 200 for a duplicate, and the dedupe_key construction.
Draw the operation state machine, marking which transitions a client may observe and which are terminal.
Write the cancellation sequence across a lease expiry, showing where the second copy reads the flag.
Define the output key, the completion update's predicate, and why both are needed for a double run.
Specify result fetch: URL lifetime, authorisation point, retention, and the distinct response once the bytes are gone.

Expected result: A 202-plus-operation contract with deduplicated submits, a documented poll interval alongside an optional completion webhook, cancellation defined as recorded intent with a terminal state gated on lease ownership, an idempotent handler writing to a deterministic key, and a separately authorised expiring result.

Verify it yourself

Submitting the same export twice while the first is running produces one job_run row and one operation id.
Killing a worker mid-export and letting the lease expire yields one output file, not two, and one terminal transition.
A client that receives the completion webhook and then polls sees the same terminal state and the same result location.

Follow-up

An operation has said 'running' for 40 minutes and the worker is gone. What does the client see, and which columns in job_run decide that?
Two tenants each submit 50 exports at once. What in this contract stops one of them delaying the other?
The customer wants the export emailed instead. What changes, and what becomes harder to make exactly-once?

Read latency spikes on a sixty-second sawtooth

Medium · cache stampede · single-flight · ttl · tail latency

The cached listing read path serves about 14k reads/second at an 85% hit rate. p99 sits at 35 ms for 57 seconds, jumps to 900 ms for 3, and repeats. During each spike the primary shows several hundred identical listing queries starting within the same millisecond, all carrying one large tenant's id. Cache entries use a 60-second TTL. Give the mechanism, the ordered checks, the fix, and the correctness hazard your fix must not introduce.

Approach

Match the period to a configured number before theorising about load. A spike every 60 seconds against a 60-second TTL is an entry expiring, and you confirm it by correlating spike timestamps with the entry's write time rather than with the traffic curve. If the period had matched a cron or a GC interval instead, this is a different investigation.
Establish the concurrency of the miss. Several hundred identical queries in one millisecond means the miss path has no coalescing: every request that arrives between expiry and repopulation recomputes. The herd size is that key's arrival rate times its recompute time, so at 1.2k reads/second for the hot key and a 250 ms recompute you expect about 300 concurrent misses, which matches what is observed.
Add single-flight on the miss path so one caller per key recomputes under a short-lived lock while the rest wait for its result. Prefer stale-while-revalidate where the read tolerates it: return the expired value immediately and refresh asynchronously, which removes the latency spike rather than serialising it into a queue of waiters.
De-synchronise the keys. Write TTLs with jitter, for example 60 seconds plus or minus 10%, so a deploy or a mass invalidation does not align every key on the same second and turn a per-key herd into a fleet-wide one.
Name the hazard the fix must not introduce. Serving a stale listing is acceptable only because the API reports the projection watermark, and a reader that loaded the old value before a write can repopulate the entry after the invalidation, so the bounded TTL is what actually caps staleness rather than the delete. Keep read-after-write pinned to the primary for the writing session regardless.
Verify on miss concurrency, not hit rate. The hit rate barely moves, because the herd is one miss multiplied; the number that must change is distinct origin queries per key per minute.

Follow-up

The same sawtooth appears on a key that is invalidated on write rather than expired. Is that the same bug?
How does your answer change if the recompute takes 4 seconds instead of 250 ms?
What exactly does a client see during a stale-while-revalidate window, and how does the watermark let them tell?

Suggested practice checklist

For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.

Day 1 — Measure before reasoning

Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.

Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.

Day 2 — References, copies, and the bugs they produce

Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.

Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.

Day 3 — Types, once, in a language that checks them

Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.

Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.

Day 4 — Concurrency, starting with what actually runs at the same time

Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.

Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.

Day 5 — Debugging as a procedure rather than an instinct

Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.

Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.

Day 6 — Tests that catch the bug you are about to write

Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.

Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.

Day 7 — Debug something broken, out loud

Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.

Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.

Know where your numbers came from

Bring the two or three numbers the story rests on and know how they were collected. A p99 whose timer starts inside your handler excludes the time a request spent queued, so it can sit flat while users wait longer. Give the window, the percentile and what the measurement left out, or drop the number.

How do you prioritize tasks when working on multiple projects?

Medium · behavioural and collaboration

Approach

Close with what you would do differently, concretely.
Name the disagreement and how you resolved it with evidence.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What did you decide not to do, and why?
How did you know your change caused the improvement?

Ship under a deadline and bound the debt you chose

Medium · pagination · technical debt · tradeoffs

You have four days to ship a tenant-facing listing endpoint. The version you would defend uses keyset pagination over (tenant_id, status, updated_at DESC, resource_id DESC); the version you can finish uses LIMIT/OFFSET with no matching index. Describe a deadline call you actually made of this shape: what you shipped, what you knowingly deferred, how you bounded the damage with a mechanism rather than an intention, and the specific numeric condition that would force the follow-up. Name who you told and where you wrote it down.

Approach

Name the deferred failure precisely instead of calling it slow. OFFSET n makes the database produce and discard n rows, so cost grows with page depth; without an index matching the sort, every matching row is read and sorted before the limit applies; and rows inserted between two page fetches shift across the boundary so items are skipped or repeated with nothing in the response to signal it.
Bound the blast radius with something mechanical rather than a promise: cap maximum page depth, cap page size, restrict the endpoint to one internal caller, or keep it behind a flag. State which failure each cap removes and which it leaves standing.
Attach a number to the trigger and wire it to an alarm: the first tenant crossing N resources, or the endpoint's p99 crossing its share of the 400 ms budget, so the debt announces itself instead of waiting to be remembered.
Write it where the next engineer looks, which is the code and the ticket, not a chat message: what was deferred, why, the cap, and the trigger.
Report what actually happened in your real example, including the case where the trigger never fired and the debt was correctly never repaid.

Follow-up

At what page depth does the offset version breach your latency budget, given your page size and row counts?
What breaks first when you switch to keyset pagination later, and what does a client holding an old page token see?
Who would have overruled you if you had asked for two more days, and did you ask?

Narrate an outage you owned from page to postmortem

Hard · incident response · blast radius · postmortems

Pick an incident you personally drove, ideally one where writes were affected rather than reads. In six to eight minutes: state the symptom as it first appeared on a dashboard, the blast radius you established before you knew the cause, the mitigation you applied and when, the mechanism you eventually proved, and the follow-up that would prevent a repeat. Bring numbers: error rate, tenants affected, minutes to mitigate, minutes to resolve. If you cannot name what you measured, choose a different incident.

Approach

Open on the signal rather than the cause: which metric at which percentile moved, on which service, at what time, so the listener follows the same evidence you had rather than a conclusion you already reached.
Separate mitigation from diagnosis out loud. State what you did to stop the bleeding (flag off, shed traffic, drain a lease, roll back a deploy) and say plainly that you did it before the mechanism was known, because those are two jobs with different deadlines.
Establish blast radius in countable terms: how many tenants, how many writes, and crucially whether the effect was loss or only delay. An append-only revision table or a pending outbox row means the change survived and the projection was merely behind, which is a repair rather than a data-loss incident.
Prove the mechanism instead of asserting it. Name the trace span that grew, the plan that flipped to a sequential scan, the lease that expired, plus one alternative you ruled out and the signal that stayed flat while you ruled it out.
Close on the durable fix and its cost, distinguishing what landed that week from what needed an expand-and-contract migration across several deploys, and say which of the two you actually finished.

Follow-up

What would you do differently in the first five minutes, given the same dashboard and no more information?
Which follow-up action did you deliberately not take, and why was dropping it the right call?
How did you convince yourself the mitigation was safe to apply while the cause was still unknown?