Interview guides · Eli Lilly and

Eli Lilly and Machine Learning Engineer Interview Questions & Guide 2026

Eli Lilly and Machine Learning Engineer interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.

The 60-second brief

As a Machine Learning Engineer at Eli Lilly and, you will play a pivotal role in harnessing advanced machine learning techniques to drive innovation in pharmaceuticals. Your work will directly impact the development of therapies and solutions that improve patient outcomes. The integration of data science and machine learning within the company enables the transformation of vast datasets into actionable insights, ultimately enhancing the efficacy of drug discovery and development processes.

If the loop includes an asynchronous take-home, treat it as a code review of you rather than as a puzzle: structure, what you chose to test, and what you wrote down about the constraint you were working under. Hold the stated time box and say what you would have done with more of it.

Focus areas

Bound every outbound call with a timeout
Detect concurrent edits instead of losing writes
Trace a symptom to a mechanism under load

Overview

This role is critical as it not only contributes to the scientific and medical advancements at Eli Lilly and but also positions the company at the forefront of technological innovation in healthcare. You will collaborate with interdisciplinary teams to tackle complex challenges, leveraging your expertise to develop models that predict drug interactions, optimize clinical trials, and personalize treatment solutions. The complexity and scale of the projects you will engage with make this position both challenging and rewarding, offering a unique opportunity to make a significant difference in the lives of patients.

The Eli Lilly and interview process

Eli Lilly and candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Initial Screening Call

Half of this call is the part candidates treat as small talk: start date, notice period, work authorisation and its timing, location and time zone, on-call, and the number. Those are what kill offers late, after several engineers have each spent a day. Surfacing a hard constraint now costs you nothing and occasionally buys you something, since a loop compressed to fit a competing deadline can usually only be arranged if it is asked for early. The common failure is deflecting the compensation question twice, then discovering at offer stage that the band never reached your number.

What to demonstrate

Whether your hard constraints are compatible with the role before a loop gets booked: earliest start, notice period, what authorisation you hold and when it needs action, days on site, willingness to carry a pager
Whether you give a compensation range with something behind it, such as current total compensation or a competing timeline, rather than leaving the band untested
Whether your stated timeline is real, since a competing deadline raised now is something scheduling can sometimes work around and the same deadline raised at offer stage usually is not

How to prepare

Write each constraint down in one line before the call and state them as facts rather than negotiating them live under a question you were not expecting
Set your range from two or three current data points for that level and location, and name the structure you are quoting in, so the number is comparable to the one they are holding
If another process is running, say where it stands and by when, and ask directly whether this loop can be scheduled inside that window

In-Depth Interviews

You cannot drill a format you do not know, so put the preparation into material that travels. Three pieces of your own work, each rehearsed until you can take a follow-up you did not anticipate, will carry a conversation or a code walkthrough equally well. Specificity is what separates that from filler. A number needs its definition before it means anything: a p99 is over some window and measured at some hop, and a server-side figure excludes the queueing and network time a client would see. The number you cannot qualify is the one to leave out.

What to demonstrate

Whether your examples carry detail only someone who did the work would hold, such as what the binding constraint actually was, which alternative you rejected and why it was worse, and what you measured on each side of the change
Whether a number survives one follow-up, meaning you can say what it was measured over and whether it moved because of your change or merely alongside it
Whether a failure is described with the specific change that followed it, rather than a lesson stated in general terms
Whether your part in a team effort is stated accurately, including what other people did

How to prepare

Write a page on each of three projects covering the constraint, the option you rejected, the measurement before and after, and what went wrong. Cut any line you cannot take a follow-up on, since you are writing the parts you will be pressed on rather than a summary.
Recover the real figures while you still have access: request volume, data size, latency with its percentile and window, team size, timeline. Note where each came from, whether a dashboard, a design document or memory, and mark the estimates so you can say which they are out loud.
Take your weakest project story to someone who works in a different area and have them ask why four times in succession. The point where you run out of answer is the part to go and re-read before the round.

Team Interviews

What to demonstrate

Whether your examples carry detail only someone who did the work would hold, such as what the binding constraint actually was, which alternative you rejected and why it was worse, and what you measured on each side of the change
Whether a number survives one follow-up, meaning you can say what it was measured over and whether it moved because of your change or merely alongside it
Whether a failure is described with the specific change that followed it, rather than a lesson stated in general terms
Whether your part in a team effort is stated accurately, including what other people did

How to prepare

Write a page on each of three projects covering the constraint, the option you rejected, the measurement before and after, and what went wrong. Cut any line you cannot take a follow-up on, since you are writing the parts you will be pressed on rather than a summary.
Recover the real figures while you still have access: request volume, data size, latency with its percentile and window, team size, timeline. Note where each came from, whether a dashboard, a design document or memory, and mark the estimates so you can say which they are out loud.
Take your weakest project story to someone who works in a different area and have them ask why four times in succession. The point where you run out of answer is the part to go and re-read before the round.

Where candidates lose points

Letting a slow dependency consume unbounded concurrency

The failure that takes a service down is usually not an error but a delay. A dependency answering in thirty seconds instead of fifty milliseconds holds each request's worker or connection six hundred times longer, and since required concurrency is arrival rate times latency, a fleet sized for sixty in-flight requests now needs thirty-six thousand to sustain the same rate - so it queues, and requests whose clients have already abandoned them still occupy resources. Retries make it precisely worse: a policy of three attempts triples the load on a dependency at the exact moment it is least able to serve, which is how one slow dependency becomes an outage of everything sharing that pool. Containment is four specific things - a timeout on every outbound call shorter than the caller's remaining budget, a bounded pool per dependency so one cannot starve the others, backoff with full jitter rather than a fixed delay so retries do not resynchronise, and a circuit that stops sending once the failure rate makes an attempt pointless.

Paginating with LIMIT/OFFSET over a set that changes while the client is reading it

OFFSET n makes the database produce and discard n rows before returning anything, so the cost of a page grows with its depth rather than with its size and page 500 costs five hundred pages of work. The correctness problem is worse than the cost: if a row is inserted or reordered between two page fetches, rows shift across the offset boundary and are either skipped entirely or returned twice, and neither outcome leaves any trace in the response for the client to detect. Keyset pagination - WHERE (sort_key, id) < ($last_sort_key, $last_id) ORDER BY sort_key DESC, id DESC LIMIT n, backed by an index in exactly that order - reads only the rows it returns and is stable against concurrent inserts. It requires the tie-break column: a timestamp is not unique, and duplicate sort keys straddling a page boundary reintroduce the skip it was adopted to remove.

Finishing a solution without stating its complexity

Give time and space in the same breath as the code, and define n explicitly when there are two sizes, since n nodes and m edges are not interchangeable. Space is the half that gets skipped: count the auxiliary structures you allocate and the recursion stack at its deepest, not only the answer you hand back.

Check-then-act on shared state

Read, decide, write is not safe under concurrency unless the decision and the write are one atomic step: a unique constraint with conflict handling, a compare-and-set, or a row lock held for the whole transaction. Two requests can both pass the existence check before either inserts, which shows up as duplicate rows under load and never in a single-threaded test.

Questions & practice

Describe how you would improve an existing machine learning model's performance.

Medium · machine learning fundamentals

Approach

State the learning problem: the label, the unit of prediction and how the model is used.
Say how you would validate it, and where leakage could enter the split.
Name the simplest model that could work and what would make you move past it.

Follow-up

Where could label leakage enter this setup?
How would you know the model is overfitting?

Describe a machine learning project you have worked on and the impact it had.

Medium · machine learning fundamentals

Approach

Say how you would validate it, and where leakage could enter the split.
Name the simplest model that could work and what would make you move past it.
Pick the metric from the cost of each error type, not from habit.

Follow-up

How would you know the model is overfitting?
Where could label leakage enter this setup?

How do you handle overfitting in a machine learning model?

Medium · machine learning fundamentals

Approach

Name the simplest model that could work and what would make you move past it.
State the learning problem: the label, the unit of prediction and how the model is used.
Pick the metric from the cost of each error type, not from habit.

Follow-up

How would you know the model is overfitting?
What changes if the classes are heavily imbalanced?

What metrics would you use to evaluate the success of a machine learning model in a clinical setting?

Medium · machine learning fundamentals

Approach

State the learning problem: the label, the unit of prediction and how the model is used.
Pick the metric from the cost of each error type, not from habit.
Name the simplest model that could work and what would make you move past it.

Follow-up

How would you know the model is overfitting?
What changes if the classes are heavily imbalanced?

Track a rolling failure rate per destination for circuit decisions

Easy · sliding window · ring buffer · circuit breaker

The egress service delivers about 1,500 webhooks per second across roughly 40,000 destinations, each call bounded by a 10 second timeout. Maintain, per destination, the failure rate over the trailing 60 seconds so a caller can ask before dispatch whether the circuit should open. Attempts arrive as (destination_id, finished_at_ms, outcome). Requirement: amortised O(1) per attempt, with total memory bounded by the destination count rather than by traffic. Give the structure, its exact memory, and the rule that stops a destination with three attempts from opening a circuit.

Approach

Name the exact-deque version and then reject it as the default. Holding timestamps and advancing a tail pointer past anything older than now minus 60 seconds is a correct two-pointer window at amortised O(1) per attempt, but its memory tracks in-window traffic, so one destination in a retry storm holds hundreds of thousands of entries while thousands of quiet destinations hold none.
Use a ring of 60 one-second buckets per destination, each bucket a pair of counters for attempts and failures. On an attempt, advance the ring by the elapsed whole seconds, zeroing at most min(elapsed, 60) buckets, then increment the head. That is amortised O(1) with a fixed footprint per destination.
State the footprint: 60 buckets times two 4-byte counters is 480 bytes of payload per destination, so 40,000 destinations is roughly 20 to 25 MB with per-entry overhead, bounded by the catalogue rather than by the rate. The cost is granularity, since the oldest bucket ages out in whole seconds, which is far tighter than the decision needs.
Require a minimum sample before the circuit may open. A destination with three attempts and three failures reads as 100 percent and is not evidence; a floor of roughly 20 attempts in the window makes the ratio meaningful, and below that floor use a run of consecutive failures as the trigger instead.
Expire idle destinations, or memory grows with every destination ever seen rather than with the live set. Hold the rings in a bounded LRU keyed on destination_id and treat a miss as no history, which is the correct default for an endpoint that has been silent for a minute.
Keep the half-open probe out of the window arithmetic. After the circuit opens, one probe per interval decides whether to close it, and folding that single success into a window that still holds a 100 percent failure history would reopen the destination on one data point.

Worked solution (20 min)

Define the bucket struct and the advance step: take floor(finished_at_ms / 1000), compare with the ring's current second, zero min(delta, 60) buckets forward, then write into the new head.
Trace a destination that receives 5 attempts, goes silent for 90 seconds, then receives one more, and confirm the rate is computed from one attempt rather than six.
Compute total memory for 40,000 destinations at 60 buckets of two 4-byte counters, and state what changes if the window widens to 300 seconds.
Write the open rule as a single predicate combining the minimum-attempt floor with the rate threshold.

Expected result: A per-destination ring of 60 one-second (attempts, failures) buckets advanced lazily for amortised O(1) cost, roughly 480 bytes of counters per destination and about 20 to 25 MB for 40,000 of them, an LRU bound on live rings, and an open rule requiring both a minimum attempt count in the window and a rate above threshold.

Verify it yourself

After 60 seconds of silence a destination's window holds zero attempts and cannot open a circuit.
Memory measured with one hot destination and 39,999 idle ones is indistinguishable from the reverse, which the deque version would fail.
Replaying the same attempt stream with the clock shifted by an hour yields the same final rate.

Follow-up

The fleet is 30 instances and each sees roughly a thirtieth of a destination's traffic. Where does the rate actually live, and what does a per-instance answer get wrong?
A destination answers in 9.5 seconds and succeeds. It is not failing but it is consuming your per-destination concurrency. What signal should open the circuit here?
How would you make the window survive a process restart, and is it worth the cost?

Hold a per-tenant active cap against concurrent creates

Hard · write skew · isolation levels · constraints

A tenant on the standard plan may hold at most 50 resources with status='active'. The create handler runs SELECT count(*) FROM resource WHERE tenant_id = $1 AND status = 'active', compares to 50, then inserts. Two creates arrive 3 ms apart on different instances and the tenant lands at 51. Name the anomaly, say whether PostgreSQL 16 READ COMMITTED or REPEATABLE READ prevents it and why, then give an implementation that holds the cap at READ COMMITTED with the exact statements. Finally, say what changes when the cap is 'at most one running export per tenant' on job_run.

Approach

Name it: write skew. The two transactions read an overlapping set and write disjoint rows, so there is no row-level conflict for the engine to detect and each commit is individually legal.
Rule out the levels precisely. READ COMMITTED takes a fresh snapshot per statement and takes no lock on the counted rows, so both see 49. PostgreSQL's REPEATABLE READ is snapshot isolation: it removes non-repeatable reads and phantoms within the snapshot but still admits write skew, because the anomaly is not a re-read of a changed row, it is a read of a set that a concurrent transaction invalidates. Only SERIALIZABLE closes it, by tracking the read dependency and aborting one transaction with SQLSTATE 40001 — a guarantee that exists only if the application re-runs the whole transaction from the read.
Convert the set predicate into a single-row conflict: keep tenant.active_resource_count and run UPDATE tenant SET active_resource_count = active_resource_count + 1 WHERE tenant_id = $1 AND active_resource_count < 50 in the same transaction as the INSERT. Zero affected rows is the cap, returned as 409. The row lock serialises the decision at any isolation level, and contention is bounded to one tenant's row — which is also the fair-scheduling unit, unlike a global counter that would convoy every tenant behind one row.
State the cost you just took on: a counter is a second source of truth that can drift, so every path that changes status must adjust it inside the same transaction, and a periodic reconciliation has to exist, with resource_revision as the authority for what the count should have been.
For the job case the invariant is expressible per row, so let the database hold it: a partial unique index on job_run (tenant_id, job_type) WHERE status IN ('queued','running') makes a second running export unwritable and the loser takes 23505, mapped to 409. That is strictly better than a counter — no drift, no reconciliation — and it is available only because the cap is one rather than fifty.
Add the retry discipline each route demands: under SERIALIZABLE both 40001 and deadlock 40P01 are retryable and the retry must re-execute the read, while under READ COMMITTED with the counter nothing retries, because the conflict is reported to the caller rather than raised as an error.

Follow-up

A resource moves from archived back to active. Which statements change, and what breaks if the counter update and the status change land in different transactions?
The cap becomes plan-dependent and a plan can change mid-month. Where does the number 50 live, and who reads it?
How do you detect after the fact that the counter drifted, without locking the table?

Stop tag and share joins from fanning out a page

Medium · join fan-out · semi-joins · many-to-many

resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.

Approach

Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.
Attach the tags for display after the page has been cut — LEFT JOIN LATERAL (SELECT array_agg(rt.tag_id) FROM resource_tag rt WHERE rt.resource_id = p.resource_id) ON TRUE over the 50 returned rows. Aggregate over the page, never over the tenant.
Index both directions and say which query each serves: PK (resource_id, tag_id) serves the lateral lookup, (tag_id, resource_id) serves the EXISTS probe by tag, and resource_share needs (shared_with_user_id, resource_id) for the same reason. An index covering one direction only leaves the other as a scan.

Worked solution (30 min)

Build a tenant where each resource carries 0-5 tags from a 20-tag vocabulary and is shared with 0-4 distinct users, then bind $2 to three tags and $3 to a user holding shares on about half the resources. Run the joined query and compare its row count to the distinct resource count on page one.
Run COUNT(*) on the joined shape and on the EXISTS shape and compare both to a ground truth computed from distinct ids; then give $3 a second permission row on 10% of resources and record which of the two counts moves.
EXPLAIN both page queries and compare rows-read plus the presence of a Sort or HashAggregate node above the join.
Add (tag_id, resource_id), re-run the EXISTS probe, and record the plan change on the inner side.

Expected result: The EXISTS page returns exactly 50 distinct resources and reads about 50 driving rows with no aggregate above the join. The EXISTS count matches the ground truth. The joined COUNT(*) is larger by roughly the average number of matching tags per qualifying resource — the tags it holds from $2, not its total tag count — times its share rows for $3, which is 1 until the duplicate permission rows exist and rises with them afterwards. The EXISTS count is unchanged by those duplicates.

Verify it yourself

The page's resource_id list has no duplicates and is 50 long whenever at least 50 rows match.
count(*) from the rewrite equals count(DISTINCT r.resource_id) over the joined shape.
EXPLAIN shows a semi-join node on the inner side rather than a full join followed by a unique or aggregate step.

Follow-up

The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
A resource can be shared with the same user twice under different permissions. Does your count change, and should it?
Where does the correct total come from when the tenant holds 4M resources and the header must not cost 200 ms?

Given a dataset with missing values, how would you approach the analysis?

Medium

Approach

Say what you would check first and why it is the highest-information step.
State your assumptions explicitly before working the problem.
Clarify what is being asked and what a complete answer contains.

Follow-up

How would you know your answer was wrong?
What assumption would you test first?

What techniques do you use for feature selection?

Medium

Approach

Say what you would check first and why it is the highest-information step.
Clarify what is being asked and what a complete answer contains.
Work from the requirement backwards to the design.

Follow-up

What assumption would you test first?
How would you know your answer was wrong?

Evolve the resource contract without breaking integrations you cannot upgrade

Hard · versioning · backward compatibility · deprecation

GET /v1/resources/{id} returns status from a four-value enum ('draft','active','archived','deleted'), a numeric version, and the body inline. Consumers are a browser app you deploy and roughly 300 server-side integrations, some untouched for two years, that switch exhaustively on status and parse ids as JSON numbers. You must add a 'pending_review' status, move bodies over 256 KB to a body_ref pointer, and expose per-field change history from resource_revision. Specify the compatibility policy, the wire changes, how both generations are served, and the evidence that lets you remove the old shape.

Approach

Write the policy first and date it: fields are added, never retyped or repurposed; consumers ignore unknown fields; an unknown enum value maps to a documented fallback; nothing is removed until telemetry shows no caller reads it. Then say the uncomfortable part out loud - v1 shipped without the unknown-value rule, so 300 running integrations have no fallback, and no server change can install one into code that is already deployed.
That single fact forces per-request negotiation rather than a server-side default. Keep one internal model and select a serialiser from an explicit version in the request, and default a caller that sends nothing to the oldest supported version. Defaulting to the newest is the change that breaks every integration that never asked for anything, on the day you ship.
Downgrade 'pending_review' for old callers to the nearest state they already handle, 'draft', and state the loss explicitly: those integrations cannot see review state and will treat the resource as editable. If that is unacceptable for one integration, the remedy is moving it to the new version, not a cleverer projection - there is no mapping that invents a state the client has no code for.
Make the body change additive. Old callers keep body inline; the new shape adds body_ref and a size field, and resources over the limit are served to old callers by resolving the pointer server-side or by refusing with a documented code, chosen once and published. Never repurpose body to carry the pointer: a client that renders it shows a storage key to a user, and that failure is silent, where a missing field would have been loud. While you are here, serialise BIGINT ids as strings in the new shape - a browser parsing JSON numbers gets IEEE-754 doubles, exact only to 2^53 - and treat that as its own breaking change requiring the same negotiation, not a quiet fix.
Add change history as a separate sub-resource, GET /v1/resources/{id}/revisions, keyset-paginated over (resource_id, version) rather than as an array inside the resource. A field added to a hot response is paid for by every caller including those that never read it, and an unbounded array inside a cached object destroys the size assumptions the cache was configured with.
Retire on evidence rather than on a date alone: count requests per negotiated version per credential, publish a Sunset header (RFC 8594) with the removal date and a link to the migration, contact the credentials still on the old version, then answer 410 Gone once it is removed. Keep each version's serialiser under snapshot tests so a refactor cannot change v1's bytes by accident.

Worked solution (40 min)

Write and date the compatibility policy as five rules, then mark which of them v1 callers cannot honour.
Choose the negotiation mechanism and the default for an unversioned request, and justify the default in one sentence.
Write the downgrade table: new state to old state, new body shape to old body shape, and what is lost in each direction.
Specify the revisions sub-resource: its URL, its cursor, and why it is not a field on the resource.
Write the sunset plan: the per-version per-credential metric, the header, the lead time, and the terminal status code.

Expected result: A dated additive policy, version negotiation over one internal model with the oldest version as the default, a downgrade table with its losses named, history as a paginated sub-resource, and a removal gated on per-credential version telemetry rather than a calendar entry.

Verify it yourself

A two-year-old integration receives a resource that is really in 'pending_review' and neither crashes nor renders a blank state.
Deleting the legacy body field from the internal model breaks exactly one serialiser and no business logic.
Every v1 response is byte-stable under snapshot tests after an unrelated refactor.

Follow-up

An old integration submits a status transition while the resource is really in 'pending_review'. What does the write path accept, and what does it reject?
Two years on you want to delete the v1 serialiser. What evidence makes that safe, and who must be contacted before it happens?
How would you test against a two-year-old integration rather than against today's source?

Read latency spikes on a sixty-second sawtooth

Medium · cache stampede · single-flight · ttl · tail latency

The cached listing read path serves about 14k reads/second at an 85% hit rate. p99 sits at 35 ms for 57 seconds, jumps to 900 ms for 3, and repeats. During each spike the primary shows several hundred identical listing queries starting within the same millisecond, all carrying one large tenant's id. Cache entries use a 60-second TTL. Give the mechanism, the ordered checks, the fix, and the correctness hazard your fix must not introduce.

Approach

Match the period to a configured number before theorising about load. A spike every 60 seconds against a 60-second TTL is an entry expiring, and you confirm it by correlating spike timestamps with the entry's write time rather than with the traffic curve. If the period had matched a cron or a GC interval instead, this is a different investigation.
Establish the concurrency of the miss. Several hundred identical queries in one millisecond means the miss path has no coalescing: every request that arrives between expiry and repopulation recomputes. The herd size is that key's arrival rate times its recompute time, so at 1.2k reads/second for the hot key and a 250 ms recompute you expect about 300 concurrent misses, which matches what is observed.
Add single-flight on the miss path so one caller per key recomputes under a short-lived lock while the rest wait for its result. Prefer stale-while-revalidate where the read tolerates it: return the expired value immediately and refresh asynchronously, which removes the latency spike rather than serialising it into a queue of waiters.
De-synchronise the keys. Write TTLs with jitter, for example 60 seconds plus or minus 10%, so a deploy or a mass invalidation does not align every key on the same second and turn a per-key herd into a fleet-wide one.
Name the hazard the fix must not introduce. Serving a stale listing is acceptable only because the API reports the projection watermark, and a reader that loaded the old value before a write can repopulate the entry after the invalidation, so the bounded TTL is what actually caps staleness rather than the delete. Keep read-after-write pinned to the primary for the writing session regardless.
Verify on miss concurrency, not hit rate. The hit rate barely moves, because the herd is one miss multiplied; the number that must change is distinct origin queries per key per minute.

Follow-up

The same sawtooth appears on a key that is invalidated on write rather than expired. Is that the same bug?
How does your answer change if the recompute takes 4 seconds instead of 250 ms?
What exactly does a client see during a stale-while-revalidate window, and how does the watermark let them tell?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

Review and mentoring: what you let through

Counting review comments or mentees proves nothing. The useful version is a specific change you approved with a reservation you stated, or one you blocked and the delay that cost. Say which standard you were holding and why it was worth the friction. A mentoring story needs the thing the other person can now do without you.

What motivates you in your work, particularly in a healthcare setting?

Medium · behavioural and collaboration

Approach

Pick a story where you made the decision, not one where you watched it.
State the situation in two sentences and spend the rest on the reasoning.
Close with what you would do differently, concretely.

Follow-up

How did you know your change caused the improvement?
What did you decide not to do, and why?

How do you handle feedback and criticism of your work?

Medium · behavioural and collaboration

Approach

Give the blast radius: what could have broken, and what you measured.
State the situation in two sentences and spend the rest on the reasoning.
Close with what you would do differently, concretely.

Follow-up

What would you do differently if you ran that again?
What did you decide not to do, and why?

Reverse your own decision and price the reversal

Medium · reversibility · measurement · migrations

Describe a technical decision you made and later reversed. Pick one that cost something: a service you split and merged back, a cache you added and removed, an index you created that pushed the planner onto a worse plan, a projection you rebuilt from scratch. State what you believed when you decided, the measurement that changed your mind, how long the wrong version ran in production, and what the reversal cost in migrations, dual writes, and a deprecation window for callers you did not own.

Approach

State the original rationale without irony, in the version you would still defend given what was known then. If it is not defensible, the story is about carelessness rather than judgement, and a different example serves you better.
Give the measurement that moved with a before and after: the p99 that did not improve, the cache hit rate that sat at 40%, the plan that flipped to a sequential scan once the table passed a size you can name.
Cost the reversal in steps, not adjectives: expand-and-contract deploys, the dual-write window, the callers who had to be notified, the rows already written in the wrong shape that had to be backfilled or abandoned.
Distinguish reversal from rewrite by naming what you kept. Most good reversals preserve the schema or the interface and undo one decision inside it, which is also why they were affordable.
Finish on the process change: the smallest experiment that would have produced the same measurement in a day, and why you did not run it the first time.

Follow-up

What in that decision was irreversible, and did you know it was irreversible when you made it?
How did you tell the people who had already built on top of the original decision?
What do you now measure before committing to a change of this size?