Evernorth Machine Learning Engineer Interview Questions & Guide 2026

The 60-second brief

The Machine Learning Engineer at Evernorth plays a pivotal role in transforming healthcare through advanced technology. This position is crucial because it directly impacts the development of predictive models and algorithms that enhance patient care and operational efficiency. By leveraging machine learning techniques, you will contribute to projects that analyze vast amounts of healthcare data, enabling informed decisions that can save lives and improve patient outcomes.

The shape of the workload matters more for your prep than the industry label does. Read-heavy serving, write-heavy ingestion and scheduled batch processing have different binding constraints and fail in different places, so find out which one the team lives in before picking design topics.

Focus areas

• Choose indexes from the query's access path
• Detect concurrent edits instead of losing writes
• Paginate large result sets with keyset cursors

Overview

Your work will intersect with various teams, including data scientists, software engineers, and product managers, as you develop solutions that address complex healthcare challenges. Whether it’s optimizing clinical workflows or personalizing treatment plans, the projects you undertake will have far-reaching effects on the business and its users. This dynamic and challenging environment requires a blend of technical expertise and innovative thinking, making the role of a Machine Learning Engineer both critically important and intellectually rewarding.

The Evernorth interview process

Evernorth candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Initial Screening

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

• Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
• Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
• Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

• Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
• Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
• Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Technical Assessment

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

• Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
• Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
• Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

• Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
• Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
• Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

Behavioral Interview

What you say here is written down by each interviewer and compared afterwards, so the unit of evaluation is a claim someone else could check, not a well-told narrative. Two things make a story checkable: detail only a participant would hold, and a clean line around which part was yours. Vague ownership is the usual failure and it is usually accidental, because engineers say we about the team's work and we about their own, so the thing they personally built disappears into the plural. Name the part you wrote, and name who did the rest.

What to demonstrate

• Whether your details are ones a participant would hold and an observer would not: the constraint that ruled out the obvious approach, the first attempt that failed, the person who objected and on what grounds
• Whether ownership survives a direct question, since a follow-up to we decided is routinely who decided, and an answer that stays plural at that point is read as the work belonging to someone else
• Whether the numbers you quote are ones you would say identically to a former colleague with the dashboard open

How to prepare

• Go through each story replacing every we with either I or a named role (the on-call engineer, the reviewer, the other team) and check the story still holds together. Wherever it stops making sense you have found a part you cannot actually speak to
• Open the artefacts for two of your stories, the pull request, the design doc, the incident notes, and read them for dates and figures you have been rounding in the retelling. Correct your version to match
• For each story write the single sentence you would least want repeated to a former teammate, then either make it accurate or take it out

Project Discussion

When a round has no standard shape, it is often there because something is still open: an area no earlier conversation reached, a round where the signal came out mixed, or a decision someone is not ready to make alone. Work out which by going back over what each earlier round actually covered rather than how it felt, and arrive able to give evidence on that point without being asked twice. Weak answers replay the loop's earlier material at the same depth. Strong ones go a level deeper and stay consistent with what you already said.

What to demonstrate

• Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
• Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
• Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
• Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened

How to prepare

• Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
• Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
• Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure

Final Interviews

Where the day includes a partner from product, design or data, that conversation is weighted like the technical ones and prepared for least. They are deciding one thing: whether having you in the room makes their decisions cheaper. That means options with costs attached, not implementation detail and not "it depends". An estimate someone can plan against — a range, the assumption that would push it to the high end, and what you would drop to hit the low one — is worth more than a confident single number, which everyone present already knows is wrong.

What to demonstrate

• Whether an estimate comes as a range with the assumption most likely to break it, and states what a specific scope cut would actually buy
• Whether a technical constraint is handed over as a choice with consequences on their side, rather than as a verdict they have no standing to argue with
• Whether you establish what decision is on the table before proposing anything
• Whether risk is raised while it can still change the plan, with the trigger that would confirm it, instead of reported afterwards as a slip

How to prepare

• Take a project that shipped late and write the two-sentence warning you could have given three weeks earlier, naming what you would have needed decided at that point
• Rehearse one estimate out loud until it arrives in three parts: the range, the single assumption that would blow it, and the smallest thing you would cut to protect the date
• Rewrite an objection you have actually made — the "we can't do that" version — as two options with their costs, so the choice ends up with the person who owns it

Where candidates lose points

Running a schema change as though the lock lasts as long as the statement

In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.

Shipping a migration and the code that depends on it as a single change

During any rolling deploy, and for as long as a rollback remains possible, old and new code execute against the same schema at the same time. A migration that drops or renames a column breaks every instance that has not restarted yet, and code that requires a column the migration has not applied breaks every instance that restarted early. The discipline is expand then contract: add the new column nullable, write both shapes, backfill in batches, move reads across once the backfill is verified, and only then stop writing the old shape and drop it - four deploys, usually spread over days. It feels disproportionate until the first rollback, at which point it is the only reason the previous version still runs.

Sorting when the problem never required a total order

Match the algorithm to the guarantee actually needed: the top k comes from a size-k heap in O(n log k) time and O(k) space, distinctness needs a set rather than an ordering, and a small bounded integer key range admits a linear counting pass. A full O(n log n) sort is the right default only when you genuinely need everything in order.

Sharing mutable state with no stated owner

Say which thread, request or task owns each mutable structure, and what protects it when the answer is more than one: a lock, a queue that hands ownership across, or an immutable copy per reader. A structure documented as safe for concurrent reads is usually not safe for a concurrent write alongside those reads.

Questions & practice

Discuss the importance of feature engineering in machine learning.

Medium · machine learning fundamentals

Approach

1. Say how you would validate it, and where leakage could enter the split.
2. Pick the metric from the cost of each error type, not from habit.
3. Name the simplest model that could work and what would make you move past it.

Follow-up

• What changes if the classes are heavily imbalanced?
• Where could label leakage enter this setup?

What is overfitting, and how can it be prevented?

Medium · machine learning fundamentals

Approach

1. Pick the metric from the cost of each error type, not from habit.
2. State the learning problem: the label, the unit of prediction and how the model is used.
3. Say how you would validate it, and where leakage could enter the split.

Follow-up

• Where could label leakage enter this setup?
• How would you know the model is overfitting?

What motivates you to work in machine learning within the healthcare sector?

Medium · machine learning fundamentals

Approach

1. Pick the metric from the cost of each error type, not from habit.
2. Name the simplest model that could work and what would make you move past it.
3. State the learning problem: the label, the unit of prediction and how the model is used.

Follow-up

• Where could label leakage enter this setup?
• How would you know the model is overfitting?

What are precision and recall, and how do they relate to each other?

Medium · machine learning fundamentals

Approach

1. State the learning problem: the label, the unit of prediction and how the model is used.
2. Name the simplest model that could work and what would make you move past it.
3. Say how you would validate it, and where leakage could enter the split.

Follow-up

• How would you know the model is overfitting?
• Where could label leakage enter this setup?

Write a function to implement a binary search algorithm.

Medium · coding and algorithms

Approach

1. State the target complexity and say which constraint rules the naive version out.
2. Choose the data structure from the access pattern, not from familiarity.
3. Walk one small example through your approach before writing the whole thing.

Follow-up

• Which test case would catch an off-by-one here?
• How does this change if the input no longer fits in memory?

Solve a problem where you need to find the longest substring without repeating characters.

Medium · coding and algorithms

Approach

1. State the target complexity and say which constraint rules the naive version out.
2. Name the brute-force solution and its complexity before improving on it.
3. Choose the data structure from the access pattern, not from familiarity.

Follow-up

• Which test case would catch an off-by-one here?
• What is the worst case, and how likely is it on real data?

Find overlapping job attempts and peak concurrency from lease records

Medium · sweep line · intervals · leases

A day of job_run history yields about 50,000,000 attempt records: (job_run_id, job_type, attempt, started_at, finished_at which is NULL when the worker died, lease_expires_at). Leases expire on a clock, so a job that outran its lease ran twice. Produce (a) every job_run_id whose attempts overlapped in wall-clock time and (b) the peak number of simultaneously running attempts per job_type with the minute it occurred. Target O(n log n). State how you treat a NULL finished_at and what clock skew does to your answer.

Approach

1. Define the interval before sorting anything: an attempt occupies [started_at, COALESCE(finished_at, lease_expires_at)). finished_at is observed and lease_expires_at is only a promise, so every attempt without a finish contributes an estimate and the whole result is a lower bound on overlap rather than an exact count.
2. For peak concurrency, sweep: emit 2n endpoints, sort by (timestamp, kind) with ends ordered before starts at equal timestamps, then walk the sequence maintaining a counter per job_type and record each type's maximum with its timestamp. O(n log n) dominated by the sort, O(n) space, or O(1) extra if the sort is external and the walk streams.
3. For overlap detection, do not compare attempts pairwise. A single global sort by (job_run_id, started_at) gives both the grouping and the order; within a group, keep the maximum end seen so far and report an overlap exactly when the next start is less than that running maximum, which is one linear pass after the sort.
4. Half-open intervals matter and are easy to get wrong: with closed intervals an attempt ending at the same millisecond another begins reads as concurrency two, and across 50,000,000 records that artefact swamps the real signal.
5. State the clock caveat: started_at and finished_at are written by different workers, so under skew of a few hundred milliseconds an apparent overlap shorter than that bound is not evidence. Filter reported overlaps by a minimum duration, or prefer timestamps written by whichever component heartbeats the lease.
6. Scale the sort rather than assuming it fits: the sweep emits two endpoints per attempt, so 50,000,000 records become 100,000,000 endpoints, and at roughly 24 bytes each, an 8-byte timestamp plus a 4-byte job_type plus a kind flag padded to alignment, that is about 2.4 GB of sort keys before any scratch space. Either push the ordering into the database behind an index on (job_type, started_at) or run an external merge sort in chunks; the overlap pass sorts n records rather than 2n, so it is the cheaper of the two.

Worked solution (30 min)

1. Write the interval derivation with the COALESCE and state in one line which of the two end sources is observed and which is assumed.
2. Write the concurrency sweep: the endpoint tuples, the sort key including the end-before-start tie-break, and the per-job_type counter.
3. Hand-trace four attempts of one job, two disjoint and two overlapping by three seconds, and confirm the overlap detector fires exactly once.
4. Add the skew filter as a minimum overlap duration, state the value you chose, and justify it from how the timestamps are written.

Expected result: One global sort by (job_run_id, started_at) driving a running-maximum-end check for overlaps, plus a sweep over 2n endpoints with ends ordered before starts for per-job_type peak concurrency, both O(n log n), with half-open intervals, NULL finished_at falling back to lease_expires_at, and overlaps shorter than the clock-skew bound excluded.

Verify it yourself

• Two attempts touching at exactly the same instant report concurrency 1, not 2.
• A job with a single attempt never appears in the overlap output regardless of its lease.
• The reported peak timestamp for a job_type lies inside at least one of that type's intervals.

Follow-up

• A handler is not idempotent and you have found 400 overlapping jobs. Which of them actually caused damage, and what would you query to find out?
• Peak concurrency for one job_type is 4 against a configured cap of 4. Is the cap working, or is the data hiding attempts that never started?
• How would you compute both answers incrementally as records arrive rather than in a daily batch?

Hold a per-tenant active cap against concurrent creates

Hard · write skew · isolation levels · constraints

A tenant on the standard plan may hold at most 50 resources with status='active'. The create handler runs SELECT count(*) FROM resource WHERE tenant_id = $1 AND status = 'active', compares to 50, then inserts. Two creates arrive 3 ms apart on different instances and the tenant lands at 51. Name the anomaly, say whether PostgreSQL 16 READ COMMITTED or REPEATABLE READ prevents it and why, then give an implementation that holds the cap at READ COMMITTED with the exact statements. Finally, say what changes when the cap is 'at most one running export per tenant' on job_run.

Approach

1. Name it: write skew. The two transactions read an overlapping set and write disjoint rows, so there is no row-level conflict for the engine to detect and each commit is individually legal.
2. Rule out the levels precisely. READ COMMITTED takes a fresh snapshot per statement and takes no lock on the counted rows, so both see 49. PostgreSQL's REPEATABLE READ is snapshot isolation: it removes non-repeatable reads and phantoms within the snapshot but still admits write skew, because the anomaly is not a re-read of a changed row, it is a read of a set that a concurrent transaction invalidates. Only SERIALIZABLE closes it, by tracking the read dependency and aborting one transaction with SQLSTATE 40001 — a guarantee that exists only if the application re-runs the whole transaction from the read.
3. Convert the set predicate into a single-row conflict: keep tenant.active_resource_count and run UPDATE tenant SET active_resource_count = active_resource_count + 1 WHERE tenant_id = $1 AND active_resource_count < 50 in the same transaction as the INSERT. Zero affected rows is the cap, returned as 409. The row lock serialises the decision at any isolation level, and contention is bounded to one tenant's row — which is also the fair-scheduling unit, unlike a global counter that would convoy every tenant behind one row.
4. State the cost you just took on: a counter is a second source of truth that can drift, so every path that changes status must adjust it inside the same transaction, and a periodic reconciliation has to exist, with resource_revision as the authority for what the count should have been.
5. For the job case the invariant is expressible per row, so let the database hold it: a partial unique index on job_run (tenant_id, job_type) WHERE status IN ('queued','running') makes a second running export unwritable and the loser takes 23505, mapped to 409. That is strictly better than a counter — no drift, no reconciliation — and it is available only because the cap is one rather than fifty.
6. Add the retry discipline each route demands: under SERIALIZABLE both 40001 and deadlock 40P01 are retryable and the retry must re-execute the read, while under READ COMMITTED with the counter nothing retries, because the conflict is reported to the caller rather than raised as an error.

Worked solution (35 min)

1. Reproduce with two sessions that both count 49, both insert and both commit, at READ COMMITTED and then at REPEATABLE READ; record the final active count for each.
2. Repeat both sessions at SERIALIZABLE and record which SQLSTATE the loser receives and at which statement it is raised.
3. Implement the counter form and run a 20-way concurrent create against a tenant sitting at 45 active resources.
4. Implement the partial unique index for the job case and race 20 enqueues of the same export.

Expected result: READ COMMITTED and REPEATABLE READ both finish at 51. SERIALIZABLE finishes at 50 with one transaction aborted at 40001 — usually at commit, though SSI may raise it at an earlier statement. The counter form finishes at exactly 50: of the twenty creates, the five that take the tenant from 45 to 50 return 201 and the remaining fifteen match zero rows and return 409, with no error raised. The job index admits one running export and rejects the rest with 23505.

Verify it yourself

• count(*) WHERE status='active' never exceeds the cap across 50 randomised concurrent runs.
• Remove the active_resource_count < 50 predicate and the same test must start failing; if it still passes, the race is not being exercised.
• At SERIALIZABLE, confirm every participating session is SERIALIZABLE — one session at a lower level and the guarantee is gone.
• After each run, the counter equals the real count of active rows.

Follow-up

• A resource moves from archived back to active. Which statements change, and what breaks if the counter update and the status change land in different transactions?
• The cap becomes plan-dependent and a plan can change mid-month. Where does the number 50 live, and who reads it?
• How do you detect after the fact that the counter drifted, without locking the table?

Stop tag and share joins from fanning out a page

Medium · join fan-out · semi-joins · many-to-many

resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.

Approach

1. Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
2. Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
3. Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
4. Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.
5. Attach the tags for display after the page has been cut — LEFT JOIN LATERAL (SELECT array_agg(rt.tag_id) FROM resource_tag rt WHERE rt.resource_id = p.resource_id) ON TRUE over the 50 returned rows. Aggregate over the page, never over the tenant.
6. Index both directions and say which query each serves: PK (resource_id, tag_id) serves the lateral lookup, (tag_id, resource_id) serves the EXISTS probe by tag, and resource_share needs (shared_with_user_id, resource_id) for the same reason. An index covering one direction only leaves the other as a scan.

Follow-up

• The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
• A resource can be shared with the same user twice under different permissions. Does your count change, and should it?
• Where does the correct total come from when the tenant holds 4M resources and the header must not cost 200 ms?

Explain the difference between supervised and unsupervised learning.

Medium

Approach

1. Work from the requirement backwards to the design.
2. Say what you would check first and why it is the highest-information step.
3. Clarify what is being asked and what a complete answer contains.

Follow-up

• How would you know your answer was wrong?
• What assumption would you test first?

Choose what to break when replication lag reaches forty seconds

Hard · replication lag · read-after-write · consistency vs availability · load shedding

Reads are served from two replicas: 14k requests/second, about 85% absorbed by cache, so roughly 2.1k reads/second reach the database. Writes go to the primary at 1.2k/second. A tenant's backfill drives replication lag from under 100 ms to 40 seconds and it is still climbing. Sessions that have just written are pinned to the primary. Decide, endpoint class by endpoint class, whether to serve stale, fail, or route to the primary, and justify each choice with the load it adds to the primary. Then state what you would have built beforehand.

Approach

1. Establish blast radius before cause, because mitigation and diagnosis have different deadlines. The decisive arithmetic is what happens if the database reads move to the primary: 2.1k reads/second on top of 1.2k writes/second roughly triples its operation count, on the node already absorbing the backfill that caused this. Reads and writes are not equal in cost, so treat that as an argument against a blanket move rather than as a capacity model - but it is enough to rule out routing everything to the primary.
2. Classify endpoints by what staleness costs, not by how important they feel. Reads whose staleness is invisible - listings, search, counters - stay on the replica and return the watermark so the client can tell. Reads that immediately follow that same session's write keep their primary pin, which is a small bounded slice of traffic rather than the whole 2.1k/second. Reads that feed a decision with a side effect - authorisation, quota, the read half of a read-modify-write - must not be stale at all, because a 40-second-old permission row is the stale-permission failure wearing a different costume; those go to the primary or fail.
3. Shed instead of queueing. If the must-be-fresh class alone exceeds the primary's headroom, refuse its lowest-value slice with 503 and a retry-after. A request queued behind a saturated primary holds a connection for a client that has already given up, and the retry storm that follows is what turns degradation into an outage. Bound the connection pool per role so the read fallback cannot consume the write path's connections - that bulkhead is the single decision that determines whether writes survive the next ten minutes.
4. Attack the cause in parallel, since it is the one thing that can be stopped. The backfill is the load generator. A backfill that reads replication lag as its throttle signal and pauses above a threshold would have made this a non-event, with batch sizes small enough that each batch's write volume is a fraction of what a replica can apply per second. That is most of the answer to what should have existed beforehand.
5. Name the mechanism you would prefer over session pinning. Capture the write position at commit and require the read path to be at or past it: compare the primary's pg_current_wal_lsn() at commit time against the replica's pg_last_wal_replay_lsn(), and fall back to the primary only for the specific request that is ahead of the replica. Session pinning is the cheap approximation and it over-pins - every read in the window goes to the primary whether or not it needed to, which is a share of the cost being paid right now.

Worked solution (35 min)

1. List the endpoints in three buckets - staleness invisible, staleness visible to the writer only, staleness unsafe - and attach the share of the 2.1k reads/second each bucket carries.
2. Compute the primary's operation count under each routing option and mark which options are arithmetically available.
3. Write the pin rule and its window, then the shed rule: which slice, what status code, what retry-after.
4. Write the backfill's throttle predicate against a measured lag value, including its pause threshold and resume condition.

Expected result: Staleness-invisible reads stay on the replica and return a watermark. Reads following the session's own write keep a bounded primary pin. Reads that gate a side effect go to the primary or fail, with the lowest-value slice shed as 503 plus retry-after if headroom runs out. Connection pools are bounded per role so the read fallback cannot starve writes. The backfill is throttled on measured replication lag. The durable fix is a commit-position token compared against the replica's replay position, replacing blanket session pinning.

Verify it yourself

• The traffic routed to the primary under the plan is stated as a rate, not as a category.
• No element of the plan increases the write path's connection usage.
• The backfill throttle references a measured lag value and has an explicit resume condition.
• Every endpoint left on a replica returns something with which a client can detect staleness.

Follow-up

• Lag returns to normal in nine minutes. Which mitigation do you remove first, and which one stays permanently?
• A user reports their change did not save, and the write committed. Trace the path that produces that report and name the signal that would have shown it before the report arrived.
• The replica is 40 seconds behind but otherwise healthy. Do you take it out of rotation? What does that do to the other replica's lag?

Every query on one table stalls for forty seconds mid-deploy

Medium · lock contention · ddl · migrations · postgres

During a release on PostgreSQL, every query touching resource times out for about 40 seconds and then recovers with no intervention. The release ran one migration, ALTER TABLE resource ADD COLUMN archived_reason TEXT, and the migration log shows it completing in 6 ms. Unrelated tables showed no change in error rate. Explain how a 6 ms statement caused a 40-second stall, give the ordered checks you would run on a live system to confirm it, and give the migration procedure that prevents a repeat.

Approach

1. Separate the statement's duration from the lock's duration. ADD COLUMN with no default is a catalogue-only change and genuinely runs in milliseconds, but it requires ACCESS EXCLUSIVE, and it cannot acquire that until every transaction already touching the table has finished.
2. Account for the queueing, which is the part that surprises people. A lock request that is waiting blocks later requests for conflicting modes behind it rather than letting them overtake, so one long-open transaction holds the DDL and the DDL holds all the traffic. The stall length is set by the longest open transaction, not by the size of the change.
3. Confirm on a live system in this order: pg_stat_activity for that table ordered by xact_start, looking for the oldest transaction and specifically for state = idle in transaction; then pg_locks where granted = false to find the waiter; then join them on pid to name blocker and blocked. pg_blocking_pids() does that join for you and is the fastest single call.
4. Prevent rather than merely time it better. Set lock_timeout to a second or two on the migration session so the DDL abandons the queue after a bounded wait and is retried, instead of holding it for as long as the oldest transaction lives. Be exact about what that buys: queries arriving during the wait still queue behind the pending ACCESS EXCLUSIVE request, so each attempt costs them up to one lock_timeout of added latency. The outage goes from 40 seconds to about one second per attempt, not to zero. Also run migrations away from deploy-time peaks, and put a statement timeout and an idle-in-transaction timeout on the analytics role that opens the long transactions.
5. Know the lock each change takes, since the mitigation differs by change. A column with a non-volatile default is a metadata-only change from PostgreSQL 11 and still needs the brief ACCESS EXCLUSIVE; an index needs CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block and leaves an INVALID index to drop if it fails; a check or foreign key is added NOT VALID and then VALIDATE CONSTRAINT as a separate statement under a weaker lock.

Follow-up

• The same release also wants NOT NULL on that column. What is the sequence that gets there without a long lock?
• Your lock_timeout retry fails ten times in a row because the analytics transaction is always open. What do you change?
• How does this differ on MySQL with InnoDB online DDL, and what is the equivalent of the waiting-lock queue there?

Suggested practice checklist

For someone who has spent the last few years shipping features and reading other people's code, and who has not solved a timed problem from a blank file in a long time. Five days rebuild the primitives and the patterns that sit on them, working from invariants rather than remembered solutions, and the last two attach that back to the rest of the loop.

Day 1 — Rebuild the primitives by implementing them

• Implement a dynamic array with doubling growth and an operation counter, then change the growth rule to add a fixed sixteen slots instead, and time both for n of ten thousand, a hundred thousand and a million. The fixed-increment version resizes n/16 times at O(n) each, so its total work is quadratic; doubling is what makes append amortised constant.
• Implement a hash map with separate chaining and a load-factor resize, then insert ten thousand keys engineered to land in one bucket and record what happens to lookup time, so that average-case O(1) becomes a claim with a stated precondition rather than a reflex.
• For dynamic-array append and hash-map insert, write down which cost is amortised rather than worst-case, which single operation pays the whole bill, and what a system with a hard per-operation deadline would have to do instead.

Deliverable: Two working implementations plus a timing table showing the input at which each structure's advertised complexity stops holding.

Day 2 — Arrays under an invariant: two pointers, sliding window, binary search

• Solve longest-subarray-with-sum-at-most-K using a sliding window, then run it on an input containing negative numbers and watch it return the wrong answer: extending the window only moves the sum monotonically when every element is non-negative, and that precondition is the whole reason the technique works.
• Write the binary search that finds the first index satisfying a predicate rather than an exact value, put the loop invariant above the loop in a comment, and verify termination on the two inputs that break careless versions: the empty range, and a range where every element satisfies the predicate.
• Compute the midpoint as lo + (hi - lo) / 2 and write one line on why the obvious (lo + hi) / 2 is a genuine defect in a fixed-width integer type and a non-issue in a language with arbitrary-precision integers.

Deliverable: Three solved problems, each with its invariant written above the loop, plus one recorded input on which the sliding window is provably wrong.

Day 3 — Sorting, heaps, and the greedy argument that has to be proved

• Solve one top-k problem three ways, by full sort, by a size-k heap, and by quickselect, then write the values of n and k at which each becomes the right choice, along with quickselect's quadratic worst case and why a randomised pivot makes that unlikely rather than impossible.
• Implement bottom-up heapify and count sift-down steps to confirm it does linear work rather than n log n, because most nodes sit near the bottom of the tree and therefore move only a short distance.
• Take interval scheduling by earliest finishing time and write the exchange argument out in full: given any optimal schedule, swapping in the earliest-finishing interval keeps it feasible and no smaller. Then construct the weighted variant where that same greedy fails and name what has to replace it.

Deliverable: A three-way top-k comparison with measured crossover points, one written exchange argument, and one counterexample to a greedy rule that looks almost identical.

Day 4 — Recursion, memoisation, and the step to a table

• Take one problem with overlapping subproblems, such as edit distance or coin change, instrument the plain recursion with a call counter to show the blow-up, then add memoisation and re-count.
• Convert the memoised version to a bottom-up table and state the two properties you relied on: each subproblem's result depends only on its arguments, and the dependencies form a DAG you can enumerate in order.
• Rewrite one deep recursion with an explicit stack, then find the input length at which the original hits the interpreter's frame limit, which defaults to about a thousand frames in CPython, so you know when the rewrite is required rather than decorative.

Deliverable: One problem in three forms, naive, memoised and tabulated, with call counts for each and the input length at which recursion depth becomes the binding constraint.

Day 5 — Graphs, where most of the work is choosing the traversal

• Implement BFS and DFS over one adjacency list, then answer for each which finds a shortest path in an unweighted graph and which you would use to detect a cycle in a directed graph, including why the in-progress versus finished distinction matters for the second.
• Implement topological sort by in-degree, feed it a graph containing a cycle, and confirm the failure signature is that fewer than V nodes come out rather than an exception, then note that the order it produces is one of several valid ones.
• Run a shortest-path search on a graph with a single negative edge weight and show the wrong answer, then write the precondition Dijkstra actually needs, non-negative weights, because it finalises a node's distance the first time that node is popped, and name the algorithm you would switch to and its own limit.

Deliverable: A small graph library with BFS, DFS and topological sort, plus two inputs that produce documented wrong answers under the wrong algorithm choice.

Day 6 — One day for everything that is not an algorithm

• Sketch one system only to the depth a coding-heavy loop tends to reach: the endpoints, what the service stores, and the single query pattern that decides the schema. Stop at twenty-five minutes.
• Prepare the project answer for an interviewer who codes, which means rehearsing the two levels they push to: the specific thing you built, and why you chose that approach over the alternative they will name. Open with a number and be ready to say what it excludes.
• Prepare the answer to what you would do differently, choosing a real technical mistake with a specific fix rather than a complaint about process or staffing.

Deliverable: One design sketch at endpoint-and-schema depth, plus a project answer rehearsed to two levels of follow-up.

Day 7 — Solve out loud, under time

• Do three timed problems at twenty-five minutes each in a plain editor with no autocomplete and no execution until the end, then tally separately the failures that were syntax and the ones that were approach, because those two numbers call for different fixes.
• Narrate one solution from the first sentence, stating the approach and its complexity before writing any code, and rehearse the sentence you will use when you realise mid-solution that the approach is wrong.
• Re-solve from blank the two problems you were slowest on this week and compare the times against the day they first appeared.

Deliverable: A recording of one fully narrated solution and a tally that separates syntax failures from approach failures.

Disagreement needs an outcome you did not choose

Conflict answers where you were right and everyone came round are the weakest ones. Stronger: the evidence you went and collected, what would have changed your mind, and what you did in the weeks after the call went against you. Implementing a design you argued against, properly, is a specific and checkable behaviour.

Describe a challenging project you worked on. What was your role, and what was the outcome?

Medium · behavioural and collaboration

Approach

1. Give the blast radius: what could have broken, and what you measured.
2. Close with what you would do differently, concretely.
3. Pick a story where you made the decision, not one where you watched it.

Follow-up

• How did you know your change caused the improvement?
• What did you decide not to do, and why?

Can you give an example of how you have worked collaboratively in a team?

Medium · behavioural and collaboration

Approach

1. Give the blast radius: what could have broken, and what you measured.
2. Name the disagreement and how you resolved it with evidence.
3. Close with what you would do differently, concretely.

Follow-up

• How did you know your change caused the improvement?
• What would you do differently if you ran that again?

Ship under a deadline and bound the debt you chose

Medium · pagination · technical debt · tradeoffs

You have four days to ship a tenant-facing listing endpoint. The version you would defend uses keyset pagination over (tenant_id, status, updated_at DESC, resource_id DESC); the version you can finish uses LIMIT/OFFSET with no matching index. Describe a deadline call you actually made of this shape: what you shipped, what you knowingly deferred, how you bounded the damage with a mechanism rather than an intention, and the specific numeric condition that would force the follow-up. Name who you told and where you wrote it down.

Approach

1. Name the deferred failure precisely instead of calling it slow. OFFSET n makes the database produce and discard n rows, so cost grows with page depth; without an index matching the sort, every matching row is read and sorted before the limit applies; and rows inserted between two page fetches shift across the boundary so items are skipped or repeated with nothing in the response to signal it.
2. Bound the blast radius with something mechanical rather than a promise: cap maximum page depth, cap page size, restrict the endpoint to one internal caller, or keep it behind a flag. State which failure each cap removes and which it leaves standing.
3. Attach a number to the trigger and wire it to an alarm: the first tenant crossing N resources, or the endpoint's p99 crossing its share of the 400 ms budget, so the debt announces itself instead of waiting to be remembered.
4. Write it where the next engineer looks, which is the code and the ticket, not a chat message: what was deferred, why, the cap, and the trigger.
5. Report what actually happened in your real example, including the case where the trigger never fired and the debt was correctly never repaid.

Follow-up

• At what page depth does the offset version breach your latency budget, given your page size and row counts?
• What breaks first when you switch to keyset pagination later, and what does a client holding an old page token see?
• Who would have overruled you if you had asked for two more days, and did you ask?