Interview guides · Providence
Providence Machine Learning Engineer Interview Questions & Guide 2026
Providence Machine Learning Engineer interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.
The 60-second brief
A Machine Learning Engineer at Providence plays a pivotal role in harnessing the power of data to drive innovative healthcare solutions. This position is central to developing machine learning models that improve patient outcomes, enhance operational efficiencies, and influence strategic decision-making across the organization. As a Machine Learning Engineer, you will be at the intersection of technology and healthcare, where your work directly impacts products that serve millions of users and influence the future of healthcare delivery.
Most rounds test how you handle an under-specified problem more than what you recall. A wrong first approach you correct out loud is recoverable; a constraint you assumed silently and were never given is the expensive mistake.
Focus areas
- Bound every outbound call with a timeout
- Trace a symptom to a mechanism under load
- Choose indexes from the query's access path
Overview
In this role, you'll engage with complex datasets, collaborate with cross-functional teams, and contribute to projects that transform raw data into actionable insights. The challenges you face will be multifaceted, ranging from building recommendation systems to optimizing predictive models for clinical applications. Your contributions will not only enhance existing services but also enable the development of new, data-driven solutions that align with Providence's mission to provide better patient care.
This position is not just a technical role; it is a strategic one that requires creativity, critical thinking, and a passion for leveraging machine learning to solve real-world problems. As you navigate through this role, you will find numerous opportunities to grow and innovate within a supportive environment that values your expertise.
The Providence interview process
Providence candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.
Application Review
The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.
What to demonstrate
- Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
- Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
- Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year
How to prepare
- Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
- Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
- Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter
Technical Assessments
The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.
What to demonstrate
- In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
- In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
- Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes
How to prepare
- Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
- Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
- For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first
Final Discussions
Where the day includes a partner from product, design or data, that conversation is weighted like the technical ones and prepared for least. They are deciding one thing: whether having you in the room makes their decisions cheaper. That means options with costs attached, not implementation detail and not "it depends". An estimate someone can plan against — a range, the assumption that would push it to the high end, and what you would drop to hit the low one — is worth more than a confident single number, which everyone present already knows is wrong.
What to demonstrate
- Whether an estimate comes as a range with the assumption most likely to break it, and states what a specific scope cut would actually buy
- Whether a technical constraint is handed over as a choice with consequences on their side, rather than as a verdict they have no standing to argue with
- Whether you establish what decision is on the table before proposing anything
- Whether risk is raised while it can still change the plan, with the trigger that would confirm it, instead of reported afterwards as a slip
How to prepare
- Take a project that shipped late and write the two-sentence warning you could have given three weeks earlier, naming what you would have needed decided at that point
- Rehearse one estimate out loud until it arrives in three parts: the range, the single assumption that would blow it, and the smallest thing you would cut to protect the date
- Rewrite an objection you have actually made — the "we can't do that" version — as two options with their costs, so the choice ends up with the person who owns it
Where candidates lose points
Running a schema change as though the lock lasts as long as the statement
In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.
Letting a slow dependency consume unbounded concurrency
The failure that takes a service down is usually not an error but a delay. A dependency answering in thirty seconds instead of fifty milliseconds holds each request's worker or connection six hundred times longer, and since required concurrency is arrival rate times latency, a fleet sized for sixty in-flight requests now needs thirty-six thousand to sustain the same rate - so it queues, and requests whose clients have already abandoned them still occupy resources. Retries make it precisely worse: a policy of three attempts triples the load on a dependency at the exact moment it is least able to serve, which is how one slow dependency becomes an outage of everything sharing that pool. Containment is four specific things - a timeout on every outbound call shorter than the caller's remaining budget, a bounded pool per dependency so one cannot starve the others, backoff with full jitter rather than a fixed delay so retries do not resynchronise, and a circuit that stops sending once the failure rate makes an attempt pointless.
Assuming the input fits in memory
Ask how large the input is in bytes before committing to an in-memory algorithm; beyond that point the options are a single streaming pass, an external sort with bounded buffers, or a sketch that trades exactness for constant memory. An algorithm that assumes random access to the whole input is a different algorithm from one that sees each element once.
Trusting input because it came from your own front end
Anything crossing a trust boundary is hostile: parameterise queries instead of building SQL by concatenation, validate against an allow-list rather than a deny-list, and bound the size of anything you allocate from a request. Raising this unprompted in an API or design question is a cheap and unusually strong signal.
Questions & practice
What considerations would you take into account when deploying a machine learning model in production?
Medium · machine learning fundamentals
Approach
- Pick the metric from the cost of each error type, not from habit.
- Name the simplest model that could work and what would make you move past it.
- Say how you would validate it, and where leakage could enter the split.
Follow-up
- How would you know the model is overfitting?
- What changes if the classes are heavily imbalanced?
Implement a decision tree classifier.
Medium · machine learning fundamentals
Approach
- Pick the metric from the cost of each error type, not from habit.
- State the learning problem: the label, the unit of prediction and how the model is used.
- Name the simplest model that could work and what would make you move past it.
Follow-up
- How would you know the model is overfitting?
- What changes if the classes are heavily imbalanced?
Can you explain how gradient descent works?
Medium · machine learning fundamentals
Approach
- State the learning problem: the label, the unit of prediction and how the model is used.
- Say how you would validate it, and where leakage could enter the split.
- Pick the metric from the cost of each error type, not from habit.
Follow-up
- What changes if the classes are heavily imbalanced?
- Where could label leakage enter this setup?
What are some common metrics for evaluating classification models?
Medium · machine learning fundamentals
Approach
- Say how you would validate it, and where leakage could enter the split.
- State the learning problem: the label, the unit of prediction and how the model is used.
- Name the simplest model that could work and what would make you move past it.
Follow-up
- Where could label leakage enter this setup?
- What changes if the classes are heavily imbalanced?
Given a list of integers, find the two numbers that add up to a specific target.
Medium · coding and algorithms
Approach
- Walk one small example through your approach before writing the whole thing.
- Restate the input: its shape, its size, and what is guaranteed about it.
- Choose the data structure from the access pattern, not from familiarity.
Follow-up
- What is the worst case, and how likely is it on real data?
- How does this change if the input no longer fits in memory?
Canonicalise a request body into a stable idempotency fingerprint
Medium · parsing · canonicalisation · hashing
idempotency_key.request_fingerprint is a SHA-256 over the method, path and canonicalised body, and a retry whose fingerprint differs must be rejected with 422 rather than served the stored response. Write the canonicaliser. Bodies are JSON up to 256 KB nested at most 32 levels; clients vary key order, whitespace and unicode escaping, and some send 64-bit ids as JSON numbers. Produce a deterministic byte string such that semantically identical bodies match and any semantic difference does not. State your complexity and name two normalisations you refuse to perform.
Approach
- Parse once into a tree, then re-serialise under fixed rules: object keys sorted, array order preserved, one escaping convention, no insignificant whitespace. Parsing is O(n) and sorting keys is O(k log k) per object, so O(n log n) overall with O(depth) stack, and the 32-level cap is enforced during parsing because hostile nesting is how a canonicaliser becomes a stack overflow.
- Sort keys by their UTF-8 bytes and say why the obvious implementation is wrong in some runtimes: a default string comparison that orders by UTF-16 code units places surrogate pairs, meaning code points from U+10000 up, below U+E000 to U+FFFF, which is not UTF-8 byte order, so two services written in different languages disagree on the same document.
- Do not re-encode numbers through a double. IEEE-754 binary64 represents integers exactly only up to 2^53, so normalising a 19-digit id through a float changes it, and 1 against 1.0 cannot be reconciled without deciding whether they are the same value. Preserve the literal token, and require ids as strings at the API boundary if you want them comparable.
- Reject duplicate keys rather than picking one. JSON permits them and parsers disagree, most keeping the last, so any choice you make ties the fingerprint to a parser detail that the code handling the request does not necessarily share.
- Frame the hash preimage so concatenation cannot collide: delimit or length-prefix the method, path and body, otherwise one request's fields can be rearranged into another request with the same byte stream and the same fingerprint.
- Name the refusals and their consequence: no case folding, no dropping of null-valued keys, no Unicode normalisation. Each makes two different requests fingerprint alike, and the resulting failure is the worst one this table has, since the second request is answered with the first one's stored response and its effect never happens.
Worked solution (25 min)
- Write the serialiser: recursive emit with a depth counter, objects sorted by UTF-8 key bytes, arrays in order, strings escaped by one fixed rule, numbers emitted as their original token.
- Run it over three bodies: the same object with keys reordered, the same object with \u0041 written as A, and one with a nested array reversed. The first two must produce identical bytes and the third must not.
- Take the id 9007199254740993, round-trip it through a double, show it returns as 9007199254740992, then state the rule that prevents this.
- Define the hash preimage explicitly with its delimiters, and construct a pair of (path, body) inputs that would collide without them.
Expected result: A canonicaliser that is O(n log n) in body size with an enforced depth cap, sorts keys by UTF-8 byte order, preserves array order, keeps number literals verbatim, rejects duplicate keys, and feeds a delimited preimage to SHA-256, together with a stated list of normalisations deliberately not performed and the failure each would cause.
Verify it yourself
- Key reordering and whitespace changes produce identical fingerprints while reordering an array does not.
- A body containing an integer above 2^53 fingerprints identically after a parse and re-serialise round trip.
- Two requests differing only in a field the endpoint ignores produce different fingerprints, and you can say why that is the safe direction to err.
Follow-up
- A client sends the same logical request with an extra field your API ignores. Same key, different fingerprint, so you return 422. Is that the right answer?
- Where does the fingerprint get computed relative to request decompression and the body-size limit?
- The endpoint takes 1,000 requests per second with 256 KB bodies. What does hashing cost, and does it belong at the edge or in the core service?
Stop tag and share joins from fanning out a page
Medium · join fan-out · semi-joins · many-to-many
resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.
Approach
- Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
- Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
- Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
- Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.
- Attach the tags for display after the page has been cut — LEFT JOIN LATERAL (SELECT array_agg(rt.tag_id) FROM resource_tag rt WHERE rt.resource_id = p.resource_id) ON TRUE over the 50 returned rows. Aggregate over the page, never over the tenant.
- Index both directions and say which query each serves: PK (resource_id, tag_id) serves the lateral lookup, (tag_id, resource_id) serves the EXISTS probe by tag, and resource_share needs (shared_with_user_id, resource_id) for the same reason. An index covering one direction only leaves the other as a scan.
Worked solution (30 min)
- Build a tenant where each resource carries 0-5 tags from a 20-tag vocabulary and is shared with 0-4 distinct users, then bind $2 to three tags and $3 to a user holding shares on about half the resources. Run the joined query and compare its row count to the distinct resource count on page one.
- Run COUNT(*) on the joined shape and on the EXISTS shape and compare both to a ground truth computed from distinct ids; then give $3 a second permission row on 10% of resources and record which of the two counts moves.
- EXPLAIN both page queries and compare rows-read plus the presence of a Sort or HashAggregate node above the join.
- Add (tag_id, resource_id), re-run the EXISTS probe, and record the plan change on the inner side.
Expected result: The EXISTS page returns exactly 50 distinct resources and reads about 50 driving rows with no aggregate above the join. The EXISTS count matches the ground truth. The joined COUNT(*) is larger by roughly the average number of matching tags per qualifying resource — the tags it holds from $2, not its total tag count — times its share rows for $3, which is 1 until the duplicate permission rows exist and rises with them afterwards. The EXISTS count is unchanged by those duplicates.
Verify it yourself
- The page's resource_id list has no duplicates and is 50 long whenever at least 50 rows match.
- count(*) from the rewrite equals count(DISTINCT r.resource_id) over the joined shape.
- EXPLAIN shows a semi-join node on the inner side rather than a full join followed by a unique or aggregate step.
Follow-up
- The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
- A resource can be shared with the same user twice under different permissions. Does your count change, and should it?
- Where does the correct total come from when the tenant holds 4M resources and the header must not cost 200 ms?
Replace offset paging on the resource feed with keyset
Easy · keyset pagination · composite index · sort order
resource holds resource_id, tenant_id, owner_user_id, title, body_ref, version, status ('draft','active','archived','deleted'), created_at, updated_at, deleted_at, with an index on (tenant_id, status, updated_at DESC, resource_id DESC). The listing endpoint returns active resources for one tenant, newest update first, 50 per page, today with LIMIT 50 OFFSET n. Tenants reach page 400 and rows are created while they read. Write the keyset query, define what the cursor carries and how it is encoded, and say which part of the index each predicate uses. Assume PostgreSQL 16.
Approach
- Name the two failures separately. OFFSET 20000 makes the server produce and discard 20,000 rows, so page cost grows with depth rather than with page size. Independently, any write that changes how many rows sort above the offset moves the window between two fetches, and the direction decides which anomaly you get: an insert lands at the head of updated_at DESC and pushes already-returned rows down past the boundary, so they are returned a second time; a delete above the offset, or a row whose updated_at is bumped above the cursor, pulls rows up and one is never returned at all. Nothing in the response reveals either.
- Write the seek: WHERE tenant_id = $1 AND status = 'active' AND (updated_at, resource_id) < ($2, $3) ORDER BY updated_at DESC, resource_id DESC LIMIT 50. The row-value comparison is one index range rather than a disjunction, and both columns are NOT NULL, which is what makes that comparison well defined.
- Map each predicate onto the index: tenant_id and status are equality on the leading columns, (updated_at, resource_id) is the range, and the ORDER BY matches the index order so no Sort node appears and the scan stops after 50 rows. The DESC in the definition only matters for mixed directions — a plain ascending btree on the same columns is read backwards for this query.
- Put both sort columns in the cursor and nothing the client can tamper with into another tenant: base64 of (updated_at, resource_id), validated server-side, with tenant_id taken from the principal.
- State the residual honestly. Keyset is stable against concurrent inserts and deletes, but not against a row whose updated_at changes mid-scroll — that row moves in the ordering and can be seen twice. If the feed must be a snapshot, order by an immutable key or bound the page set with updated_at <= the cursor's start value.
- Keep a total out of the page path. A tenant-wide COUNT(*) is the scan keyset just removed; fetch LIMIT 51 and return has_more instead.
Follow-up
- The client asks for 'jump to page 400'. What do you offer instead, and what does the honest version cost?
- Sort order becomes user-selectable across four columns. How many indexes is that, and which would you refuse to add?
- What does the cursor do when the row it points at has since been deleted?
Design a recommendation system for a healthcare application.
Medium · ml system design and production
Approach
- Say where features come from at serving time and how they match training.
- Separate the offline training path from the online serving path.
- Name what you would monitor after launch and what triggers a retrain.
Follow-up
- What happens when a feature is missing at serving time?
- How would you detect drift before the metric drops?
Explain the difference between supervised and unsupervised learning.
Medium
Approach
- State your assumptions explicitly before working the problem.
- Say what you would check first and why it is the highest-information step.
- Clarify what is being asked and what a complete answer contains.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
Design the async export contract a client can resume safely
Hard · async operations · idempotency · leases
A tenant asks for a CSV of every resource. The work runs for minutes on the worker fleet through a job_run row carrying a lease, an attempt count and a dedupe_key, far past the edge's 400 ms budget. Callers are a browser that polls and a script that walks away and checks later. Specify what the submit call returns, the operation resource and its states, how a duplicate submit is handled, how a client learns about completion, what cancellation means given that a lease can expire mid-run, and how the result is fetched and when it expires.
Approach
- Split the API in two. Submit returns 202 with an operation id and a location to poll, and never blocks on the work. The operation is a real resource with its own lifecycle - queued, running, succeeded, failed, cancelled - plus attempt, a monotonic progress figure, and a terminal error drawn from the same code taxonomy the synchronous endpoints use, so a client needs one error vocabulary rather than two.
- Deduplicate at submit using job_run.dedupe_key, unique over (job_type, dedupe_key) while status is 'queued' or 'running': a repeat submit of the same logical export returns 200 with the existing operation instead of 202 with a new one, and the partial index deliberately permits a legitimate re-run once the first has finished. Pair it with the request's idempotency key so an HTTP-level retry of the submit is exact rather than merely similar.
- Tell the poller how to poll: Retry-After on the polling response, a minimum interval enforced at the edge, and a documented maximum lifetime after which an operation is reaped. Polling is the contract of record; the webhook is the fast path, and both must lead to the same terminal state, so a client that receives the completion event and then polls anyway sees no contradiction.
- Be exact about cancellation. A cancel request records intent; it cannot stop work already executing. The handler reads the flag at checkpoints, and because a lease expires on a clock that cannot distinguish a dead worker from a slow one, a second copy may start after the cancel was recorded - so the handler re-reads the flag immediately after claiming the lease. 'cancelled' becomes terminal only when no lease is outstanding; reporting it earlier shows a client a stopped job while a worker is still writing output.
- Make the handler safe to run twice, because the lease guarantees that it will be. Write output to a deterministic object key derived from the operation id so a second copy overwrites its own work instead of appending a second file, and record completion with a conditional update that only the copy holding the current lease can win.
- Treat result fetch as a separate authorised read: a short-lived signed URL, the tenant checked when it is issued rather than only when the file was produced, and a documented retention after which the operation remains terminal but the bytes are gone - a state the client must be able to tell apart from a failure.
Worked solution (40 min)
- Write the submit request and its two possible responses, 202 for new and 200 for a duplicate, and the dedupe_key construction.
- Draw the operation state machine, marking which transitions a client may observe and which are terminal.
- Write the cancellation sequence across a lease expiry, showing where the second copy reads the flag.
- Define the output key, the completion update's predicate, and why both are needed for a double run.
- Specify result fetch: URL lifetime, authorisation point, retention, and the distinct response once the bytes are gone.
Expected result: A 202-plus-operation contract with deduplicated submits, a documented poll interval alongside an optional completion webhook, cancellation defined as recorded intent with a terminal state gated on lease ownership, an idempotent handler writing to a deterministic key, and a separately authorised expiring result.
Verify it yourself
- Submitting the same export twice while the first is running produces one job_run row and one operation id.
- Killing a worker mid-export and letting the lease expire yields one output file, not two, and one terminal transition.
- A client that receives the completion webhook and then polls sees the same terminal state and the same result location.
Follow-up
- An operation has said 'running' for 40 minutes and the worker is gone. What does the client see, and which columns in job_run decide that?
- Two tenants each submit 50 exports at once. What in this contract stops one of them delaying the other?
- The customer wants the export emailed instead. What changes, and what becomes harder to make exactly-once?
Listing latency scales with page size, not with filters
Easy · n+1 · latency · query batching
The tenant listing endpoint reads resource filtered by tenant_id and status, ordered by updated_at DESC, and returns each row plus the owner's display name from app_user and the actor of that resource's latest resource_revision. p99 is 55 ms at 10 rows per page and 1.4 s at 200. Database telemetry shows 401 statements per request, each under 1 ms, and nothing in the slow-query log. Diagnose the cause and give the fix, stating the statement count per request and the p99 you expect afterwards.
Approach
- Read the counters before forming a theory. 401 statements for 200 rows is one driver query plus two per row, and sub-millisecond execution with an empty slow-query log rules out a bad plan. The time is round trips, which is why it is invisible in every per-query metric and scales with rows returned rather than with filter selectivity.
- Name the two per-row statements from their normalised text: a single-row app_user lookup by user_id, and a resource_revision lookup by resource_id ordered by version DESC LIMIT 1. Confirm by dropping those two response fields and watching the statement count fall to one. That locates the calls in the serialisation layer, not the repository.
- Check that the arithmetic accounts for the whole gap. Measure one round trip to the replica in isolation; 400 trips at roughly 3 ms of network plus 0.2 ms of execution is about 1.3 s on top of a 55 ms baseline, which matches. If the multiplication had fallen short, the N+1 would only be part of the story and you would keep looking.
- Batch both lookups. Collect owner_user_ids and resource_ids from the driver query, then issue WHERE tenant_id = $1 AND user_id = ANY($2) for the users, and PostgreSQL's SELECT DISTINCT ON (resource_id) ... WHERE resource_id = ANY($2) ORDER BY resource_id, version DESC for the latest revision, which the UNIQUE (resource_id, version) index serves directly. On an engine without DISTINCT ON, use a lateral join or a row_number window. Three statements per request at any page size.
- Keep the tenant predicate in the batched query. The per-row version was implicitly scoped because its ids came from tenant-scoped rows; a batched user_id = ANY(...) with no tenant_id is an unscoped read that behaves correctly only as long as the id list is trustworthy.
- Re-measure at 10, 50 and 200 rows and confirm the statement count is constant. Latency should now track bytes returned.
Follow-up
- The page size is capped at 200 today. What breaks first if it is raised to 2,000, and is it still this bug?
- How do you stop the next N+1 from reaching production, given that no individual query is slow and the endpoint's tests pass?
- The latest-revision actor is only used to render an avatar. Make the case for denormalising it onto resource, and name the write anomaly that introduces.
Suggested practice checklist
For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.
Day 1 — Numbers before diagrams
- Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
- Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
- Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.
Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.
Day 2 — One system, from requirements to schema
- Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
- Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
- Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.
Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.
Day 3 — The consistency you are actually buying
- Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
- Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
- Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.
Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.
Day 4 — Failure is the design
- For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
- Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
- Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
- Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.
Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.
Day 5 — Scaling the hot path
- Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
- Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
- Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
- Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.
Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.
Day 6 — Keep the coding hand in, at the bar that applies to you
- Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
- Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
- Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.
Deliverable: One problem hardened to review standard, with its test list and one written review comment.
Day 7 — Defend it while being interrupted
- Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
- Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
- Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.
Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.
The bug you decided not to fix
Engineers over-index on what they repaired. A stronger answer covers something you knowingly left broken: the alert you tuned down, the data inconsistency you documented instead of chasing, the cleanup you deferred past two quarters. Give the reasoning and the condition that would have reopened it, so it reads as a decision and not as neglect.
Describe a challenging project you led. How did you handle the difficulties?
Medium · behavioural and collaboration
Approach
- Close with what you would do differently, concretely.
- Pick a story where you made the decision, not one where you watched it.
- Give the blast radius: what could have broken, and what you measured.
Follow-up
- What would you do differently if you ran that again?
- What did you decide not to do, and why?
How do you prioritize tasks when managing multiple projects?
Medium · behavioural and collaboration
Approach
- Close with what you would do differently, concretely.
- Name the disagreement and how you resolved it with evidence.
- Pick a story where you made the decision, not one where you watched it.
Follow-up
- How did you know your change caused the improvement?
- What did you decide not to do, and why?
Argue against a design, lose, and commit anyway
Medium · disagreement · service boundaries · decision records
Describe a design you argued against and lost. State the failure you predicted as a named mechanism, not a feeling about complexity: two services that would need one transaction, a projection with no rebuild path, a write path with no idempotency key. Say what evidence you brought, what the decision maker weighed instead, and what you did after the decision was made: what you instrumented, what you wrote down, and whether the prediction came true. Five minutes.
Approach
- State the prediction in falsifiable form up front: the mechanism, the condition that triggers it, and the observable outcome. A prediction that cannot be checked also cannot be credited to you later.
- Show the evidence you had at the time and label each piece honestly as measured, analogous, or intuition. Keeping the intuition is fine; disguising it as data is the thing that erodes your standing in the next argument.
- Represent the opposing case at full strength, including the constraint you did not control: a fixed date, a team boundary, or the fact that the decision was cheap to reverse and yours was not.
- Make disagree-and-commit concrete. Name the artefact you left behind so the prediction could be settled without you: the alert and its threshold, the counter on the dashboard, the decision note that recorded the trade-off and the condition that would revisit it.
- Report the outcome without editing it. If the design held and your predicted mechanism never fired, say so and say what you had mis-weighted, which is more persuasive than a vindication story.
Follow-up
- What threshold on that alert would have proved you right, and did anyone ever look at it?
- If the same proposal arrived tomorrow with the same deadline, would you argue it the same way?
- How did you behave toward the design once it shipped and started failing in a different way than you predicted?