Interview guides · Covar

Covar Machine Learning Engineer Interview Questions & Guide 2026

Covar Machine Learning Engineer interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.

The 60-second brief

At Covar, a Machine Learning Engineer plays a pivotal role in bridging the gap between cutting-edge artificial intelligence research and robust, production-grade software systems. Operating at the intersection of data science, software engineering, and systems architecture, engineers in this role are responsible for designing, training, and deploying models that solve complex, real-world problems. Whether working out of the technology hubs in Durham, North Carolina, or McLean, Virginia, the engineering team focuses on delivering high-impact solutions that scale efficiently and maintain extreme reliability.

The loop does not sample the job evenly, and arguing about that in the room costs you. Daily work is mostly incremental change inside code someone else wrote, while the loop samples narrow slices of it; prepare for the slices and save the realism argument for your questions at the end.

Focus areas

Detect concurrent edits instead of losing writes
Paginate large result sets with keyset cursors
Trace a symptom to a mechanism under load

Overview

The impact of a Machine Learning Engineer at Covar is felt directly across the organization’s core product offerings and client engagements. You will work on translating sophisticated mathematical concepts into clean, maintainable Python code, ensuring that models can operate under tight computational constraints and handle diverse, noisy datasets. The work is highly collaborative, requiring close interaction with cross-functional teams to align machine learning capabilities with broader business objectives and client needs.

This role is ideal for engineers who thrive in dynamic, technically demanding environments and enjoy solving open-ended problems. Because Covar tackles highly specialized engineering challenges, successful candidates must demonstrate not only deep theoretical knowledge of machine learning but also the practical software engineering skills required to build and optimize end-to-end pipelines.

The Covar interview process

Covar candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Video Screening

Because the format is not fixed, the first job in the room is classification. Listen to the opening question and decide what it is: a probe into work you have already described, a fresh problem to solve now, or a conversation about how you operate. Each wants a different register, and the common failure is forcing a rehearsed structure onto a question that did not ask for it. Running a full design ritual on a ten-minute debugging question reads as not listening. When you cannot tell which it is, ask how long they want to spend and answer at that depth.

What to demonstrate

Whether the shape of your answer matches the question, so a yes-or-no gets answered before it is justified and an open prompt gets a direction before a detour
Whether you check how much depth is wanted instead of deciding for them, and whether you stop when the answer is complete rather than continuing until someone interrupts
Whether you can be redirected in the middle of an answer without restarting it from the beginning
Whether a question outside your experience gets an honest boundary followed by reasoning from what you do know, instead of a confident answer with nothing behind it

How to prepare

Rehearse one project at three lengths, roughly thirty seconds, three minutes, and a full walkthrough at the depth of a design review, and practise switching between them when someone interrupts mid-telling
Have someone ask you five questions of deliberately mixed type in one sitting without telling you the types, and score only whether you identified each one correctly before you started answering
Draft the sentence you will use to check depth, along the lines of asking whether the short version is useful here or they want the detail, and use it in a real conversation this week so the day of the round is not its first outing

Coding Assessment

What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.

What to demonstrate

Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly

How to prepare

Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live

ML Breadth Interview

What to demonstrate

Whether the shape of your answer matches the question, so a yes-or-no gets answered before it is justified and an open prompt gets a direction before a detour
Whether you check how much depth is wanted instead of deciding for them, and whether you stop when the answer is complete rather than continuing until someone interrupts
Whether you can be redirected in the middle of an answer without restarting it from the beginning
Whether a question outside your experience gets an honest boundary followed by reasoning from what you do know, instead of a confident answer with nothing behind it

How to prepare

Rehearse one project at three lengths, roughly thirty seconds, three minutes, and a full walkthrough at the depth of a design review, and practise switching between them when someone interrupts mid-telling
Have someone ask you five questions of deliberately mixed type in one sitting without telling you the types, and score only whether you identified each one correctly before you started answering
Draft the sentence you will use to check depth, along the lines of asking whether the short version is useful here or they want the detail, and use it in a real conversation this week so the day of the round is not its first outing

Full-Day Interview Loop

Where the day includes a partner from product, design or data, that conversation is weighted like the technical ones and prepared for least. They are deciding one thing: whether having you in the room makes their decisions cheaper. That means options with costs attached, not implementation detail and not "it depends". An estimate someone can plan against — a range, the assumption that would push it to the high end, and what you would drop to hit the low one — is worth more than a confident single number, which everyone present already knows is wrong.

What to demonstrate

Whether an estimate comes as a range with the assumption most likely to break it, and states what a specific scope cut would actually buy
Whether a technical constraint is handed over as a choice with consequences on their side, rather than as a verdict they have no standing to argue with
Whether you establish what decision is on the table before proposing anything
Whether risk is raised while it can still change the plan, with the trigger that would confirm it, instead of reported afterwards as a slip

How to prepare

Take a project that shipped late and write the two-sentence warning you could have given three weeks earlier, naming what you would have needed decided at that point
Rehearse one estimate out loud until it arrives in three parts: the range, the single assumption that would blow it, and the smallest thing you would cut to protect the date
Rewrite an objection you have actually made — the "we can't do that" version — as two options with their costs, so the choice ends up with the person who owns it

Where candidates lose points

Running a schema change as though the lock lasts as long as the statement

In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.

Paginating with LIMIT/OFFSET over a set that changes while the client is reading it

OFFSET n makes the database produce and discard n rows before returning anything, so the cost of a page grows with its depth rather than with its size and page 500 costs five hundred pages of work. The correctness problem is worse than the cost: if a row is inserted or reordered between two page fetches, rows shift across the offset boundary and are either skipped entirely or returned twice, and neither outcome leaves any trace in the response for the client to detect. Keyset pagination - WHERE (sort_key, id) < ($last_sort_key, $last_id) ORDER BY sort_key DESC, id DESC LIMIT n, backed by an index in exactly that order - reads only the rows it returns and is stable against concurrent inserts. It requires the tie-break column: a timestamp is not unique, and duplicate sort keys straddling a page boundary reintroduce the skip it was adopted to remove.

Listing technologies instead of trade-offs

Name the property the design needs first, such as ordered range scans, multi-entity transactions, cheap appends, or a predictable p99, then pick something that provides it and say what it gives up in exchange. Almost any component is defensible once you state the requirement it satisfies and the one it sacrifices.

Trusting input because it came from your own front end

Anything crossing a trust boundary is hostile: parameterise queries instead of building SQL by concatenation, validate against an allow-list rather than a deny-list, and bound the size of anything you allocate from a request. Raising this unprompted in an API or design question is a cheap and unusually strong signal.

Questions & practice

Walk through how you would design a recommendation system from scratch, including data ingestion, feature engineering, and model evaluation.

Medium · machine learning fundamentals

Approach

Name the simplest model that could work and what would make you move past it.
Say how you would validate it, and where leakage could enter the split.
State the learning problem: the label, the unit of prediction and how the model is used.

Follow-up

What changes if the classes are heavily imbalanced?
Where could label leakage enter this setup?

Explain the process of hyperparameter tuning and how you prevent overfitting in deep learning models.

Medium · machine learning fundamentals

Approach

Pick the metric from the cost of each error type, not from habit.
State the learning problem: the label, the unit of prediction and how the model is used.
Name the simplest model that could work and what would make you move past it.

Follow-up

Where could label leakage enter this setup?
What changes if the classes are heavily imbalanced?

Describe the difference between generative and discriminative models, providing examples of when to use each.

Medium · machine learning fundamentals

Approach

Name the simplest model that could work and what would make you move past it.
Say how you would validate it, and where leakage could enter the split.
State the learning problem: the label, the unit of prediction and how the model is used.

Follow-up

How would you know the model is overfitting?
What changes if the classes are heavily imbalanced?

Describe your process for debugging a model that performs well on training data but poorly in a live production environment.

Medium · machine learning fundamentals

Approach

Say how you would validate it, and where leakage could enter the split.
State the learning problem: the label, the unit of prediction and how the model is used.
Name the simplest model that could work and what would make you move past it.

Follow-up

Where could label leakage enter this setup?
How would you know the model is overfitting?

Implement a basic custom data structure in Python, such as a queue or a stack, using native lists.

Medium · coding and algorithms

Approach

Choose the data structure from the access pattern, not from familiarity.
Walk one small example through your approach before writing the whole thing.
Restate the input: its shape, its size, and what is guaranteed about it.

Follow-up

What is the worst case, and how likely is it on real data?
Which test case would catch an off-by-one here?

Given a list of integers, write an efficient algorithm to group duplicates and return their frequencies.

Medium · coding and algorithms

Approach

Restate the input: its shape, its size, and what is guaranteed about it.
Choose the data structure from the access pattern, not from familiarity.
State the target complexity and say which constraint rules the naive version out.

Follow-up

Which test case would catch an off-by-one here?
How does this change if the input no longer fits in memory?

Find overlapping job attempts and peak concurrency from lease records

Medium · sweep line · intervals · leases

A day of job_run history yields about 50,000,000 attempt records: (job_run_id, job_type, attempt, started_at, finished_at which is NULL when the worker died, lease_expires_at). Leases expire on a clock, so a job that outran its lease ran twice. Produce (a) every job_run_id whose attempts overlapped in wall-clock time and (b) the peak number of simultaneously running attempts per job_type with the minute it occurred. Target O(n log n). State how you treat a NULL finished_at and what clock skew does to your answer.

Approach

Define the interval before sorting anything: an attempt occupies [started_at, COALESCE(finished_at, lease_expires_at)). finished_at is observed and lease_expires_at is only a promise, so every attempt without a finish contributes an estimate and the whole result is a lower bound on overlap rather than an exact count.
For peak concurrency, sweep: emit 2n endpoints, sort by (timestamp, kind) with ends ordered before starts at equal timestamps, then walk the sequence maintaining a counter per job_type and record each type's maximum with its timestamp. O(n log n) dominated by the sort, O(n) space, or O(1) extra if the sort is external and the walk streams.
For overlap detection, do not compare attempts pairwise. A single global sort by (job_run_id, started_at) gives both the grouping and the order; within a group, keep the maximum end seen so far and report an overlap exactly when the next start is less than that running maximum, which is one linear pass after the sort.
Half-open intervals matter and are easy to get wrong: with closed intervals an attempt ending at the same millisecond another begins reads as concurrency two, and across 50,000,000 records that artefact swamps the real signal.
State the clock caveat: started_at and finished_at are written by different workers, so under skew of a few hundred milliseconds an apparent overlap shorter than that bound is not evidence. Filter reported overlaps by a minimum duration, or prefer timestamps written by whichever component heartbeats the lease.
Scale the sort rather than assuming it fits: the sweep emits two endpoints per attempt, so 50,000,000 records become 100,000,000 endpoints, and at roughly 24 bytes each, an 8-byte timestamp plus a 4-byte job_type plus a kind flag padded to alignment, that is about 2.4 GB of sort keys before any scratch space. Either push the ordering into the database behind an index on (job_type, started_at) or run an external merge sort in chunks; the overlap pass sorts n records rather than 2n, so it is the cheaper of the two.

Worked solution (30 min)

Write the interval derivation with the COALESCE and state in one line which of the two end sources is observed and which is assumed.
Write the concurrency sweep: the endpoint tuples, the sort key including the end-before-start tie-break, and the per-job_type counter.
Hand-trace four attempts of one job, two disjoint and two overlapping by three seconds, and confirm the overlap detector fires exactly once.
Add the skew filter as a minimum overlap duration, state the value you chose, and justify it from how the timestamps are written.

Expected result: One global sort by (job_run_id, started_at) driving a running-maximum-end check for overlaps, plus a sweep over 2n endpoints with ends ordered before starts for per-job_type peak concurrency, both O(n log n), with half-open intervals, NULL finished_at falling back to lease_expires_at, and overlaps shorter than the clock-skew bound excluded.

Verify it yourself

Two attempts touching at exactly the same instant report concurrency 1, not 2.
A job with a single attempt never appears in the overlap output regardless of its lease.
The reported peak timestamp for a job_type lies inside at least one of that type's intervals.

Follow-up

A handler is not idempotent and you have found 400 overlapping jobs. Which of them actually caused damage, and what would you query to find out?
Peak concurrency for one job_type is 4 against a configured cap of 4. Is the cap working, or is the data hiding attempts that never started?
How would you compute both answers incrementally as records arrive rather than in a daily batch?

Replace offset paging on the resource feed with keyset

Easy · keyset pagination · composite index · sort order

resource holds resource_id, tenant_id, owner_user_id, title, body_ref, version, status ('draft','active','archived','deleted'), created_at, updated_at, deleted_at, with an index on (tenant_id, status, updated_at DESC, resource_id DESC). The listing endpoint returns active resources for one tenant, newest update first, 50 per page, today with LIMIT 50 OFFSET n. Tenants reach page 400 and rows are created while they read. Write the keyset query, define what the cursor carries and how it is encoded, and say which part of the index each predicate uses. Assume PostgreSQL 16.

Approach

Name the two failures separately. OFFSET 20000 makes the server produce and discard 20,000 rows, so page cost grows with depth rather than with page size. Independently, any write that changes how many rows sort above the offset moves the window between two fetches, and the direction decides which anomaly you get: an insert lands at the head of updated_at DESC and pushes already-returned rows down past the boundary, so they are returned a second time; a delete above the offset, or a row whose updated_at is bumped above the cursor, pulls rows up and one is never returned at all. Nothing in the response reveals either.
Write the seek: WHERE tenant_id = $1 AND status = 'active' AND (updated_at, resource_id) < ($2, $3) ORDER BY updated_at DESC, resource_id DESC LIMIT 50. The row-value comparison is one index range rather than a disjunction, and both columns are NOT NULL, which is what makes that comparison well defined.
Map each predicate onto the index: tenant_id and status are equality on the leading columns, (updated_at, resource_id) is the range, and the ORDER BY matches the index order so no Sort node appears and the scan stops after 50 rows. The DESC in the definition only matters for mixed directions — a plain ascending btree on the same columns is read backwards for this query.
Put both sort columns in the cursor and nothing the client can tamper with into another tenant: base64 of (updated_at, resource_id), validated server-side, with tenant_id taken from the principal.
State the residual honestly. Keyset is stable against concurrent inserts and deletes, but not against a row whose updated_at changes mid-scroll — that row moves in the ordering and can be seen twice. If the feed must be a snapshot, order by an immutable key or bound the page set with updated_at <= the cursor's start value.
Keep a total out of the page path. A tenant-wide COUNT(*) is the scan keyset just removed; fetch LIMIT 51 and return has_more instead.

Follow-up

The client asks for 'jump to page 400'. What do you offer instead, and what does the honest version cost?
Sort order becomes user-selectable across four columns. How many indexes is that, and which would you refuse to add?
What does the cursor do when the row it points at has since been deleted?

Keep soft-deleted accounts from blocking re-registration

Easy · soft delete · unique constraints · engine differences

app_user holds user_id, tenant_id, email CITEXT, password_hash (NULL for SSO principals), email_verified_at, auth_version, status ('invited','active','suspended','deactivated'), created_at, updated_at, deleted_at. Two live accounts for one address inside a tenant must be impossible, but an address freed by a soft delete must be reusable, and the same tenant may delete and re-register it repeatedly. Write the uniqueness DDL for PostgreSQL 16, then the equivalent for MySQL 8 where partial indexes do not exist, and say what each permits once three deleted rows already hold that address.

Approach

Start from what is actually unique: not (tenant_id, email), but (tenant_id, email) among live rows. PostgreSQL says that directly — CREATE UNIQUE INDEX app_user_live_email ON app_user (tenant_id, email) WHERE deleted_at IS NULL. A full constraint over the same two columns burns the address permanently the first time someone deletes an account.
Keep case-insensitivity in the type or the index, never in the application: CITEXT as given, or UNIQUE (tenant_id, lower(email)) as an expression index where the extension is unavailable. A case-sensitive unique column is exactly how two accounts for one human appear.
For MySQL 8 the predicate has to move inside the key: add a discriminator column that is a constant 0 while the row is live and is set to user_id on delete, with UNIQUE (tenant_id, email, deleted_marker). Live rows share the constant and still collide; deleted rows differ from each other and stop colliding.
State the NULL variant and its dependency: leaving the marker NULL for deleted rows also works, because a unique index treats NULLs as distinct — true in MySQL, and true in PostgreSQL only under the default NULLS DISTINCT, which PostgreSQL 15 lets you reverse. Check the polarity against the three existing deleted rows: constant-on-live is what preserves the collision you want, and reversing it silently admits duplicate live accounts.
Say what a soft delete must do besides setting deleted_at: increment auth_version so existing tokens stop validating, leave resource.owner_user_id and resource_revision.actor_user_id intact, and accept that the address is retained — erasure is a different requirement answered by scrubbing the column, not by a DELETE that would break those references.

Worked solution (20 min)

Create the PostgreSQL partial unique index, insert a live row, soft delete it, and insert the same address again.
Repeat the delete-and-reinsert cycle three times and confirm three deleted rows coexist with exactly one live row.
Write the MySQL form with the discriminator, then deliberately reverse the polarity so live rows carry NULL, and show two live duplicates commit.
Attempt a second live insert on both engines and map the resulting 23505 / ER_DUP_ENTRY to the 409 the handler should return.

Expected result: PostgreSQL rejects only the second live row and admits unlimited deleted ones. MySQL behaves identically with UNIQUE (tenant_id, email, deleted_marker) where live = 0; with the polarity reversed, two live rows commit and nothing afterwards can tell them apart.

Verify it yourself

Group live rows by (tenant_id, lower(email)) after a randomised insert/delete run — no group exceeds one.
Inserting 'User@x.test' after 'user@x.test' is rejected; if it is accepted, the case handling is in application code rather than in the index.
EXPLAIN a login lookup and confirm the partial index is chosen — it is only usable by queries that carry the deleted_at IS NULL predicate, so a query without it falls back to a scan.

Follow-up

A deleted account re-registers with the same address the next day. Do the old resource rows follow the new user_id, and how does the API keep the two principals apart?
How do you honour an erasure request while resource_revision.actor_user_id still references this table?
What changes if a user may hold membership in two tenants?

How would you approach deploying a large language model under strict latency constraints?

Medium · ml system design and production

Approach

Name what you would monitor after launch and what triggers a retrain.
Say where features come from at serving time and how they match training.
Separate the offline training path from the online serving path.

Follow-up

How would you roll the new model out safely?
How would you detect drift before the metric drops?

Specify the outbound webhook contract a customer endpoint can verify

Medium · webhooks · hmac · at-least-once · ordering

Egress delivery sends resource.created, resource.updated and resource.archived to roughly 40k customer endpoints at about 1.5k deliveries/second, with a per-destination concurrency cap of 4 and a 10 second connect-plus-read timeout, at-least-once from the relay. The receiver is code you do not control, often behind a proxy that may re-encode JSON. Specify the payload envelope, how the receiver verifies authenticity and rejects replays, what ordering it may assume, how it deduplicates, which status codes it should return and what each means to us, and the retry, circuit and replay behaviour.

Approach

Send the fact, not a pointer: event_id, event_type, aggregate_type, aggregate_id, aggregate_version, occurred_at and the payload copied from outbox_event. A receiver told only that resource 42 changed has to call back, and the call back returns a newer state than the event describes - that is precisely how a consumer applies changes out of order while believing it is following them.
Sign the exact bytes on the wire: HMAC-SHA256 over the timestamp, a separator, and the raw body, in a header, compared in constant time before the body is parsed. Instruct receivers to verify the raw bytes, never a re-serialised copy, since a proxy that reorders keys or changes whitespace invalidates a signature computed over re-encoded JSON. Reject a timestamp outside a tolerance - five minutes is typical - and keep a short seen-set of event_ids, because a capture replayed inside the tolerance still verifies.
Support key rotation by sending several signature values in one header during the overlap window and publishing that any one that verifies is sufficient. Without that rule a key rotation is a synchronised outage across every destination at once.
State the guarantee as it actually is: at-least-once and unordered, including within a single aggregate_id. The outbox is ordered per aggregate at the source, but the relay holds up to 4 requests in flight per destination and backs off each retry independently, so a version 6 delivery that needs one retry lands after a version 7 that succeeded on its first attempt. Publishing per-aggregate ordering we do not enforce is worse than publishing none, because a receiver that trusts it overwrites new state with old. Receivers deduplicate on event_id and apply conditionally - UPDATE ... WHERE aggregate_version < :incoming, or the same comparison under a row lock on the aggregate - which discards stale redeliveries and stays correct when two deliveries for one aggregate are processed at the same time. A duplicate and an out-of-order delivery then have the same outcome: no-op.
Define the response contract in the receiver's terms. 2xx means durably accepted, and accepting means writing it down - a receiver that does its processing inline inside our 10 second timeout will be retried while it is still working. 4xx is permanent and we stop. 429, 5xx and timeouts are retried with capped exponential backoff and full jitter within the per-destination concurrency cap, and a circuit opens after a run of consecutive failures and half-opens on a schedule, so a permanently dead endpoint costs one probe per interval instead of a growing queue.
Give them a recovery path that does not involve us holding a queue for everyone: a visible parked state, and a replay endpoint keyed by event_id or time range, so a receiver that was down for two hours asks for the gap on its own schedule.

Worked solution (30 min)

Write the envelope fields and mark which are needed for deduplication and which for discarding a stale delivery.
Write the signature string construction and the receiver's verification steps in order, including the constant-time compare and the tolerance check.
Write the receiver's idempotent apply: dedupe on event_id, then the version-conditional write that discards stale and concurrently delivered redeliveries.
Write the status-to-action table from our side: which codes retry, which stop, and what opens the circuit.
Describe the parked state and the replay call, including what the customer must supply.

Expected result: A self-contained signed envelope, verification over raw bytes with a replay window and a seen-set, an at-least-once guarantee that states delivery is unordered even within one aggregate, a version-conditional apply, a status table that distinguishes permanent from transient, and a customer-driven replay path.

Verify it yourself

Delivering the same event twice leaves the receiver's state identical to delivering it once.
Delivering version 7 and then version 6 for one aggregate leaves version 7 applied.
A destination that always times out costs one probe per circuit interval rather than a retry queue.

Follow-up

A customer insists an event never arrived. What do you check, in order, and which table settles the argument?
A receiver cannot store a version and asks us to deliver their events in order. What would the relay have to give up to offer that, and what does one failing event then do to the rest of their stream?

p99 jumped on one listing filter while p50 stayed flat

Medium · query planning · composite index · percentiles · tail latency

After a release that added an owner_user_id filter to the resource listing, p99 rose from 90 ms to 1.9 s while p50 stayed at 40 ms. Traffic and row counts are unchanged. resource carries the index (tenant_id, status, updated_at DESC, resource_id DESC). The new query filters tenant_id and owner_user_id, orders by updated_at DESC, resource_id DESC, and takes 20 rows. On PostgreSQL, explain the shape of the regression, prove it from a query plan, and give the index you would add.

Approach

Start from the shape. A flat p50 with a moved p99 means a subset of requests changed cost, not all of them, so the first job is naming the subset. Bucket the endpoint's latency by the tenant's row count; the natural hypothesis is that large tenants are a small share of requests and all of the tail.
Get the plan for the new query on a large tenant with EXPLAIN (ANALYZE, BUFFERS). Expect an index scan over the tenant's range, a filter discarding most of it, then a Sort feeding the Limit, possibly reporting Sort Method: external merge Disk. Read actual rows on the scan node, not estimated.
Explain why the existing index cannot serve it. A composite B-tree is seekable only as a left prefix, and with no equality predicate on status the scan cannot treat updated_at as an ordering, because rows in the tenant's range are ordered by status first. Everything matching must be read and sorted before LIMIT 20 can apply, so a tenant with 400,000 rows pays 400,000 rows to return 20.
Add (tenant_id, owner_user_id, updated_at DESC, resource_id DESC). Equality on the first two columns leaves the index ordered by updated_at within that pair, so the plan becomes an index scan that stops after 20 rows with no Sort node. PostgreSQL can scan a B-tree backwards, so the DESC markers matter only if the two sort columns ever disagree in direction; keeping them explicit documents the order the keyset cursor depends on.
Price the fix. This is a fourth index on a table taking about 1.2k writes/second, and every insert and version bump maintains it. Justify it against the query it serves, and check whether it makes an existing index redundant, which here it does not, since the original still serves the status-filtered default listing.
Re-measure per tenant-size bucket rather than in aggregate. A fleet-wide p99 can improve while the largest tenant is still on the old plan.

Follow-up

The endpoint paginates with OFFSET. What does page 500 cost with your index, and what does the keyset version cost?
How would you have caught this before release, given that a 10,000-row seed database produces the same plan shape at an unnoticeable cost?
If a fourth index were unacceptable on write grounds, what else could serve this query?

Suggested practice checklist

For someone who has spent the last few years shipping features and reading other people's code, and who has not solved a timed problem from a blank file in a long time. Five days rebuild the primitives and the patterns that sit on them, working from invariants rather than remembered solutions, and the last two attach that back to the rest of the loop.

Day 1 — Rebuild the primitives by implementing them

Implement a dynamic array with doubling growth and an operation counter, then change the growth rule to add a fixed sixteen slots instead, and time both for n of ten thousand, a hundred thousand and a million. The fixed-increment version resizes n/16 times at O(n) each, so its total work is quadratic; doubling is what makes append amortised constant.
Implement a hash map with separate chaining and a load-factor resize, then insert ten thousand keys engineered to land in one bucket and record what happens to lookup time, so that average-case O(1) becomes a claim with a stated precondition rather than a reflex.
For dynamic-array append and hash-map insert, write down which cost is amortised rather than worst-case, which single operation pays the whole bill, and what a system with a hard per-operation deadline would have to do instead.

Deliverable: Two working implementations plus a timing table showing the input at which each structure's advertised complexity stops holding.

Day 2 — Arrays under an invariant: two pointers, sliding window, binary search

Solve longest-subarray-with-sum-at-most-K using a sliding window, then run it on an input containing negative numbers and watch it return the wrong answer: extending the window only moves the sum monotonically when every element is non-negative, and that precondition is the whole reason the technique works.
Write the binary search that finds the first index satisfying a predicate rather than an exact value, put the loop invariant above the loop in a comment, and verify termination on the two inputs that break careless versions: the empty range, and a range where every element satisfies the predicate.
Compute the midpoint as lo + (hi - lo) / 2 and write one line on why the obvious (lo + hi) / 2 is a genuine defect in a fixed-width integer type and a non-issue in a language with arbitrary-precision integers.

Deliverable: Three solved problems, each with its invariant written above the loop, plus one recorded input on which the sliding window is provably wrong.

Day 3 — Sorting, heaps, and the greedy argument that has to be proved

Solve one top-k problem three ways, by full sort, by a size-k heap, and by quickselect, then write the values of n and k at which each becomes the right choice, along with quickselect's quadratic worst case and why a randomised pivot makes that unlikely rather than impossible.
Implement bottom-up heapify and count sift-down steps to confirm it does linear work rather than n log n, because most nodes sit near the bottom of the tree and therefore move only a short distance.
Take interval scheduling by earliest finishing time and write the exchange argument out in full: given any optimal schedule, swapping in the earliest-finishing interval keeps it feasible and no smaller. Then construct the weighted variant where that same greedy fails and name what has to replace it.

Deliverable: A three-way top-k comparison with measured crossover points, one written exchange argument, and one counterexample to a greedy rule that looks almost identical.

Day 4 — Recursion, memoisation, and the step to a table

Take one problem with overlapping subproblems, such as edit distance or coin change, instrument the plain recursion with a call counter to show the blow-up, then add memoisation and re-count.
Convert the memoised version to a bottom-up table and state the two properties you relied on: each subproblem's result depends only on its arguments, and the dependencies form a DAG you can enumerate in order.
Rewrite one deep recursion with an explicit stack, then find the input length at which the original hits the interpreter's frame limit, which defaults to about a thousand frames in CPython, so you know when the rewrite is required rather than decorative.

Deliverable: One problem in three forms, naive, memoised and tabulated, with call counts for each and the input length at which recursion depth becomes the binding constraint.

Day 5 — Graphs, where most of the work is choosing the traversal

Implement BFS and DFS over one adjacency list, then answer for each which finds a shortest path in an unweighted graph and which you would use to detect a cycle in a directed graph, including why the in-progress versus finished distinction matters for the second.
Implement topological sort by in-degree, feed it a graph containing a cycle, and confirm the failure signature is that fewer than V nodes come out rather than an exception, then note that the order it produces is one of several valid ones.
Run a shortest-path search on a graph with a single negative edge weight and show the wrong answer, then write the precondition Dijkstra actually needs, non-negative weights, because it finalises a node's distance the first time that node is popped, and name the algorithm you would switch to and its own limit.

Deliverable: A small graph library with BFS, DFS and topological sort, plus two inputs that produce documented wrong answers under the wrong algorithm choice.

Day 6 — One day for everything that is not an algorithm

Sketch one system only to the depth a coding-heavy loop tends to reach: the endpoints, what the service stores, and the single query pattern that decides the schema. Stop at twenty-five minutes.
Prepare the project answer for an interviewer who codes, which means rehearsing the two levels they push to: the specific thing you built, and why you chose that approach over the alternative they will name. Open with a number and be ready to say what it excludes.
Prepare the answer to what you would do differently, choosing a real technical mistake with a specific fix rather than a complaint about process or staffing.

Deliverable: One design sketch at endpoint-and-schema depth, plus a project answer rehearsed to two levels of follow-up.

Day 7 — Solve out loud, under time

Do three timed problems at twenty-five minutes each in a plain editor with no autocomplete and no execution until the end, then tally separately the failures that were syntax and the ones that were approach, because those two numbers call for different fixes.
Narrate one solution from the first sentence, stating the approach and its complexity before writing any code, and rehearse the sentence you will use when you realise mid-solution that the approach is wrong.
Re-solve from blank the two problems you were slowest on this week and compare the times against the day they first appeared.

Deliverable: A recording of one fully narrated solution and a tally that separates syntax failures from approach failures.

Risky changes and the reversal path

For anything that touched live traffic, be ready to say how you would have undone it: a flag, a staged rollout, dual writes with the old path still authoritative. Once the old column is dropped or the source rows are overwritten there is no reverse, so name what you kept a copy of and for how long.

Ship under a deadline and bound the debt you chose

Medium · pagination · technical debt · tradeoffs

You have four days to ship a tenant-facing listing endpoint. The version you would defend uses keyset pagination over (tenant_id, status, updated_at DESC, resource_id DESC); the version you can finish uses LIMIT/OFFSET with no matching index. Describe a deadline call you actually made of this shape: what you shipped, what you knowingly deferred, how you bounded the damage with a mechanism rather than an intention, and the specific numeric condition that would force the follow-up. Name who you told and where you wrote it down.

Approach

Name the deferred failure precisely instead of calling it slow. OFFSET n makes the database produce and discard n rows, so cost grows with page depth; without an index matching the sort, every matching row is read and sorted before the limit applies; and rows inserted between two page fetches shift across the boundary so items are skipped or repeated with nothing in the response to signal it.
Bound the blast radius with something mechanical rather than a promise: cap maximum page depth, cap page size, restrict the endpoint to one internal caller, or keep it behind a flag. State which failure each cap removes and which it leaves standing.
Attach a number to the trigger and wire it to an alarm: the first tenant crossing N resources, or the endpoint's p99 crossing its share of the 400 ms budget, so the debt announces itself instead of waiting to be remembered.
Write it where the next engineer looks, which is the code and the ticket, not a chat message: what was deferred, why, the cap, and the trigger.
Report what actually happened in your real example, including the case where the trigger never fired and the debt was correctly never repaid.

Follow-up

At what page depth does the offset version breach your latency budget, given your page size and row counts?
What breaks first when you switch to keyset pagination later, and what does a client holding an old page token see?
Who would have overruled you if you had asked for two more days, and did you ask?

Tell callers you do not own that their integration breaks

Medium · deprecation · compatibility · stakeholders

A field in a write endpoint's response must change shape. You own the endpoint; you do not own the four internal callers or the outbound webhook consumers who read it. Describe a deprecation you were responsible for: what you shipped first, how you established who was actually reading the field, the window you gave and what set its length, what you did about the consumer who never moved, and how you decided removal was safe. Name the signal you used, not the announcement you sent.

Approach

Establish the reader set empirically rather than from a wiki of owners: per-field usage counters keyed by principal, or access logs attributed to a consumer. State the blind spot of whichever you pick, since a consumer that reads the field only on a monthly job will not appear in a week of logs.
Ship additive first. Populate the new field alongside the old one so no reader is forced to move, which is also what keeps a rolling deploy safe, because old and new instances answer the same requests at the same time and a rollback must still find the old shape present.
Set the window from the slowest legitimate consumer's release cadence, not from your calendar, and decide separately what to do for a consumer with no release process at all, such as an external webhook endpoint you can only email.
Convert silence into evidence before you rely on it: a short, low-traffic removal window that makes a still-dependent consumer fail visibly and loudly while you are watching, rather than at three in the morning after you have moved on.
State the removal criterion as a measurement with a duration attached, such as observed reads at zero across a full billing cycle, and keep the change reversible for one release after removal.

Follow-up

How would you detect a consumer that reads the field only during a monthly export?
One caller refuses to move and has a commercial relationship behind it. What changes in your plan and what does not?
After removal, what makes the change irreversible, and how long before you cross that line?

Narrate an outage you owned from page to postmortem

Hard · incident response · blast radius · postmortems

Pick an incident you personally drove, ideally one where writes were affected rather than reads. In six to eight minutes: state the symptom as it first appeared on a dashboard, the blast radius you established before you knew the cause, the mitigation you applied and when, the mechanism you eventually proved, and the follow-up that would prevent a repeat. Bring numbers: error rate, tenants affected, minutes to mitigate, minutes to resolve. If you cannot name what you measured, choose a different incident.

Approach

Open on the signal rather than the cause: which metric at which percentile moved, on which service, at what time, so the listener follows the same evidence you had rather than a conclusion you already reached.
Separate mitigation from diagnosis out loud. State what you did to stop the bleeding (flag off, shed traffic, drain a lease, roll back a deploy) and say plainly that you did it before the mechanism was known, because those are two jobs with different deadlines.
Establish blast radius in countable terms: how many tenants, how many writes, and crucially whether the effect was loss or only delay. An append-only revision table or a pending outbox row means the change survived and the projection was merely behind, which is a repair rather than a data-loss incident.
Prove the mechanism instead of asserting it. Name the trace span that grew, the plan that flipped to a sequential scan, the lease that expired, plus one alternative you ruled out and the signal that stayed flat while you ruled it out.
Close on the durable fix and its cost, distinguishing what landed that week from what needed an expand-and-contract migration across several deploys, and say which of the two you actually finished.

Follow-up

What would you do differently in the first five minutes, given the same dashboard and no more information?
Which follow-up action did you deliberately not take, and why was dropping it the right call?
How did you convince yourself the mitigation was safe to apply while the cause was still unknown?