Deepgram Machine Learning Engineer Interview Questions & Guide 2026

The 60-second brief

As a Machine Learning Engineer at Deepgram, you play a pivotal role in the development of cutting-edge speech recognition technology. Your expertise in machine learning algorithms and audio data processing directly contributes to improving the accuracy and efficiency of Deepgram's products. This role is essential for enhancing user experiences across various applications, from transcription services to voice-activated interfaces, ultimately driving business growth and customer satisfaction.

Most rounds test how you handle an under-specified problem more than what you recall. A wrong first approach you correct out loud is recoverable; a constraint you assumed silently and were never given is the expensive mistake.

Focus areas

• Detect concurrent edits instead of losing writes
• Make every write idempotent under retry
• Choose indexes from the query's access path

Overview

In this position, you will tackle complex problems involving large-scale data sets, requiring a deep understanding of both machine learning principles and audio processing. You will collaborate closely with cross-functional teams, including data scientists, software engineers, and product managers, to design innovative solutions that leverage Deepgram's unique capabilities. The work is not only technically challenging but also strategically significant, as it helps shape the future of voice technology.

The Deepgram interview process

Deepgram candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screening

The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.

What to demonstrate

• Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
• Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
• Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year

How to prepare

• Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
• Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
• Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter

Technical Interview

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

• In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
• In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
• Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

• Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
• Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
• For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Take-Home Project

Nobody is in the room when this is read, so the submission stands in for what an ordinary pull request from you would cost to review. That makes navigability the first thing being judged. A reader should reach the code that does the actual work within a minute of cloning, without tracing three layers of indirection to get there. What most separates a strong submission from a weak one is separation itself: the logic that implements the prompt kept apart from file reading, argument parsing and printing, so each piece can be read, and tested, on its own.

What to demonstrate

• Whether the entry point makes the shape of the program obvious: what is read, what is computed, what is written, in that order
• Whether the core logic can be called without touching the filesystem, a socket or stdin, since that is what makes it testable at all
• Whether names hold up on a second reading; a function called parse that also writes, or a total that means something different from the README, costs the reader more than a longer name would have
• Whether the commit history reads as a sequence of steps someone could follow, rather than one commit containing everything

How to prepare

• Clone your own finished submission into an empty directory and read it in the order a stranger would, README then entry point then downward, noting every place you rely on something the code never told you, and fix those places
• Take an exercise you have already solved and restructure it so the module holding the rules imports no file, CLI or network library, leaving input parsing and output as a thin shell around it
• Build the history while you work, committing at points where the tests pass, so the log shows the order you actually solved the problem in

Follow-Up Discussions

When a round has no standard shape, it is often there because something is still open: an area no earlier conversation reached, a round where the signal came out mixed, or a decision someone is not ready to make alone. Work out which by going back over what each earlier round actually covered rather than how it felt, and arrive able to give evidence on that point without being asked twice. Weak answers replay the loop's earlier material at the same depth. Strong ones go a level deeper and stay consistent with what you already said.

What to demonstrate

• Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
• Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
• Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
• Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened

How to prepare

• Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
• Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
• Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure

Where candidates lose points

Assuming an isolation level prevents the anomaly you actually have

Isolation levels are named by the SQL standard but implemented differently, so any claim about one is only true of a named engine. PostgreSQL defaults to READ COMMITTED, where every statement takes a fresh snapshot, so two statements inside one transaction can legitimately disagree about the same row. Its REPEATABLE READ is snapshot isolation: it removes non-repeatable and phantom reads but permits write skew, where two transactions each read a set, each conclude their own write is safe, both commit, and the combined result violates a constraint that no single row expresses. Only SERIALIZABLE closes that, and it closes it by aborting a transaction with a serialization failure (SQLSTATE 40001), which means the guarantee is theoretical unless the application has a retry loop. InnoDB's REPEATABLE READ is a different mechanism again - plain SELECTs read a consistent snapshot while locking reads and writes see the latest committed row - so a read-modify-write inside one transaction can act on a value that the transaction's own earlier SELECT never returned.

Shipping a migration and the code that depends on it as a single change

During any rolling deploy, and for as long as a rollback remains possible, old and new code execute against the same schema at the same time. A migration that drops or renames a column breaks every instance that has not restarted yet, and code that requires a column the migration has not applied breaks every instance that restarted early. The discipline is expand then contract: add the new column nullable, write both shapes, backfill in batches, move reads across once the backfill is verified, and only then stop writing the old shape and drop it - four deploys, usually spread over days. It feels disproportionate until the first rollback, at which point it is the only reason the previous version still runs.

Treating a network call as though it were a local function call

A remote call can be slow, fail, or return after you stopped waiting, so name the timeout, the retry policy, and what the caller sees while the dependency is down. A call with no timeout turns one slow dependency into an exhausted thread or connection pool in every service upstream of it.

Assuming the input fits in memory

Ask how large the input is in bytes before committing to an in-memory algorithm; beyond that point the options are a single streaming pass, an external sort with bounded buffers, or a sketch that trades exactness for constant memory. An algorithm that assumes random access to the whole input is a different algorithm from one that sees each element once.

Questions & practice

How would you approach improving the accuracy of an existing machine learning model?

Medium · machine learning fundamentals

Approach

1. Say how you would validate it, and where leakage could enter the split.
2. State the learning problem: the label, the unit of prediction and how the model is used.
3. Pick the metric from the cost of each error type, not from habit.

Follow-up

• How would you know the model is overfitting?
• Where could label leakage enter this setup?

Describe a project where you implemented a machine learning model. What were the outcomes?

Medium · machine learning fundamentals

Approach

1. State the learning problem: the label, the unit of prediction and how the model is used.
2. Say how you would validate it, and where leakage could enter the split.
3. Pick the metric from the cost of each error type, not from habit.

Follow-up

• How would you know the model is overfitting?
• What changes if the classes are heavily imbalanced?

Discuss a recent advancement in machine learning that you believe could impact audio processing.

Medium · machine learning fundamentals

Approach

1. Say how you would validate it, and where leakage could enter the split.
2. State the learning problem: the label, the unit of prediction and how the model is used.
3. Name the simplest model that could work and what would make you move past it.

Follow-up

• Where could label leakage enter this setup?
• What changes if the classes are heavily imbalanced?

Discuss the importance of feature engineering in machine learning applications.

Medium · machine learning fundamentals

Approach

1. State the learning problem: the label, the unit of prediction and how the model is used.
2. Pick the metric from the cost of each error type, not from habit.
3. Say how you would validate it, and where leakage could enter the split.

Follow-up

• How would you know the model is overfitting?
• Where could label leakage enter this setup?

Write a function to implement a basic speech recognition algorithm.

Medium · coding and algorithms

Approach

1. Choose the data structure from the access pattern, not from familiarity.
2. Name the brute-force solution and its complexity before improving on it.
3. State the target complexity and say which constraint rules the naive version out.

Follow-up

• What is the worst case, and how likely is it on real data?
• Which test case would catch an off-by-one here?

How would you optimize an algorithm for real-time audio processing?

Medium · coding and algorithms

Approach

1. Choose the data structure from the access pattern, not from familiarity.
2. Name the brute-force solution and its complexity before improving on it.
3. Walk one small example through your approach before writing the whole thing.

Follow-up

• How does this change if the input no longer fits in memory?
• What is the worst case, and how likely is it on real data?

Find overlapping job attempts and peak concurrency from lease records

Medium · sweep line · intervals · leases

A day of job_run history yields about 50,000,000 attempt records: (job_run_id, job_type, attempt, started_at, finished_at which is NULL when the worker died, lease_expires_at). Leases expire on a clock, so a job that outran its lease ran twice. Produce (a) every job_run_id whose attempts overlapped in wall-clock time and (b) the peak number of simultaneously running attempts per job_type with the minute it occurred. Target O(n log n). State how you treat a NULL finished_at and what clock skew does to your answer.

Approach

1. Define the interval before sorting anything: an attempt occupies [started_at, COALESCE(finished_at, lease_expires_at)). finished_at is observed and lease_expires_at is only a promise, so every attempt without a finish contributes an estimate and the whole result is a lower bound on overlap rather than an exact count.
2. For peak concurrency, sweep: emit 2n endpoints, sort by (timestamp, kind) with ends ordered before starts at equal timestamps, then walk the sequence maintaining a counter per job_type and record each type's maximum with its timestamp. O(n log n) dominated by the sort, O(n) space, or O(1) extra if the sort is external and the walk streams.
3. For overlap detection, do not compare attempts pairwise. A single global sort by (job_run_id, started_at) gives both the grouping and the order; within a group, keep the maximum end seen so far and report an overlap exactly when the next start is less than that running maximum, which is one linear pass after the sort.
4. Half-open intervals matter and are easy to get wrong: with closed intervals an attempt ending at the same millisecond another begins reads as concurrency two, and across 50,000,000 records that artefact swamps the real signal.
5. State the clock caveat: started_at and finished_at are written by different workers, so under skew of a few hundred milliseconds an apparent overlap shorter than that bound is not evidence. Filter reported overlaps by a minimum duration, or prefer timestamps written by whichever component heartbeats the lease.
6. Scale the sort rather than assuming it fits: the sweep emits two endpoints per attempt, so 50,000,000 records become 100,000,000 endpoints, and at roughly 24 bytes each, an 8-byte timestamp plus a 4-byte job_type plus a kind flag padded to alignment, that is about 2.4 GB of sort keys before any scratch space. Either push the ordering into the database behind an index on (job_type, started_at) or run an external merge sort in chunks; the overlap pass sorts n records rather than 2n, so it is the cheaper of the two.

Worked solution (30 min)

1. Write the interval derivation with the COALESCE and state in one line which of the two end sources is observed and which is assumed.
2. Write the concurrency sweep: the endpoint tuples, the sort key including the end-before-start tie-break, and the per-job_type counter.
3. Hand-trace four attempts of one job, two disjoint and two overlapping by three seconds, and confirm the overlap detector fires exactly once.
4. Add the skew filter as a minimum overlap duration, state the value you chose, and justify it from how the timestamps are written.

Expected result: One global sort by (job_run_id, started_at) driving a running-maximum-end check for overlaps, plus a sweep over 2n endpoints with ends ordered before starts for per-job_type peak concurrency, both O(n log n), with half-open intervals, NULL finished_at falling back to lease_expires_at, and overlaps shorter than the clock-skew bound excluded.

Verify it yourself

• Two attempts touching at exactly the same instant report concurrency 1, not 2.
• A job with a single attempt never appears in the overlap output regardless of its lease.
• The reported peak timestamp for a job_type lies inside at least one of that type's intervals.

Follow-up

• A handler is not idempotent and you have found 400 overlapping jobs. Which of them actually caused damage, and what would you query to find out?
• Peak concurrency for one job_type is 4 against a configured cap of 4. Is the cap working, or is the data hiding attempts that never started?
• How would you compute both answers incrementally as records arrive rather than in a daily batch?

Denormalise tenant onto revisions and backfill it live

Hard · denormalisation · online migration · backfill

resource_revision (revision_id, resource_id, version, actor_user_id, change_kind, patch, request_id, created_at) has 400M rows and no tenant column; tenant_id lives only on resource. Two reads need it: a tenant-scoped audit feed ordered by created_at DESC, and an offboarding purge. Both join back to resource today. Justify adding tenant_id to resource_revision against those two reads, name the anomaly the copy introduces and the constraint that prevents it, then give the ordered migration for a live table taking 1.2k writes/second — the lock each step takes, how the backfill is batched, and where each step stops being reversible. PostgreSQL 16.

Approach

1. Justify from the access path rather than from taste. Without the column, the audit feed either scans resource_revision by created_at and discards other tenants' rows, or resolves the tenant's resource_ids first and probes with them — both proportional to the tenant's whole history rather than to one page. With (tenant_id, created_at DESC, revision_id DESC) it is a seek that stops at 50 rows, and the purge becomes a ranged delete instead of a join.
2. Name the cost exactly: a second copy of a fact can disagree with the first. Make the disagreement unwritable rather than documented — add UNIQUE (resource_id, tenant_id) on resource so it can serve as a foreign-key target, then FOREIGN KEY (resource_id, tenant_id) REFERENCES resource (resource_id, tenant_id) on the revision table. A revision can then only ever carry its parent's tenant.
3. Step one, expand: ALTER TABLE resource_revision ADD COLUMN tenant_id BIGINT NULL, with no default, so it is a catalogue change and no rewrite. It still needs ACCESS EXCLUSIVE for an instant, and that instant queues behind the longest open transaction on the table while every later query queues behind it — set lock_timeout to 2s and retry rather than wait.
4. Step two, dual-write: deploy the writer that populates tenant_id on every new revision while reads still use the join. Reversible by redeploying the previous build, because nothing reads the column yet.
5. Step three, backfill: batch by primary key rather than by created_at so the cursor is dense and resumable — UPDATE resource_revision rr SET tenant_id = r.tenant_id FROM resource r WHERE r.resource_id = rr.resource_id AND rr.revision_id > $1 AND rr.revision_id <= $1 + 5000 AND rr.tenant_id IS NULL — committing per batch and persisting the cursor. Throttle on replica replay lag and on dead-tuple count, since each batch writes 5,000 new row versions. Run the backfill before the index exists so those updates can stay HOT.
6. Step four, index then enforce then contract: CREATE INDEX CONCURRENTLY (cannot run inside a transaction block, scans the table twice, waits on open transactions, and leaves an INVALID index to drop concurrently if it fails); ADD CONSTRAINT ... CHECK (tenant_id IS NOT NULL) NOT VALID, then VALIDATE CONSTRAINT, which takes only SHARE UPDATE EXCLUSIVE, after which SET NOT NULL uses the validated check instead of re-scanning on PostgreSQL 12 and later. Only then move the audit reads onto the column and, in a later deploy, delete the join path.

Worked solution (40 min)

1. Write the five steps as separate scripts and state, for each, the lock mode it acquires and the deploy it pairs with.
2. On a 20M-row copy, run the ADD COLUMN while a 30-second transaction holds a lock on the table, and record how long unrelated queries queue behind it.
3. Run the batched backfill at 5,000 rows, kill it mid-run, restart from the persisted cursor, and confirm no row is processed twice and none is skipped.
4. Build the index concurrently under concurrent write load, then add the CHECK ... NOT VALID, VALIDATE it and SET NOT NULL, timing each.
5. Compare the audit-feed plan before and after: join-and-filter versus an index seek with no Sort.

Expected result: ADD COLUMN completes in milliseconds once it holds the lock, but with a 30-second transaction open it stalls all traffic on that table for the full 30 seconds unless lock_timeout cuts it off. The backfill resumes exactly at the persisted revision_id. After VALIDATE, SET NOT NULL returns without a table scan. The audit feed drops from scanning the tenant's history to reading about 50 rows.

Verify it yourself

• SELECT count(*) FROM resource_revision WHERE tenant_id IS NULL returns 0, re-checked after the dual-writer has been live for a full deploy cycle.
• A sampled join confirms rr.tenant_id = r.tenant_id across 100k random revisions; the composite foreign key makes a mismatch unwritable from then on.
• pg_stat_activity shows no transaction older than one batch's duration while the backfill runs.
• EXPLAIN of the audit feed shows an Index Scan with no Sort node.

Follow-up

• The backfill is half finished and a rollback is required. What state is the table in, and what does the previous build do with a half-populated column?
• How do you verify the backfill actually finished, given rows are still being inserted while it runs?
• A resource must now be movable between tenants. What does that do to the composite foreign key and to the revisions already written?

Write the update path that detects a concurrent edit

Medium · optimistic concurrency · transactional outbox · audit history

resource carries version INT NOT NULL DEFAULT 1. resource_revision holds revision_id, resource_id, version, actor_user_id, change_kind, patch JSONB, request_id, created_at with UNIQUE (resource_id, version). outbox_event holds aggregate_type, aggregate_id, aggregate_version, event_type, payload, status. A PUT carries the version the client read. Write the exact statements for the single transaction that applies the edit, records the revision and enqueues 'resource.updated', and give the handler's branch on zero affected rows. Then say what PostgreSQL 16 does under READ COMMITTED when two of these updates hit one row at once.

Approach

1. One transaction, three writes, no network call inside it: UPDATE resource SET title = $3, version = version + 1, updated_at = now() WHERE resource_id = $1 AND tenant_id = $4 AND version = $2; then INSERT the resource_revision row at version $2 + 1; then INSERT the outbox_event row at the same aggregate_version. The event goes to a table rather than a broker because no transaction spans both.
2. Branch on the affected-row count before doing anything else. Zero has three causes — stale version, wrong tenant, row gone — so re-read once and map to 409 carrying the current version, or 404 for an id outside the caller's tenant, which also stops the endpoint confirming that another tenant's id exists.
3. State the engine behaviour instead of assuming it. Under READ COMMITTED the second UPDATE blocks on the row lock, and when the first commits PostgreSQL re-evaluates the WHERE clause against the newly committed row, so the version predicate now fails and the statement reports zero rows. Under REPEATABLE READ the identical collision raises SQLSTATE 40001 instead, so the handler must fold both shapes into one conflict response.
4. Keep UNIQUE (resource_id, version) even though the predicate already serialises writers. It is what makes a lost update unwritable if any other path ever reaches the revision table, and it converts a logic bug into 23505 rather than into a silently missing history row.
5. Refuse to auto-retry the whole PUT. A retry re-reads the winner's state and reapplies an intent formed against data that no longer exists — the silent overwrite the version token was added to detect. Return the conflict; merge field-wise only if the patches are provably disjoint.
6. Note that now() is the transaction timestamp in PostgreSQL, so resource.updated_at, the revision's created_at and the outbox row share one instant, which is what later makes reconciliation between the three tables unambiguous.

Follow-up

• A client sends the version it read ten minutes ago and the resource has moved three versions. What is in your 409 so it can resolve the conflict without a full re-fetch?
• Two editors, two disjoint fields, no overlap. Does your answer still refuse the second write, and should it?
• Every write now touches a second hot table. How do you keep the outbox insert and its partial index from becoming the write bottleneck at 1.2k writes/second?

If tasked with designing a new feature for a speech-to-text application, how would you prioritize your approach?

Medium

Approach

1. Say what you would check first and why it is the highest-information step.
2. Work from the requirement backwards to the design.
3. State your assumptions explicitly before working the problem.

Follow-up

• How would you know your answer was wrong?
• What assumption would you test first?

Make resource creation safe for a client that retried a timeout

Medium · idempotency · retries · race conditions

POST /v1/resources creates a resource row and, in the same transaction, a resource_revision and an outbox_event. A partner's HTTP client times out at 2 seconds and retries twice; under load the first request is still executing when the retry arrives, and duplicate resources are appearing. Using idempotency_key (tenant_id, idempotency_key, request_fingerprint, state, response_status, response_body, resource_id, locked_until, expires_at), specify the whole contract: first call, concurrent retry, retry after completion, a key reused with a different body, and what the client does with each response.

Approach

1. Name why the plain REST shape fails here. POST is not idempotent, a timeout leaves the client unable to distinguish a lost request from a lost response, so it must retry - and the retry is a second create. Checking whether the key exists and then inserting it is check-then-act: under exactly the concurrency that caused the timeout, both attempts look, both see nothing, both proceed.
2. Let the unique constraint arbitrate instead. INSERT the key row with state 'in_flight' and locked_until set to now plus the request deadline, and commit that reservation on its own before any work is attempted - an uncommitted row is invisible to the concurrent retry, which is the entire point of writing it first. One caller wins the insert and proceeds; the other takes the unique violation and reads the row rather than falling through to the work.
3. Branch the loser on what it found. 'succeeded' replays response_status and response_body verbatim, with a header marking it a replay. 'in_flight' with locked_until in the future returns 409 and a Retry-After - do not park a request thread waiting on another request, that is how a concurrency bound is consumed by retries. 'in_flight' past locked_until is reclaimable by a conditional UPDATE predicated on the old locked_until, so exactly one reclaimer wins. A fingerprint mismatch is 422 and never the stored response: a different body under the same key is a client defect, and serving the old response hides it forever.
4. Commit the effect and the key's completion together - resource insert, revision row, outbox row, and the UPDATE of the reserved key to 'succeeded' with the stored response, in one transaction. That makes a crash binary: either all of it is durable and the key reads 'succeeded', or the transaction rolls back and the only durable trace is the reservation row, still 'in_flight', with no resource behind it. The reclaim branch finishes that case - once locked_until passes, one retry takes the row by conditional UPDATE and runs the work for the first time. Splitting the two commits is what creates the window with no recovery: the effect durable, the key still 'in_flight', and a reclaimer that cannot distinguish that from work never started, so it creates a second resource.
5. State the lifetime explicitly. Keys expire - 24 hours is a common choice - and are swept, so the client's total retry window must be shorter than the TTL, or a late retry re-executes with no record that it is a duplicate. The key is scoped by tenant so two tenants choosing the same string do not collide, and the client generates one key per logical attempt, not per network send.
6. Write the client rules: on timeout, retry the same key with capped backoff; on 409, wait Retry-After and retry the same key; on 422, stop and fix the caller; on 2xx, treat the body as authoritative whether it was executed or replayed.

Worked solution (30 min)

1. Write the two-request interleaving that defeats read-then-insert, with the line each request is executing at each step.
2. Write the insert-first sequence and the four branches taken on unique violation, as a table of state, locked_until and response.
3. Put the completion UPDATE inside the same transaction as the resource, revision and outbox writes, then state what a crash before that commit leaves durable and which branch cleans it up.
4. Define the key's scope, lifetime and generation rule, then bound the client's retry budget by the TTL.
5. Fire two concurrent requests carrying one key and count rows in resource and resource_revision.

Expected result: One resource row, one revision row and one outbox row; the second caller either replays the first response or receives 409 with Retry-After; a different body under the same key is rejected 422; no request thread blocks waiting on another request.

Verify it yourself

• A hundred runs of two simultaneous same-key requests produce exactly one resource and one revision every time.
• Killing the process mid-work leaves zero resource rows and the reservation still 'in_flight'; after locked_until passes a retry reclaims the key and executes, so the crash and the retry together produce exactly one resource row.
• The documented client retry budget is strictly shorter than expires_at minus created_at.

Follow-up

• Response bodies are pruned at 24 hours but resource_id is kept. What do you return to a replay that arrives after pruning?
• The endpoint now also sends a welcome email. Which part of this contract prevents a second send, and which part cannot?
• What exactly breaks if the key is scoped globally rather than per tenant?

p99 jumped on one listing filter while p50 stayed flat

Medium · query planning · composite index · percentiles · tail latency

After a release that added an owner_user_id filter to the resource listing, p99 rose from 90 ms to 1.9 s while p50 stayed at 40 ms. Traffic and row counts are unchanged. resource carries the index (tenant_id, status, updated_at DESC, resource_id DESC). The new query filters tenant_id and owner_user_id, orders by updated_at DESC, resource_id DESC, and takes 20 rows. On PostgreSQL, explain the shape of the regression, prove it from a query plan, and give the index you would add.

Approach

1. Start from the shape. A flat p50 with a moved p99 means a subset of requests changed cost, not all of them, so the first job is naming the subset. Bucket the endpoint's latency by the tenant's row count; the natural hypothesis is that large tenants are a small share of requests and all of the tail.
2. Get the plan for the new query on a large tenant with EXPLAIN (ANALYZE, BUFFERS). Expect an index scan over the tenant's range, a filter discarding most of it, then a Sort feeding the Limit, possibly reporting Sort Method: external merge Disk. Read actual rows on the scan node, not estimated.
3. Explain why the existing index cannot serve it. A composite B-tree is seekable only as a left prefix, and with no equality predicate on status the scan cannot treat updated_at as an ordering, because rows in the tenant's range are ordered by status first. Everything matching must be read and sorted before LIMIT 20 can apply, so a tenant with 400,000 rows pays 400,000 rows to return 20.
4. Add (tenant_id, owner_user_id, updated_at DESC, resource_id DESC). Equality on the first two columns leaves the index ordered by updated_at within that pair, so the plan becomes an index scan that stops after 20 rows with no Sort node. PostgreSQL can scan a B-tree backwards, so the DESC markers matter only if the two sort columns ever disagree in direction; keeping them explicit documents the order the keyset cursor depends on.
5. Price the fix. This is a fourth index on a table taking about 1.2k writes/second, and every insert and version bump maintains it. Justify it against the query it serves, and check whether it makes an existing index redundant, which here it does not, since the original still serves the status-filtered default listing.
6. Re-measure per tenant-size bucket rather than in aggregate. A fleet-wide p99 can improve while the largest tenant is still on the old plan.

Follow-up

• The endpoint paginates with OFFSET. What does page 500 cost with your index, and what does the keyset version cost?
• How would you have caught this before release, given that a 10,000-row seed database produces the same plan shape at an unnoticeable cost?
• If a fourth index were unacceptable on write grounds, what else could serve this query?

Suggested practice checklist

For someone who has spent the last few years shipping features and reading other people's code, and who has not solved a timed problem from a blank file in a long time. Five days rebuild the primitives and the patterns that sit on them, working from invariants rather than remembered solutions, and the last two attach that back to the rest of the loop.

Day 1 — Rebuild the primitives by implementing them

• Implement a dynamic array with doubling growth and an operation counter, then change the growth rule to add a fixed sixteen slots instead, and time both for n of ten thousand, a hundred thousand and a million. The fixed-increment version resizes n/16 times at O(n) each, so its total work is quadratic; doubling is what makes append amortised constant.
• Implement a hash map with separate chaining and a load-factor resize, then insert ten thousand keys engineered to land in one bucket and record what happens to lookup time, so that average-case O(1) becomes a claim with a stated precondition rather than a reflex.
• For dynamic-array append and hash-map insert, write down which cost is amortised rather than worst-case, which single operation pays the whole bill, and what a system with a hard per-operation deadline would have to do instead.

Deliverable: Two working implementations plus a timing table showing the input at which each structure's advertised complexity stops holding.

Day 2 — Arrays under an invariant: two pointers, sliding window, binary search

• Solve longest-subarray-with-sum-at-most-K using a sliding window, then run it on an input containing negative numbers and watch it return the wrong answer: extending the window only moves the sum monotonically when every element is non-negative, and that precondition is the whole reason the technique works.
• Write the binary search that finds the first index satisfying a predicate rather than an exact value, put the loop invariant above the loop in a comment, and verify termination on the two inputs that break careless versions: the empty range, and a range where every element satisfies the predicate.
• Compute the midpoint as lo + (hi - lo) / 2 and write one line on why the obvious (lo + hi) / 2 is a genuine defect in a fixed-width integer type and a non-issue in a language with arbitrary-precision integers.

Deliverable: Three solved problems, each with its invariant written above the loop, plus one recorded input on which the sliding window is provably wrong.

Day 3 — Sorting, heaps, and the greedy argument that has to be proved

• Solve one top-k problem three ways, by full sort, by a size-k heap, and by quickselect, then write the values of n and k at which each becomes the right choice, along with quickselect's quadratic worst case and why a randomised pivot makes that unlikely rather than impossible.
• Implement bottom-up heapify and count sift-down steps to confirm it does linear work rather than n log n, because most nodes sit near the bottom of the tree and therefore move only a short distance.
• Take interval scheduling by earliest finishing time and write the exchange argument out in full: given any optimal schedule, swapping in the earliest-finishing interval keeps it feasible and no smaller. Then construct the weighted variant where that same greedy fails and name what has to replace it.

Deliverable: A three-way top-k comparison with measured crossover points, one written exchange argument, and one counterexample to a greedy rule that looks almost identical.

Day 4 — Recursion, memoisation, and the step to a table

• Take one problem with overlapping subproblems, such as edit distance or coin change, instrument the plain recursion with a call counter to show the blow-up, then add memoisation and re-count.
• Convert the memoised version to a bottom-up table and state the two properties you relied on: each subproblem's result depends only on its arguments, and the dependencies form a DAG you can enumerate in order.
• Rewrite one deep recursion with an explicit stack, then find the input length at which the original hits the interpreter's frame limit, which defaults to about a thousand frames in CPython, so you know when the rewrite is required rather than decorative.

Deliverable: One problem in three forms, naive, memoised and tabulated, with call counts for each and the input length at which recursion depth becomes the binding constraint.

Day 5 — Graphs, where most of the work is choosing the traversal

• Implement BFS and DFS over one adjacency list, then answer for each which finds a shortest path in an unweighted graph and which you would use to detect a cycle in a directed graph, including why the in-progress versus finished distinction matters for the second.
• Implement topological sort by in-degree, feed it a graph containing a cycle, and confirm the failure signature is that fewer than V nodes come out rather than an exception, then note that the order it produces is one of several valid ones.
• Run a shortest-path search on a graph with a single negative edge weight and show the wrong answer, then write the precondition Dijkstra actually needs, non-negative weights, because it finalises a node's distance the first time that node is popped, and name the algorithm you would switch to and its own limit.

Deliverable: A small graph library with BFS, DFS and topological sort, plus two inputs that produce documented wrong answers under the wrong algorithm choice.

Day 6 — One day for everything that is not an algorithm

• Sketch one system only to the depth a coding-heavy loop tends to reach: the endpoints, what the service stores, and the single query pattern that decides the schema. Stop at twenty-five minutes.
• Prepare the project answer for an interviewer who codes, which means rehearsing the two levels they push to: the specific thing you built, and why you chose that approach over the alternative they will name. Open with a number and be ready to say what it excludes.
• Prepare the answer to what you would do differently, choosing a real technical mistake with a specific fix rather than a complaint about process or staffing.

Deliverable: One design sketch at endpoint-and-schema depth, plus a project answer rehearsed to two levels of follow-up.

Day 7 — Solve out loud, under time

• Do three timed problems at twenty-five minutes each in a plain editor with no autocomplete and no execution until the end, then tally separately the failures that were syntax and the ones that were approach, because those two numbers call for different fixes.
• Narrate one solution from the first sentence, stating the approach and its complexity before writing any code, and rehearse the sentence you will use when you realise mid-solution that the approach is wrong.
• Re-solve from blank the two problems you were slowest on this week and compare the times against the day they first appeared.

Deliverable: A recording of one fully narrated solution and a tally that separates syntax failures from approach failures.

The estimate and the day you knew

A slipped date is only a bad story if you sat on it. What matters is what you believed when you gave the number, the signal that told you it was wrong, how many days passed before you said so, and what you cut rather than asking for more time. Scope you defended counts as much as scope you dropped.

Discuss a situation where you took the lead on a project. What was the outcome?

Medium · behavioural and collaboration

Approach

1. Pick a story where you made the decision, not one where you watched it.
2. Give the blast radius: what could have broken, and what you measured.
3. Close with what you would do differently, concretely.

Follow-up

• How did you know your change caused the improvement?
• What did you decide not to do, and why?

How do you ensure effective communication when working with non-technical stakeholders?

Medium · behavioural and collaboration

Approach

1. Give the blast radius: what could have broken, and what you measured.
2. State the situation in two sentences and spend the rest on the reasoning.
3. Pick a story where you made the decision, not one where you watched it.

Follow-up

• How did you know your change caused the improvement?
• What did you decide not to do, and why?

Estimate work you have never done and defend the range

Hard · estimation · backfills · expand-contract

You are asked to estimate a change you have never attempted: add a column to a 100-million-row table, populate it, move reads across, and drop the old shape. Give a range with the assumptions that generate it, including batch size, the signal your backfill throttles on, and wall-clock hours, and name the three unknowns that would move the number most. Then describe a real estimate you gave under comparable ignorance: how you expressed its uncertainty, what you committed to, and how wrong you turned out to be.

Approach

1. Decompose into independently deployable steps before estimating anything: add the column nullable, write both shapes, backfill in batches, verify, move reads, stop writing the old shape, drop it. That is four deploys spread over days, and the calendar estimate is dominated by them rather than by the loop's runtime.
2. Do the arithmetic aloud for the part that has arithmetic in it: batch size times number of batches times per-batch duration, at a write rate the primary can absorb alongside roughly 1.2k writes per second of production traffic. The loop is throttled by replication lag and lock waits, not by how fast it can issue statements.
3. Price the schema step by its lock rather than its statement duration. In PostgreSQL an ALTER TABLE taking ACCESS EXCLUSIVE waits for every open transaction on that table while later queries queue behind it, so a millisecond change issued during a thirty-second analytics query stalls that table for thirty seconds. Adding a nullable column with a non-volatile default avoids a rewrite from version 11; a new index wants CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block and leaves an invalid index behind if it fails.
4. Express the answer as a range whose endpoints each trace to a stated assumption, then name the cheapest experiment that collapses it, which is almost always running one real batch against the real table and multiplying.
5. Commit to a checkpoint rather than a completion date: the day you report a measured number from that first batch. That is a promise you can keep under uncertainty, and it is what the asker actually needs in order to plan.

Follow-up

• How do you verify the backfill genuinely finished, given rows written by production traffic while it ran?
• Where does the backfill resume from after a worker is killed mid-batch, and what makes that resume point trustworthy?
• Your first batch comes back ten times slower than assumed. What do you tell the person waiting on the estimate, and when?