Interview guides · Observe.AI

Observe.AI Machine Learning Engineer Interview Questions & Guide 2026

Observe.AI Machine Learning Engineer interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.

The 60-second brief

At Observe.AI, a Machine Learning Engineer is at the very center of the company's core product offering: transforming contact center interactions into actionable business intelligence. Observe.AI relies heavily on advanced speech-to-text, natural language processing (NLP), sentiment analysis, and generative AI models to analyze millions of customer service calls and chats. As a Machine Learning Engineer, you will not simply be applying off-the-shelf models; you will be designing, training, and optimizing proprietary algorithms that can interpret nuances, emotion, and context in real-time.

Prepare in one language you know well enough to debug in rather than the one you think reads best. Under a clock an unfamiliar language costs you standard-library lookups and iteration mechanics, and that time comes out of your thinking budget, not your typing budget.

Focus areas

Paginate large result sets with keyset cursors
Detect concurrent edits instead of losing writes
Choose indexes from the query's access path

Overview

The impact of this role is massive. The models you build directly power the automated agent coaching, compliance monitoring, and conversation analytics that top-tier enterprises rely on daily. Because contact center data is highly unstructured, noisy, and massive in scale, your work will involve solving complex engineering and scientific challenges. This includes optimizing models for low-latency inference, handling diverse audio qualities, and building robust pipelines that can scale seamlessly.

To succeed in this role, you must possess a rare blend of deep theoretical machine learning knowledge and exceptional software engineering execution. You will work closely with product managers, platform engineers, and data scientists to move models from conceptual research into high-throughput production environments. It is a highly challenging but rewarding environment where your technical decisions directly shape the capabilities of the core product.

The Observe.AI interview process

Observe.AI candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Profile Selection

When a round has no standard shape, it is often there because something is still open: an area no earlier conversation reached, a round where the signal came out mixed, or a decision someone is not ready to make alone. Work out which by going back over what each earlier round actually covered rather than how it felt, and arrive able to give evidence on that point without being asked twice. Weak answers replay the loop's earlier material at the same depth. Strong ones go a level deeper and stay consistent with what you already said.

What to demonstrate

Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened

How to prepare

Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure

Hands-on ML Coding Round

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

ML System Design Round

What to demonstrate

Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened

How to prepare

Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure

Managerial Discussion

What to demonstrate

Whether your account of a project matches the one you gave earlier in the loop, since what you said before may be available to whoever runs this round
Whether you can go a level deeper on something already covered, reaching the decision and its alternatives rather than repeating the summary
Whether you state your own uncertainty accurately, including parts of a system you did not build and decisions you inherited, instead of claiming even ownership across all of it
Whether you can answer a question you handled poorly earlier by naming what you missed, rather than delivering a polished second version as if the first had not happened

How to prepare

Reconstruct the loop on one page: for each round, the questions you were asked and the answer you actually gave, not the better one you thought of afterwards. The gaps on that page are your best available guess at why this round exists.
Take the two claims you made earlier that carry the most weight and assemble the backing for each: the measurement, the date, what broke, the decision you would make differently now.
Write down the three facts about your work that must not drift between tellings, such as team size, timeline and your own role, and check your stories against that list rather than trusting recall under pressure

Final HR Round

Many of these questions are about something that went wrong, and the grading sits mostly in the hours after you knew. Who found out first, whether that was you or an alert or a user, how long it took you to say it out loud, and whether the people who needed the news got it while they could still act on it. Engineers under-tell this part because it feels like confessing. The pattern it is looking for is the opposite: the quiet fix, an incident absorbed without telling anyone, after which nothing changed and the same failure is still available.

What to demonstrate

How the problem was found, and whether that route was one you had built or one that happened to you, since a user reporting it first means your instrumentation did not cover that failure
Whether time-to-detect and time-to-tell are separate numbers in your account and whether you know both, because a fast fix that nobody heard about until the retro is a different answer from a slow one that was announced immediately
Whether the resolution left something durable behind, a check that fires or a default that changed, rather than depending on people remembering to be careful
Whether you can say what the failure cost without either inflating it or waving it away

How to prepare

Reconstruct one incident you were part of as a timeline with clock times: first bad request, first signal, first person who knew, first message outside the team, mitigation, permanent fix. The gaps between those entries are what gets asked about
Look up the configuration of the signal that caught it, including its evaluation window and threshold. An alert defined on a five-minute aggregate cannot fire until the condition holds across that window, which puts a floor under time-to-detect that has nothing to do with how severe the failure was. Be able to say what that floor was and whether anyone had chosen it deliberately
Prepare one story where you escalated early and the severity turned out to be smaller than you thought, including what it cost the people you pulled in. Without it, every answer you give about raising alarms is unfalsifiable

Where candidates lose points

Assuming an isolation level prevents the anomaly you actually have

Isolation levels are named by the SQL standard but implemented differently, so any claim about one is only true of a named engine. PostgreSQL defaults to READ COMMITTED, where every statement takes a fresh snapshot, so two statements inside one transaction can legitimately disagree about the same row. Its REPEATABLE READ is snapshot isolation: it removes non-repeatable and phantom reads but permits write skew, where two transactions each read a set, each conclude their own write is safe, both commit, and the combined result violates a constraint that no single row expresses. Only SERIALIZABLE closes that, and it closes it by aborting a transaction with a serialization failure (SQLSTATE 40001), which means the guarantee is theoretical unless the application has a retry loop. InnoDB's REPEATABLE READ is a different mechanism again - plain SELECTs read a consistent snapshot while locking reads and writes see the latest committed row - so a read-modify-write inside one transaction can act on a value that the transaction's own earlier SELECT never returned.

Shipping a migration and the code that depends on it as a single change

During any rolling deploy, and for as long as a rollback remains possible, old and new code execute against the same schema at the same time. A migration that drops or renames a column breaks every instance that has not restarted yet, and code that requires a column the migration has not applied breaks every instance that restarted early. The discipline is expand then contract: add the new column nullable, write both shapes, backfill in batches, move reads across once the backfill is verified, and only then stop writing the old shape and drop it - four deploys, usually spread over days. It feels disproportionate until the first rollback, at which point it is the only reason the previous version still runs.

Assuming fixed-width integer arithmetic cannot overflow

In languages with fixed-width integers, including C, C++, Java, Go and Rust, computing a midpoint as (lo + hi) / 2 overflows once the sum passes the type's maximum, so write lo + (hi - lo) / 2 instead. Say which language you are in: arbitrary-precision integers, as in Python or Ruby, remove this specific hazard and none of the others.

Reading the constraints as preamble rather than as part of the problem

The bounds are usually there to eliminate the obvious approach: n up to 10^5 makes an O(n^2) scan roughly 10^10 operations, far outside any per-test time budget, and an input larger than memory rules out loading it at all. When a bound is not given, ask for it, then say out loud which approach it kills.

Questions & practice

Implement a custom k-nearest neighbors (KNN) or decision tree algorithm from scratch without using high-level ML libraries.

Medium · machine learning fundamentals

Approach

Pick the metric from the cost of each error type, not from habit.
State the learning problem: the label, the unit of prediction and how the model is used.
Say how you would validate it, and where leakage could enter the split.

Follow-up

How would you know the model is overfitting?
What changes if the classes are heavily imbalanced?

Write a Python script from scratch to tokenize, clean, and preprocess a raw conversational dataset for an NLP task.

Medium · machine learning fundamentals

Approach

Name the simplest model that could work and what would make you move past it.
Say how you would validate it, and where leakage could enter the split.
Pick the metric from the cost of each error type, not from habit.

Follow-up

Where could label leakage enter this setup?
How would you know the model is overfitting?

How do transformer architectures handle long-range dependencies in text compared to recurrent neural networks (RNNs)?

Medium · machine learning fundamentals

Approach

Pick the metric from the cost of each error type, not from habit.
State the learning problem: the label, the unit of prediction and how the model is used.
Say how you would validate it, and where leakage could enter the split.

Follow-up

Where could label leakage enter this setup?
How would you know the model is overfitting?

How do you address class imbalance when training a classifier on highly skewed conversation data?

Medium · machine learning fundamentals

Approach

Say how you would validate it, and where leakage could enter the split.
Pick the metric from the cost of each error type, not from habit.
Name the simplest model that could work and what would make you move past it.

Follow-up

Where could label leakage enter this setup?
What changes if the classes are heavily imbalanced?

Merge partitioned event streams into one ordered feed with bounded lateness

Hard · k-way merge · watermarks · out-of-order streams

The read-model service consumes 64 log partitions carrying about 4,000 events per second in total. Each partition is ordered within itself, but partitions drift by up to 30 seconds, and the activity feed must present a tenant's events in occurred_at order. Produce the merge. State its complexity, the buffer it requires in events and in bytes, what happens when one partition is idle, and what you do with an event that arrives after you have already emitted its position. Payloads average 1 KB.

Approach

Merge with a min-heap over the 64 partition heads keyed on (occurred_at, event_id): O(log P) per event and O(n log P) overall. The tie-break on event_id is what makes the output deterministic when two partitions carry the same millisecond, which matters because the feed is paginated and a non-deterministic order reorders pages under the reader.
Emitting the heap head is only correct once every partition has produced everything up to that timestamp, so the emit condition is a watermark: the minimum across partitions of the highest occurred_at seen, less the allowed lateness. Events are held until the watermark passes them, which is what turns individually ordered streams into a jointly ordered one.
Size the buffer from the lateness rather than guessing: 4,000 events per second times 30 seconds is 120,000 buffered events, and at 1 KB each about 120 MB of heap. That number is the real price of the ordering guarantee and belongs in front of whoever asked for it.
Handle the idle partition explicitly, because it fails the feed rather than corrupting it: a partition with no traffic never advances its own maximum, so the watermark freezes and output stops entirely. Either every partition emits a periodic idle marker carrying the broker's current time, or the watermark falls back to wall clock for a partition silent beyond a threshold.
Choose the late-event policy from what the projection is keyed on. The projection upserts on (aggregate_id, aggregate_version) and discards a version it has already applied, so a late event is safe to apply out of order and correctness never depended on the merge at all. Apply it, recompute the affected feed page, and count lateness so the 30-second budget can be re-derived from data rather than folklore.
Say what the merge does not buy: ordering is guaranteed within one aggregate by the log's partitioning, and no watermark makes the cross-aggregate order authoritative. Two events from different aggregates in the same millisecond have no true order, so the feed's order is a presentation choice that must be stable rather than correct.

Worked solution (35 min)

Write the heap comparator on (occurred_at, event_id) and the per-partition head refill.
Write the watermark computation and the emit-loop condition, then list which buffered events are held at a chosen instant.
Compute the buffer at 4,000 events per second, 30 seconds and 1 KB per event, and state what fraction of a worker's heap that represents.
Add the idle-partition marker and trace the watermark with one silent partition, both with and without the marker.
Write the late-event path and name the key that makes applying it safe.

Expected result: A 64-way min-heap merge at O(n log P) with a deterministic (occurred_at, event_id) comparator, a watermark of the per-partition minimum less 30 seconds gating emission, a stated buffer of 120,000 events and roughly 120 MB, idle markers so a silent partition cannot freeze the watermark, and a late-event policy justified by the projection's idempotency on (aggregate_id, aggregate_version).

Verify it yourself

Feeding the partitions in an adversarial interleaving produces the same output sequence as feeding them in timestamp order.
Silencing one partition stalls output without the idle marker and does not stall it with the marker.
Applying a late event twice leaves the projection byte-identical, which is what permits the out-of-order path to exist.

Follow-up

The lateness budget is raised to five minutes. What is the new buffer, and what besides memory changes?
The consumer restarts. Where does it resume from, and what does the feed look like for the first 30 seconds?
One partition is ten minutes behind because its producer is slow. Do you stall the feed or emit without it?

Replace offset paging on the resource feed with keyset

Easy · keyset pagination · composite index · sort order

resource holds resource_id, tenant_id, owner_user_id, title, body_ref, version, status ('draft','active','archived','deleted'), created_at, updated_at, deleted_at, with an index on (tenant_id, status, updated_at DESC, resource_id DESC). The listing endpoint returns active resources for one tenant, newest update first, 50 per page, today with LIMIT 50 OFFSET n. Tenants reach page 400 and rows are created while they read. Write the keyset query, define what the cursor carries and how it is encoded, and say which part of the index each predicate uses. Assume PostgreSQL 16.

Approach

Name the two failures separately. OFFSET 20000 makes the server produce and discard 20,000 rows, so page cost grows with depth rather than with page size. Independently, any write that changes how many rows sort above the offset moves the window between two fetches, and the direction decides which anomaly you get: an insert lands at the head of updated_at DESC and pushes already-returned rows down past the boundary, so they are returned a second time; a delete above the offset, or a row whose updated_at is bumped above the cursor, pulls rows up and one is never returned at all. Nothing in the response reveals either.
Write the seek: WHERE tenant_id = $1 AND status = 'active' AND (updated_at, resource_id) < ($2, $3) ORDER BY updated_at DESC, resource_id DESC LIMIT 50. The row-value comparison is one index range rather than a disjunction, and both columns are NOT NULL, which is what makes that comparison well defined.
Map each predicate onto the index: tenant_id and status are equality on the leading columns, (updated_at, resource_id) is the range, and the ORDER BY matches the index order so no Sort node appears and the scan stops after 50 rows. The DESC in the definition only matters for mixed directions — a plain ascending btree on the same columns is read backwards for this query.
Put both sort columns in the cursor and nothing the client can tamper with into another tenant: base64 of (updated_at, resource_id), validated server-side, with tenant_id taken from the principal.
State the residual honestly. Keyset is stable against concurrent inserts and deletes, but not against a row whose updated_at changes mid-scroll — that row moves in the ordering and can be seen twice. If the feed must be a snapshot, order by an immutable key or bound the page set with updated_at <= the cursor's start value.
Keep a total out of the page path. A tenant-wide COUNT(*) is the scan keyset just removed; fetch LIMIT 51 and return has_more instead.

Worked solution (25 min)

Seed one tenant with 500k active resources, 2% of them sharing an identical updated_at.
Time LIMIT 50 OFFSET 0 against OFFSET 20000 and record rows-read from EXPLAIN (ANALYZE, BUFFERS) for each.
Page the whole set with the keyset query while an insert-only writer adds 100 rows/second, collecting resource_ids, and repeat the run with OFFSET.
Repeat both runs under a second writer profile that also deletes 20 rows/second from pages already returned and bumps updated_at on 20 more, and diff each collected id set against the rows that existed for the whole run.
Remove resource_id from the cursor so the seek degrades to updated_at < $2, and re-run the tie-heavy section of the feed.

Expected result: Keyset reads about 50 index entries per page and its latency is flat from page 1 to page 10,000; under both writer profiles the collected ids contain no duplicate and omit no row that existed for the whole scroll. OFFSET at page 400 reads 20,050 rows to return 50. Under the insert-only writer it repeats rows and skips none — an insert can only push a row down past the boundary — and only the second profile, whose deletes and updated_at bumps move rows up past the cursor, also makes it skip. Without the tie-break, the seek is updated_at < $2, which drops every still-unread row sharing the boundary timestamp.

Verify it yourself

EXPLAIN shows an Index Scan with no Sort node — a Sort means the ORDER BY is not satisfied by the index.
Buffers read per page stays constant as page depth grows.
Ids collected across the scroll are unique, and complete for rows untouched during the run.

Follow-up

The client asks for 'jump to page 400'. What do you offer instead, and what does the honest version cost?
Sort order becomes user-selectable across four columns. How many indexes is that, and which would you refuse to add?
What does the cursor do when the row it points at has since been deleted?

Find version gaps and relay lag with window functions

Medium · window functions · event tables · frame semantics

outbox_event holds event_id, aggregate_type, aggregate_id, aggregate_version, event_type, payload, status ('pending','published','dead'), attempts, created_at, published_at. A projection is missing rows and you must decide whether the relay skipped events or the consumer dropped them. Write three queries over the last seven days: one listing every aggregate_id whose published aggregate_version sequence has a hole, one giving per-day counts with a running total, and one returning the newest published event per aggregate. For each, say where the window function is evaluated relative to WHERE and LIMIT. PostgreSQL 16.

Approach

Gaps: compute lead(aggregate_version) OVER (PARTITION BY aggregate_id ORDER BY aggregate_version) in a subquery, then filter next_version <> aggregate_version + 1 in the outer query. Window functions are evaluated after WHERE, GROUP BY and HAVING and before the outer ORDER BY and LIMIT, so the predicate cannot sit in the same WHERE clause and PostgreSQL 16 has no QUALIFY.
Say what the seven-day filter does to the answer: it truncates every partition, so the first row per aggregate has no predecessor inside the window and a hole spanning the boundary is invisible. Widen the window, or join to resource.version as the authority for the true maximum.
Running total: SELECT date_trunc('day', created_at) AS d, count() AS n, sum(count()) OVER (ORDER BY date_trunc('day', created_at) ROWS UNBOUNDED PRECEDING). An aggregate inside a window call is legal because grouping runs before windowing. The grouping key is unique per row here so ROWS and RANGE agree, but write the frame anyway — over ungrouped rows with tied timestamps the default RANGE frame pulls in every peer row and the total jumps.
Newest per aggregate: DISTINCT ON (aggregate_id) ... ORDER BY aggregate_id, aggregate_version DESC is the cheap PostgreSQL-only form when an index matches that order; row_number() OVER (PARTITION BY aggregate_id ORDER BY aggregate_version DESC) = 1 is the portable form and needs a subquery for the same evaluation-order reason as the gap query.
Interpret rather than report: no gaps plus a normal p95 of published_at - created_at points at the consumer; gaps or a fat lag tail point at the relay; rows still 'pending' with attempts > 0 point at neither, because they never left the database.
Be explicit that the partial index on (created_at, event_id) WHERE status = 'pending' does not serve any of these — they read published rows. Name the index a recurring monitor would need, and say why a query run twice a year may not deserve one.

Follow-up

Relay failover redelivers events. Does a duplicate break the gap query, and how would you detect one from this table alone?
Turn the gap check into a continuous monitor rather than a query someone runs after an incident. What does it watch?
The consumer claims it never received event 4,812,006. What do you look at, in what order?

How would you design a system to automatically detect compliance violations in customer support chats at scale?

Medium · ml system design and production

Approach

Separate the offline training path from the online serving path.
Name what you would monitor after launch and what triggers a retrain.
Say where features come from at serving time and how they match training.

Follow-up

How would you detect drift before the metric drops?
What happens when a feature is missing at serving time?

Describe how you would build, deploy, and monitor an LLM-based summarization feature for support agents, including handling model drift and feedback loops.

Medium · ml system design and production

Approach

Separate the offline training path from the online serving path.
Name what you would monitor after launch and what triggers a retrain.
Fix the product goal and the online metric before choosing any model.

Follow-up

How would you roll the new model out safely?
What happens when a feature is missing at serving time?

Design a scalable pipeline for processing and embedding millions of voice transcriptions for semantic search.

Medium · system design

Approach

Choose a partition key and say what query it makes expensive.
Name the read and write paths separately; they rarely have the same bottleneck.
Name the failure you are designing for, then the recovery path.

Follow-up

What would you drop to keep the system up under load?
What breaks first when traffic grows ten times?

Rebuild the search projection while it serves nine thousand queries

Medium · projections · eventual consistency · rebuild · watermark

The read-model service answers about 9k queries/second at a 120 ms p99 from a projection built off the event log, normally under 2 seconds behind. A mapping change forces a full rebuild from resource_revision, during which apply lag rises to minutes. Writes continue at 1.2k/second and events at 4k/second. Design the rebuild: how the new index is populated and cut over, how position is tracked per log partition, what the API returns alongside results so a client can tell a stale answer from a current one, and the criterion for cutting over.

Approach

Build into a second index and swap an alias rather than mutating the live one. The rebuild is then reversible by pointing the alias back, so a bad mapping costs a wasted rebuild instead of an outage. The price is peak storage for two full copies and double apply load during catch-up, and both numbers should be stated up front rather than discovered when disk fills.
Track position per log partition, not globally. Order is guaranteed only within an aggregate's partition, so progress is a vector, and the only number safe to publish is taken from the least advanced partition - the slowest one is what bounds completeness. Publishing the most advanced partition's position declares the projection current while another partition sits twenty minutes behind.
Make apply idempotent so the backfill and the live tail can overlap without a freeze. Each document records the aggregate_version it reflects, and any event at or below that version is discarded. This is why the event carries the full fact and not a delta: a delta cannot be discarded safely, and a consumer that calls back to read current state applies a state newer than the event it is processing, which is how a projection ends up with changes applied out of order.
Turn lag into a contract instead of a surprise. Return the watermark with every result set, and return the version produced by a write so the client can compare the two. A client that wrote version 7 and receives results at a watermark older than its own commit can show that its change is still landing, rather than rendering the previous value as current. Blocking the read until the projection catches up would convert a staleness problem into an availability problem at 9k queries/second, and choosing not to do that is the trade.
Define the cutover numerically. Writes are untouched by the rebuild - they commit to the primary and land in the outbox - so the only coupling is apply throughput. If catch-up applies slower than the 4k events/second arriving, it never converges. The cutover criterion is that lag is measurably decreasing and below a stated threshold, not that the backfill loop reached the end of its range.

Worked solution (30 min)

Write the rebuild sequence: snapshot source, populate the second index, attach the live tail, verify, swap alias, retire the old index - naming what is reversible at each step.
Define the per-partition position record and the single watermark derived from it, and show what each would report when one partition stalls.
Write the idempotent apply rule using the document's recorded aggregate_version, and replay one event twice against it.
State the cutover threshold in lag terms and the measurement that shows convergence rather than completion.

Expected result: A parallel index populated from resource_revision snapshots plus the live tail, cut over by an alias swap that is reversible. Position is stored per log partition and the published watermark comes from the least advanced one. Apply is idempotent on the document's recorded aggregate_version, so backfill and tail can overlap without a freeze. Results carry the watermark and writes return their version so clients can detect staleness. Cutover happens when lag is decreasing and under the stated threshold.

Verify it yourself

Applying the same event twice leaves the document unchanged.
The published watermark never advances past the least advanced partition.
Pointing the alias back restores previous behaviour with no reindex.
Catch-up rate exceeds the 4k events/second arrival rate before cutover, shown by the lag trend rather than by the loop finishing.

Follow-up

The rebuilt index disagrees with the primary tables for 300 documents. Which is authoritative, and how do you decide without freezing writes?
The rebuild doubles load on the log and pushes the projection p99 from 120 ms to 400 ms. What do you throttle, and which signal sets how much?
Clients start polling until the watermark passes their write. What does that do at 9k queries/second, and what do you offer instead?

Read latency spikes on a sixty-second sawtooth

Medium · cache stampede · single-flight · ttl · tail latency

The cached listing read path serves about 14k reads/second at an 85% hit rate. p99 sits at 35 ms for 57 seconds, jumps to 900 ms for 3, and repeats. During each spike the primary shows several hundred identical listing queries starting within the same millisecond, all carrying one large tenant's id. Cache entries use a 60-second TTL. Give the mechanism, the ordered checks, the fix, and the correctness hazard your fix must not introduce.

Approach

Match the period to a configured number before theorising about load. A spike every 60 seconds against a 60-second TTL is an entry expiring, and you confirm it by correlating spike timestamps with the entry's write time rather than with the traffic curve. If the period had matched a cron or a GC interval instead, this is a different investigation.
Establish the concurrency of the miss. Several hundred identical queries in one millisecond means the miss path has no coalescing: every request that arrives between expiry and repopulation recomputes. The herd size is that key's arrival rate times its recompute time, so at 1.2k reads/second for the hot key and a 250 ms recompute you expect about 300 concurrent misses, which matches what is observed.
Add single-flight on the miss path so one caller per key recomputes under a short-lived lock while the rest wait for its result. Prefer stale-while-revalidate where the read tolerates it: return the expired value immediately and refresh asynchronously, which removes the latency spike rather than serialising it into a queue of waiters.
De-synchronise the keys. Write TTLs with jitter, for example 60 seconds plus or minus 10%, so a deploy or a mass invalidation does not align every key on the same second and turn a per-key herd into a fleet-wide one.
Name the hazard the fix must not introduce. Serving a stale listing is acceptable only because the API reports the projection watermark, and a reader that loaded the old value before a write can repopulate the entry after the invalidation, so the bounded TTL is what actually caps staleness rather than the delete. Keep read-after-write pinned to the primary for the writing session regardless.
Verify on miss concurrency, not hit rate. The hit rate barely moves, because the herd is one miss multiplied; the number that must change is distinct origin queries per key per minute.

Follow-up

The same sawtooth appears on a key that is invalidated on write rather than expired. Is that the same bug?
How does your answer change if the recompute takes 4 seconds instead of 250 ms?
What exactly does a client see during a stale-while-revalidate window, and how does the watermark let them tell?

Suggested practice checklist

Roughly ninety minutes on weeknights with one longer weekend block. The plan cuts scope rather than compressing everything, on the assumption that one thing finished per night beats four half-started.

Day 1 — Fix the scope and take a cold baseline

Read the role description and write the three things the loop will almost certainly test, then write an explicit not-doing list and keep it visible all week.
Take one twenty-five-minute coding problem and one fifteen-minute design prompt cold, and write the single sentence naming what blocked each, because those two sentences decide where the remaining evenings go.
Set the week's rule: one thing finished every night, including the night you only have forty minutes.

Deliverable: A one-page scope with a not-doing list and two cold attempts, each carrying one sentence on what blocked it.

Day 2 — One pattern, written three times from blank

Choose the single pattern most likely to appear in your loop and write it three times from an empty file rather than editing the previous attempt.
On the third pass, write the invariant as a comment before the loop body and the complexity before the first line of code.
Stop at ninety minutes even if the third version is imperfect, and write the one thing you would fix given another hour.

Deliverable: Three independent implementations of the same pattern plus a note on what changed between them.

Day 3 — One design, only to the depth you can defend

Take one system shape and go only as far as requirements, interface and data model, refusing to draw a box you could not survive a follow-up about.
Attach one number to each non-functional requirement, deriving it rather than asserting it, and write the assumption the number rests on.
Write the one tradeoff you are choosing against and the observation that would make you reverse it.

Deliverable: One design at interface-and-schema depth with derived numbers and one written reversible tradeoff.

Day 4 — Only the fundamentals you will have to defend

Write, in under two hundred words each, the answers to the two questions that follow almost any implementation: why this structure and not the obvious alternative, and what happens to this code at a hundred times the input.
Write what an index actually costs: faster lookups on the indexed columns against a write that now maintains a second structure, plus the cases where the planner declines to use it anyway, low selectivity, or a predicate wrapping the column in a function.
Delete any answer you cannot deliver aloud in under a minute, since an answer that needs reading is not an answer you have.

Deliverable: Three written answers, each under two hundred words and each timed aloud.

Day 5 — Your own work, timed

Write a ninety-second and a four-minute version of your main project and time both aloud rather than reading them.
Prepare the two follow-ups that always come: what you would do differently, and how you knew it worked.
Put one number in the first sentence and be ready to say exactly where it came from and what it excludes.

Deliverable: Two timed narratives with one defensible number in the opening line.

Day 6 — The one full rehearsal, in the weekend block

Run a sixty-minute mock covering a coding round and a design round in one sitting with no break, because sustained attention is the thing evenings have not trained.
Immediately afterwards, and before hearing any feedback, write the three moments you lost the thread.
Spend the rest of the block only on those three moments, and on nothing you merely feel shaky about.

Deliverable: Mock notes naming three failure moments with a specific fix written under each.

Day 7 — Taper

Write the twenty-minute warm-up you will actually do on the morning: one problem you can already solve from a blank file, one design you can narrate, and nothing you have never seen.
Re-read only your own notes from this week and open no new material.
Write the logistics down: the editor or shared document you will be working in, whether execution and lookups are permitted, and the sentence you will use when you do not know something.

Deliverable: A one-page card holding the design structure, the project numbers, and the logistics.

Disagreement needs an outcome you did not choose

Conflict answers where you were right and everyone came round are the weakest ones. Stronger: the evidence you went and collected, what would have changed your mind, and what you did in the weeks after the call went against you. Implementing a design you argued against, properly, is a specific and checkable behaviour.

How do you handle a situation where a product requirement is vague, and the data available is highly unstructured or unlabeled?

Medium · behavioural and collaboration

Approach

Give the blast radius: what could have broken, and what you measured.
Name the disagreement and how you resolved it with evidence.
Pick a story where you made the decision, not one where you watched it.

Follow-up

What did you decide not to do, and why?
How did you know your change caused the improvement?

Reverse your own decision and price the reversal

Medium · reversibility · measurement · migrations

Describe a technical decision you made and later reversed. Pick one that cost something: a service you split and merged back, a cache you added and removed, an index you created that pushed the planner onto a worse plan, a projection you rebuilt from scratch. State what you believed when you decided, the measurement that changed your mind, how long the wrong version ran in production, and what the reversal cost in migrations, dual writes, and a deprecation window for callers you did not own.

Approach

State the original rationale without irony, in the version you would still defend given what was known then. If it is not defensible, the story is about carelessness rather than judgement, and a different example serves you better.
Give the measurement that moved with a before and after: the p99 that did not improve, the cache hit rate that sat at 40%, the plan that flipped to a sequential scan once the table passed a size you can name.
Cost the reversal in steps, not adjectives: expand-and-contract deploys, the dual-write window, the callers who had to be notified, the rows already written in the wrong shape that had to be backfilled or abandoned.
Distinguish reversal from rewrite by naming what you kept. Most good reversals preserve the schema or the interface and undo one decision inside it, which is also why they were affordable.
Finish on the process change: the smallest experiment that would have produced the same measurement in a day, and why you did not run it the first time.

Follow-up

What in that decision was irreversible, and did you know it was irreversible when you made it?
How did you tell the people who had already built on top of the original decision?
What do you now measure before committing to a change of this size?

Narrate an outage you owned from page to postmortem

Hard · incident response · blast radius · postmortems

Pick an incident you personally drove, ideally one where writes were affected rather than reads. In six to eight minutes: state the symptom as it first appeared on a dashboard, the blast radius you established before you knew the cause, the mitigation you applied and when, the mechanism you eventually proved, and the follow-up that would prevent a repeat. Bring numbers: error rate, tenants affected, minutes to mitigate, minutes to resolve. If you cannot name what you measured, choose a different incident.

Approach

Open on the signal rather than the cause: which metric at which percentile moved, on which service, at what time, so the listener follows the same evidence you had rather than a conclusion you already reached.
Separate mitigation from diagnosis out loud. State what you did to stop the bleeding (flag off, shed traffic, drain a lease, roll back a deploy) and say plainly that you did it before the mechanism was known, because those are two jobs with different deadlines.
Establish blast radius in countable terms: how many tenants, how many writes, and crucially whether the effect was loss or only delay. An append-only revision table or a pending outbox row means the change survived and the projection was merely behind, which is a repair rather than a data-loss incident.
Prove the mechanism instead of asserting it. Name the trace span that grew, the plan that flipped to a sequential scan, the lease that expired, plus one alternative you ruled out and the signal that stayed flat while you ruled it out.
Close on the durable fix and its cost, distinguishing what landed that week from what needed an expand-and-contract migration across several deploys, and say which of the two you actually finished.

Follow-up

What would you do differently in the first five minutes, given the same dashboard and no more information?
Which follow-up action did you deliberately not take, and why was dropping it the right call?
How did you convince yourself the mitigation was safe to apply while the cause was still unknown?