cswork / libraryOpen practice workspace

Interview guides · Hive (CA)

Hive (CA) Machine Learning Engineer Interview Questions & Guide 2026

Hive (CA) Machine Learning Engineer interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.

The 60-second brief

As a Machine Learning Engineer at Hive (CA), you play a critical role in building, scaling, and optimizing the core artificial intelligence systems that drive the company's platform. Your day-to-day work directly influences how massive volumes of visual and textual data are processed, analyzed, and served to enterprise customers. You will design and implement robust models, bridge the gap between experimental research and production-grade software, and solve complex machine learning challenges at scale.

Browser-facing seats are not covered by algorithm practice. Scope in state ownership, what the page does on a slow or failed request, and how you would diagnose something that renders correctly but feels slow.

Focus areas

  • Trace a symptom to a mechanism under load
  • Bound every outbound call with a timeout
  • Paginate large result sets with keyset cursors

Overview

This role sits at the intersection of high-performance computing, advanced modeling, and practical software engineering. You will collaborate closely with cross-functional teams including product managers, infrastructure engineers, and research scientists to deploy state-of-the-art models into production. The impact of your work is immediate and visible, touching high-throughput systems where latency, accuracy, and efficiency are paramount.

Expect a fast-paced, intellectually demanding environment that values technical excellence and rapid execution. Whether you are tuning deep learning architectures or optimizing inference pipelines, you will be expected to bring rigorous problem-solving skills and a deep understanding of machine learning fundamentals. Success in this position requires a balance of strong algorithmic chops, production-level coding capabilities, and the ability to navigate ambiguous technical scenarios.

The Hive (CA) interview process

Hive (CA) candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screening Call

Half of this call is the part candidates treat as small talk: start date, notice period, work authorisation and its timing, location and time zone, on-call, and the number. Those are what kill offers late, after several engineers have each spent a day. Surfacing a hard constraint now costs you nothing and occasionally buys you something, since a loop compressed to fit a competing deadline can usually only be arranged if it is asked for early. The common failure is deflecting the compensation question twice, then discovering at offer stage that the band never reached your number.

What to demonstrate

  • Whether your hard constraints are compatible with the role before a loop gets booked: earliest start, notice period, what authorisation you hold and when it needs action, days on site, willingness to carry a pager
  • Whether you give a compensation range with something behind it, such as current total compensation or a competing timeline, rather than leaving the band untested
  • Whether your stated timeline is real, since a competing deadline raised now is something scheduling can sometimes work around and the same deadline raised at offer stage usually is not

How to prepare

  • Write each constraint down in one line before the call and state them as facts rather than negotiating them live under a question you were not expecting
  • Set your range from two or three current data points for that level and location, and name the structure you are quoting in, so the number is comparable to the one they are holding
  • If another process is running, say where it stands and by when, and ask directly whether this loop can be scheduled inside that window

Technical Evaluations

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

  • Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
  • Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
  • Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
  • Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

  • For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
  • For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
  • Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Senior Leadership Evaluation

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

  • Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
  • Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
  • Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
  • Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

  • For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
  • For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
  • Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Where candidates lose points

Running a schema change as though the lock lasts as long as the statement

In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.

Assuming an isolation level prevents the anomaly you actually have

Isolation levels are named by the SQL standard but implemented differently, so any claim about one is only true of a named engine. PostgreSQL defaults to READ COMMITTED, where every statement takes a fresh snapshot, so two statements inside one transaction can legitimately disagree about the same row. Its REPEATABLE READ is snapshot isolation: it removes non-repeatable and phantom reads but permits write skew, where two transactions each read a set, each conclude their own write is safe, both commit, and the combined result violates a constraint that no single row expresses. Only SERIALIZABLE closes that, and it closes it by aborting a transaction with a serialization failure (SQLSTATE 40001), which means the guarantee is theoretical unless the application has a retry loop. InnoDB's REPEATABLE READ is a different mechanism again - plain SELECTs read a consistent snapshot while locking reads and writes see the latest committed row - so a read-modify-write inside one transaction can act on a value that the transaction's own earlier SELECT never returned.

A cache with no invalidation story

Say how an entry goes stale, how long you can serve it stale, and what happens when many requests miss the same key at the same instant. One popular key expiring under load sends every concurrent request to the origin together; single-flight coalescing, jittered expiry, or serving stale while revalidating are the standard answers.

Hardcoding to the sample inputs

Solve the stated problem rather than the two examples; special-casing a literal to make a sample pass is obvious immediately and reads as either a misunderstanding or an attempt to fake progress. If you genuinely cannot generalise yet, say which part is a stub and what would replace it.

Questions & practice

Why are you interested in building machine learning systems at Hive (CA)?

Medium · machine learning fundamentals

Approach

  1. Pick the metric from the cost of each error type, not from habit.
  2. Name the simplest model that could work and what would make you move past it.
  3. Say how you would validate it, and where leakage could enter the split.

Follow-up

  • How would you know the model is overfitting?
  • Where could label leakage enter this setup?

What would you do if a newly deployed model showed a sudden drop in accuracy in production?

Medium · machine learning fundamentals

Approach

  1. Pick the metric from the cost of each error type, not from habit.
  2. State the learning problem: the label, the unit of prediction and how the model is used.
  3. Name the simplest model that could work and what would make you move past it.

Follow-up

  • What changes if the classes are heavily imbalanced?
  • Where could label leakage enter this setup?

How would you handle class imbalance in a large-scale classification dataset?

Medium · machine learning fundamentals

Approach

  1. Pick the metric from the cost of each error type, not from habit.
  2. State the learning problem: the label, the unit of prediction and how the model is used.
  3. Name the simplest model that could work and what would make you move past it.

Follow-up

  • What changes if the classes are heavily imbalanced?
  • How would you know the model is overfitting?

What storage and retrieval strategies would you use for billions of embedding vectors?

Medium · machine learning fundamentals

Approach

  1. Say how you would validate it, and where leakage could enter the split.
  2. Pick the metric from the cost of each error type, not from habit.
  3. Name the simplest model that could work and what would make you move past it.

Follow-up

  • What changes if the classes are heavily imbalanced?
  • Where could label leakage enter this setup?

Write a function to serialize and deserialize an N-ary tree.

Medium · coding and algorithms

Approach

  1. State the target complexity and say which constraint rules the naive version out.
  2. Walk one small example through your approach before writing the whole thing.
  3. Choose the data structure from the access pattern, not from familiarity.

Follow-up

  • How does this change if the input no longer fits in memory?
  • Which test case would catch an off-by-one here?

Given a matrix, implement an algorithm to find the longest path with specific constraints using dynamic programming.

Medium · coding and algorithms

Approach

  1. Restate the input: its shape, its size, and what is guaranteed about it.
  2. Walk one small example through your approach before writing the whole thing.
  3. Choose the data structure from the access pattern, not from familiarity.

Follow-up

  • How does this change if the input no longer fits in memory?
  • What is the worst case, and how likely is it on real data?

Merge partitioned event streams into one ordered feed with bounded lateness

Hard · k-way merge · watermarks · out-of-order streams

The read-model service consumes 64 log partitions carrying about 4,000 events per second in total. Each partition is ordered within itself, but partitions drift by up to 30 seconds, and the activity feed must present a tenant's events in occurred_at order. Produce the merge. State its complexity, the buffer it requires in events and in bytes, what happens when one partition is idle, and what you do with an event that arrives after you have already emitted its position. Payloads average 1 KB.

Approach

  1. Merge with a min-heap over the 64 partition heads keyed on (occurred_at, event_id): O(log P) per event and O(n log P) overall. The tie-break on event_id is what makes the output deterministic when two partitions carry the same millisecond, which matters because the feed is paginated and a non-deterministic order reorders pages under the reader.
  2. Emitting the heap head is only correct once every partition has produced everything up to that timestamp, so the emit condition is a watermark: the minimum across partitions of the highest occurred_at seen, less the allowed lateness. Events are held until the watermark passes them, which is what turns individually ordered streams into a jointly ordered one.
  3. Size the buffer from the lateness rather than guessing: 4,000 events per second times 30 seconds is 120,000 buffered events, and at 1 KB each about 120 MB of heap. That number is the real price of the ordering guarantee and belongs in front of whoever asked for it.
  4. Handle the idle partition explicitly, because it fails the feed rather than corrupting it: a partition with no traffic never advances its own maximum, so the watermark freezes and output stops entirely. Either every partition emits a periodic idle marker carrying the broker's current time, or the watermark falls back to wall clock for a partition silent beyond a threshold.
  5. Choose the late-event policy from what the projection is keyed on. The projection upserts on (aggregate_id, aggregate_version) and discards a version it has already applied, so a late event is safe to apply out of order and correctness never depended on the merge at all. Apply it, recompute the affected feed page, and count lateness so the 30-second budget can be re-derived from data rather than folklore.
  6. Say what the merge does not buy: ordering is guaranteed within one aggregate by the log's partitioning, and no watermark makes the cross-aggregate order authoritative. Two events from different aggregates in the same millisecond have no true order, so the feed's order is a presentation choice that must be stable rather than correct.

Worked solution (35 min)

  1. Write the heap comparator on (occurred_at, event_id) and the per-partition head refill.
  2. Write the watermark computation and the emit-loop condition, then list which buffered events are held at a chosen instant.
  3. Compute the buffer at 4,000 events per second, 30 seconds and 1 KB per event, and state what fraction of a worker's heap that represents.
  4. Add the idle-partition marker and trace the watermark with one silent partition, both with and without the marker.
  5. Write the late-event path and name the key that makes applying it safe.

Expected result: A 64-way min-heap merge at O(n log P) with a deterministic (occurred_at, event_id) comparator, a watermark of the per-partition minimum less 30 seconds gating emission, a stated buffer of 120,000 events and roughly 120 MB, idle markers so a silent partition cannot freeze the watermark, and a late-event policy justified by the projection's idempotency on (aggregate_id, aggregate_version).

Verify it yourself

  • Feeding the partitions in an adversarial interleaving produces the same output sequence as feeding them in timestamp order.
  • Silencing one partition stalls output without the idle marker and does not stall it with the marker.
  • Applying a late event twice leaves the projection byte-identical, which is what permits the out-of-order path to exist.

Follow-up

  • The lateness budget is raised to five minutes. What is the new buffer, and what besides memory changes?
  • The consumer restarts. Where does it resume from, and what does the feed look like for the first 30 seconds?
  • One partition is ten minutes behind because its producer is slow. Do you stall the feed or emit without it?

Stop tag and share joins from fanning out a page

Medium · join fan-out · semi-joins · many-to-many

resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.

Approach

  1. Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
  2. Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
  3. Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
  4. Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.
  5. Attach the tags for display after the page has been cut — LEFT JOIN LATERAL (SELECT array_agg(rt.tag_id) FROM resource_tag rt WHERE rt.resource_id = p.resource_id) ON TRUE over the 50 returned rows. Aggregate over the page, never over the tenant.
  6. Index both directions and say which query each serves: PK (resource_id, tag_id) serves the lateral lookup, (tag_id, resource_id) serves the EXISTS probe by tag, and resource_share needs (shared_with_user_id, resource_id) for the same reason. An index covering one direction only leaves the other as a scan.

Worked solution (30 min)

  1. Build a tenant where each resource carries 0-5 tags from a 20-tag vocabulary and is shared with 0-4 distinct users, then bind $2 to three tags and $3 to a user holding shares on about half the resources. Run the joined query and compare its row count to the distinct resource count on page one.
  2. Run COUNT(*) on the joined shape and on the EXISTS shape and compare both to a ground truth computed from distinct ids; then give $3 a second permission row on 10% of resources and record which of the two counts moves.
  3. EXPLAIN both page queries and compare rows-read plus the presence of a Sort or HashAggregate node above the join.
  4. Add (tag_id, resource_id), re-run the EXISTS probe, and record the plan change on the inner side.

Expected result: The EXISTS page returns exactly 50 distinct resources and reads about 50 driving rows with no aggregate above the join. The EXISTS count matches the ground truth. The joined COUNT(*) is larger by roughly the average number of matching tags per qualifying resource — the tags it holds from $2, not its total tag count — times its share rows for $3, which is 1 until the duplicate permission rows exist and rises with them afterwards. The EXISTS count is unchanged by those duplicates.

Verify it yourself

  • The page's resource_id list has no duplicates and is 50 long whenever at least 50 rows match.
  • count(*) from the rewrite equals count(DISTINCT r.resource_id) over the joined shape.
  • EXPLAIN shows a semi-join node on the inner side rather than a full join followed by a unique or aggregate step.

Follow-up

  • The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
  • A resource can be shared with the same user twice under different permissions. Does your count change, and should it?
  • Where does the correct total come from when the tenant holds 4M resources and the header must not cost 200 ms?

Replace offset paging on the resource feed with keyset

Easy · keyset pagination · composite index · sort order

resource holds resource_id, tenant_id, owner_user_id, title, body_ref, version, status ('draft','active','archived','deleted'), created_at, updated_at, deleted_at, with an index on (tenant_id, status, updated_at DESC, resource_id DESC). The listing endpoint returns active resources for one tenant, newest update first, 50 per page, today with LIMIT 50 OFFSET n. Tenants reach page 400 and rows are created while they read. Write the keyset query, define what the cursor carries and how it is encoded, and say which part of the index each predicate uses. Assume PostgreSQL 16.

Approach

  1. Name the two failures separately. OFFSET 20000 makes the server produce and discard 20,000 rows, so page cost grows with depth rather than with page size. Independently, any write that changes how many rows sort above the offset moves the window between two fetches, and the direction decides which anomaly you get: an insert lands at the head of updated_at DESC and pushes already-returned rows down past the boundary, so they are returned a second time; a delete above the offset, or a row whose updated_at is bumped above the cursor, pulls rows up and one is never returned at all. Nothing in the response reveals either.
  2. Write the seek: WHERE tenant_id = $1 AND status = 'active' AND (updated_at, resource_id) < ($2, $3) ORDER BY updated_at DESC, resource_id DESC LIMIT 50. The row-value comparison is one index range rather than a disjunction, and both columns are NOT NULL, which is what makes that comparison well defined.
  3. Map each predicate onto the index: tenant_id and status are equality on the leading columns, (updated_at, resource_id) is the range, and the ORDER BY matches the index order so no Sort node appears and the scan stops after 50 rows. The DESC in the definition only matters for mixed directions — a plain ascending btree on the same columns is read backwards for this query.
  4. Put both sort columns in the cursor and nothing the client can tamper with into another tenant: base64 of (updated_at, resource_id), validated server-side, with tenant_id taken from the principal.
  5. State the residual honestly. Keyset is stable against concurrent inserts and deletes, but not against a row whose updated_at changes mid-scroll — that row moves in the ordering and can be seen twice. If the feed must be a snapshot, order by an immutable key or bound the page set with updated_at <= the cursor's start value.
  6. Keep a total out of the page path. A tenant-wide COUNT(*) is the scan keyset just removed; fetch LIMIT 51 and return has_more instead.

Follow-up

  • The client asks for 'jump to page 400'. What do you offer instead, and what does the honest version cost?
  • Sort order becomes user-selectable across four columns. How many indexes is that, and which would you refuse to add?
  • What does the cursor do when the row it points at has since been deleted?

Walk through how you would scale a model inference service to handle sudden traffic spikes.

Medium · ml system design and production

Approach

  1. Say where features come from at serving time and how they match training.
  2. Fix the product goal and the online metric before choosing any model.
  3. Name what you would monitor after launch and what triggers a retrain.

Follow-up

  • How would you detect drift before the metric drops?
  • What happens when a feature is missing at serving time?

How would you design an automated evaluation pipeline for continuous model deployment?

Medium · ml system design and production

Approach

  1. Fix the product goal and the online metric before choosing any model.
  2. Name what you would monitor after launch and what triggers a retrain.
  3. Say where features come from at serving time and how they match training.

Follow-up

  • How would you detect drift before the metric drops?
  • How would you roll the new model out safely?

How would you architect a distributed training setup for a massive language model?

Medium · system design

Approach

  1. Name the read and write paths separately; they rarely have the same bottleneck.
  2. Choose a partition key and say what query it makes expensive.
  3. Name the failure you are designing for, then the recovery path.

Follow-up

  • What would you drop to keep the system up under load?
  • How does this behave when that dependency is down for an hour?

Walk me through your resume and highlight a project where you solved a difficult ML scaling problem.

Medium

Approach

  1. Say what you would check first and why it is the highest-information step.
  2. Work from the requirement backwards to the design.
  3. Clarify what is being asked and what a complete answer contains.

Follow-up

  • What assumption would you test first?
  • How would you know your answer was wrong?

Design the bulk write endpoint a migration script retries blindly

Medium · batch api · partial failure · idempotency

A customer's migration script pushes 2 million resources through POST /v1/resources:batch, up to 500 items per call, and retries any call that errors or times out. Within a call some items fail validation, some collide with rows that already exist, and some succeed. Specify the request and response shape, whether a batch is atomic or per-item, how idempotency works for the call and for each item, the status code for a mixed outcome, the size and item-count limits with their error codes, and exactly what the script does after a timeout mid-batch.

Approach

  1. Choose atomicity deliberately and price it. All-or-nothing means one transaction holding locks for the whole batch on a primary already absorbing about 1.2k writes/second, which bounds batch size by lock duration, and it turns one bad row into 499 rejections the script must resend. Per-item partial success is the right default for a migration, and the contract's job is then to make a partial outcome impossible to miss.
  2. Require a client-supplied id on every item and echo it in every result. Deriving a per-item key from the array index breaks the first time the script resends a batch with the failures removed: the indices shift, previously-succeeded items acquire new keys, and they are created a second time.
  3. Key the effects at two levels. The call's Idempotency-Key covers an exact resend of the same bytes; per-item keys of (tenant_id, client_item_id) make a partially-applied batch safe to resend whole. Resending an identical batch must reproduce the same per-item results, not 500 conflicts the script has to interpret.
  4. Answer a mixed outcome with one status plus per-item detail: 200, or 207 borrowed from WebDAV if you prefer it - document whichever you pick - carrying an array of client_item_id, per-item status, and either resource_id or an error code from the same taxonomy the single-item endpoint uses. Reserve 4xx for the request as a whole: unparseable body, too many items, payload over the limit (413). Put a failed count at the top level so that even a script checking only the cheapest thing cannot conclude success while rows were dropped.
  5. Cap the request before doing any of it: item count, total bytes, and concurrent batches per tenant, since one tenant's migration otherwise consumes write capacity everyone shares. Anything that cannot finish inside the request deadline belongs on job_run behind a 202, not in a synchronous call that will time out halfway.
  6. Script behaviour after a timeout: the outcome is unknown and no results were received, so resend the identical batch with the same keys and read the results. Never resend 'only the items I have no result for' - a timeout yields no results at all, and that rule silently means resend everything anyway.

Worked solution (30 min)

  1. Write the request schema with the per-item client id, and the response schema with per-item status and a top-level failed count.
  2. Write the atomicity decision and the sentence of justification that names the lock cost or the resend cost.
  3. Define both key levels and trace a resend of a half-applied batch through them, item by item.
  4. List the whole-request rejections with their status codes and limits.
  5. Write the script's timeout rule and the pacing it should apply between calls.

Expected result: A per-item batch with client-supplied item ids, per-item idempotency plus a call-level key, a mixed-outcome response that a status-only check cannot misread, explicit request caps, and a resend rule that is correct when no results were received.

Verify it yourself

  • Resending an identical batch twice leaves exactly the same set of resource rows as sending it once.
  • Resending a batch with the successful items removed does not create duplicates of anything.
  • A caller that reads only the top-level fields can still tell that some items failed.

Follow-up

  • At 500 items per call, what is the wall-clock time for 2 million rows, and what pacing do you publish so the migration does not become an incident?
  • One item in every batch fails with the same code. How does the script discover that without a human reading logs?

Listing latency scales with page size, not with filters

Easy · n+1 · latency · query batching

The tenant listing endpoint reads resource filtered by tenant_id and status, ordered by updated_at DESC, and returns each row plus the owner's display name from app_user and the actor of that resource's latest resource_revision. p99 is 55 ms at 10 rows per page and 1.4 s at 200. Database telemetry shows 401 statements per request, each under 1 ms, and nothing in the slow-query log. Diagnose the cause and give the fix, stating the statement count per request and the p99 you expect afterwards.

Approach

  1. Read the counters before forming a theory. 401 statements for 200 rows is one driver query plus two per row, and sub-millisecond execution with an empty slow-query log rules out a bad plan. The time is round trips, which is why it is invisible in every per-query metric and scales with rows returned rather than with filter selectivity.
  2. Name the two per-row statements from their normalised text: a single-row app_user lookup by user_id, and a resource_revision lookup by resource_id ordered by version DESC LIMIT 1. Confirm by dropping those two response fields and watching the statement count fall to one. That locates the calls in the serialisation layer, not the repository.
  3. Check that the arithmetic accounts for the whole gap. Measure one round trip to the replica in isolation; 400 trips at roughly 3 ms of network plus 0.2 ms of execution is about 1.3 s on top of a 55 ms baseline, which matches. If the multiplication had fallen short, the N+1 would only be part of the story and you would keep looking.
  4. Batch both lookups. Collect owner_user_ids and resource_ids from the driver query, then issue WHERE tenant_id = $1 AND user_id = ANY($2) for the users, and PostgreSQL's SELECT DISTINCT ON (resource_id) ... WHERE resource_id = ANY($2) ORDER BY resource_id, version DESC for the latest revision, which the UNIQUE (resource_id, version) index serves directly. On an engine without DISTINCT ON, use a lateral join or a row_number window. Three statements per request at any page size.
  5. Keep the tenant predicate in the batched query. The per-row version was implicitly scoped because its ids came from tenant-scoped rows; a batched user_id = ANY(...) with no tenant_id is an unscoped read that behaves correctly only as long as the id list is trustworthy.
  6. Re-measure at 10, 50 and 200 rows and confirm the statement count is constant. Latency should now track bytes returned.

Follow-up

  • The page size is capped at 200 today. What breaks first if it is raised to 2,000, and is it still this bug?
  • How do you stop the next N+1 from reaching production, given that no individual query is slow and the endpoint's tests pass?
  • The latest-revision actor is only used to render an avatar. Make the case for denormalising it onto resource, and name the write anomaly that introduces.

Suggested practice checklist

Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.

Day 1 — Coding, one pass at shallow depth

  • Solve one problem from each of six families, an array with two pointers, hash counting, binary search, a tree traversal, a graph traversal and one dynamic program, under a hard twenty-minute cap with no extensions, marking each finished, late, or stalled.
  • For every stall, write the exact move you could not make rather than the subject, so the note reads could not turn the recurrence into a loop rather than bad at dynamic programming.
  • Fix nothing today. The value of the pass is the unfixed record.

Deliverable: Six timed attempts marked finished, late or stalled, each stall carrying a named blocking move.

Day 2 — Design, one pass at shallow depth

  • Spend twenty minutes each on three different shapes, a read-heavy feed, a write-heavy ingest path, and something needing a transaction across two entities, stopping each at requirements, interface and data model.
  • After each, write the first question you could not answer, which is usually a number you could not estimate or a failure mode you had no vocabulary for.
  • Mark which of the three you would be most relieved not to be asked, and treat that as data rather than as a preference.

Deliverable: Three shallow designs, each with the first unanswerable question written at the bottom.

Day 3 — Fundamentals and the practical rounds

  • Answer eight short questions in writing at four minutes each, covering the material that fills the gaps between the big rounds: what happens between a URL and a rendered page, what an index costs on write, when a process is preferable to a thread, and what conditions a deadlock requires.
  • Do one thirty-minute practical task of the kind a take-home compresses: read an unfamiliar two-hundred-line file and write what it does, what you would change, and the one thing you remain unsure of.
  • Score every answer fluent, correct but slow, or absent, and keep the absent ones visible.

Deliverable: Eight scored short answers and one written reading of unfamiliar code.

Day 4 — The rounds that are about you, and the map

  • Deliver three behavioural answers aloud against a timer, a conflict, a failure you owned, and a decision made without enough information, marking any that ran past three minutes or contained no number.
  • Assemble the map: every marked item from days one to three on a single page, sorted by how likely it is to appear in your loop rather than by how uncomfortable it felt.
  • Choose exactly two areas for the remaining three days and write down what you are deliberately abandoning.

Deliverable: A one-page scored map of the whole surface area with two areas chosen and the rest explicitly abandoned.

Day 5 — First chosen area, to the depth you skipped

  • Work the higher-ranked area in four focused blocks, choosing items one level above where you stalled rather than repeating what already works.
  • After each block write the rule you extracted in one sentence with its precondition attached, since a rule carrying no precondition is exactly what fails under a variation.
  • Re-attempt the day-one or day-two item that exposed this area and compare against the original timing.

Deliverable: Four worked blocks, a timed re-attempt against the original, and three one-sentence rules with preconditions.

Day 6 — Second chosen area, where the gap is coverage rather than speed

  • Treat the second area differently from the first. Day five drilled something you could already half-do; this one is usually a topic you had simply never met, so build one worked reference example end to end and keep it, rather than attempting six problems badly.
  • Write down the vocabulary you were missing on day two or three, five terms at most, each with the one sentence that makes it usable in an answer rather than the textbook definition.
  • Redo the shallow attempt that exposed this area and note whether you now fail later in the problem, because moving the failure point is the realistic gain from a single day and is worth more than a score that did not change.

Deliverable: One worked reference example for the newly covered area, a five-term vocabulary list, and a note on where the failure point moved.

Day 7 — Reassemble the loop

  • Sit two rounds back to back with no gap, ordering them so the area you chose second comes last, because the map was built from rested, isolated attempts and the loop will reach your weaker area when you are already spent.
  • Write where the second round suffered from the first, which is normally the point at which structure collapses into narration.
  • Reduce the week to one page holding only the rules you can state without reading them.

Deliverable: Mock notes on cross-round carryover plus a one-page card of rules you can recite from memory.

Bounding a vague ticket before writing code

When the requirements were thin, the interesting part is how you fenced the problem off: the assumption you wrote down, who you got to confirm it, the narrow version you shipped first so the rest stayed cheap to change. Guessing and being right is luck. Guessing in writing, where someone could correct you, is method.

Ship under a deadline and bound the debt you chose

Medium · pagination · technical debt · tradeoffs

You have four days to ship a tenant-facing listing endpoint. The version you would defend uses keyset pagination over (tenant_id, status, updated_at DESC, resource_id DESC); the version you can finish uses LIMIT/OFFSET with no matching index. Describe a deadline call you actually made of this shape: what you shipped, what you knowingly deferred, how you bounded the damage with a mechanism rather than an intention, and the specific numeric condition that would force the follow-up. Name who you told and where you wrote it down.

Approach

  1. Name the deferred failure precisely instead of calling it slow. OFFSET n makes the database produce and discard n rows, so cost grows with page depth; without an index matching the sort, every matching row is read and sorted before the limit applies; and rows inserted between two page fetches shift across the boundary so items are skipped or repeated with nothing in the response to signal it.
  2. Bound the blast radius with something mechanical rather than a promise: cap maximum page depth, cap page size, restrict the endpoint to one internal caller, or keep it behind a flag. State which failure each cap removes and which it leaves standing.
  3. Attach a number to the trigger and wire it to an alarm: the first tenant crossing N resources, or the endpoint's p99 crossing its share of the 400 ms budget, so the debt announces itself instead of waiting to be remembered.
  4. Write it where the next engineer looks, which is the code and the ticket, not a chat message: what was deferred, why, the cap, and the trigger.
  5. Report what actually happened in your real example, including the case where the trigger never fired and the debt was correctly never repaid.

Follow-up

  • At what page depth does the offset version breach your latency budget, given your page size and row counts?
  • What breaks first when you switch to keyset pagination later, and what does a client holding an old page token see?
  • Who would have overruled you if you had asked for two more days, and did you ask?

Reverse your own decision and price the reversal

Medium · reversibility · measurement · migrations

Describe a technical decision you made and later reversed. Pick one that cost something: a service you split and merged back, a cache you added and removed, an index you created that pushed the planner onto a worse plan, a projection you rebuilt from scratch. State what you believed when you decided, the measurement that changed your mind, how long the wrong version ran in production, and what the reversal cost in migrations, dual writes, and a deprecation window for callers you did not own.

Approach

  1. State the original rationale without irony, in the version you would still defend given what was known then. If it is not defensible, the story is about carelessness rather than judgement, and a different example serves you better.
  2. Give the measurement that moved with a before and after: the p99 that did not improve, the cache hit rate that sat at 40%, the plan that flipped to a sequential scan once the table passed a size you can name.
  3. Cost the reversal in steps, not adjectives: expand-and-contract deploys, the dual-write window, the callers who had to be notified, the rows already written in the wrong shape that had to be backfilled or abandoned.
  4. Distinguish reversal from rewrite by naming what you kept. Most good reversals preserve the schema or the interface and undo one decision inside it, which is also why they were affordable.
  5. Finish on the process change: the smallest experiment that would have produced the same measurement in a day, and why you did not run it the first time.

Follow-up

  • What in that decision was irreversible, and did you know it was irreversible when you made it?
  • How did you tell the people who had already built on top of the original decision?
  • What do you now measure before committing to a change of this size?

Estimate work you have never done and defend the range

Hard · estimation · backfills · expand-contract

You are asked to estimate a change you have never attempted: add a column to a 100-million-row table, populate it, move reads across, and drop the old shape. Give a range with the assumptions that generate it, including batch size, the signal your backfill throttles on, and wall-clock hours, and name the three unknowns that would move the number most. Then describe a real estimate you gave under comparable ignorance: how you expressed its uncertainty, what you committed to, and how wrong you turned out to be.

Approach

  1. Decompose into independently deployable steps before estimating anything: add the column nullable, write both shapes, backfill in batches, verify, move reads, stop writing the old shape, drop it. That is four deploys spread over days, and the calendar estimate is dominated by them rather than by the loop's runtime.
  2. Do the arithmetic aloud for the part that has arithmetic in it: batch size times number of batches times per-batch duration, at a write rate the primary can absorb alongside roughly 1.2k writes per second of production traffic. The loop is throttled by replication lag and lock waits, not by how fast it can issue statements.
  3. Price the schema step by its lock rather than its statement duration. In PostgreSQL an ALTER TABLE taking ACCESS EXCLUSIVE waits for every open transaction on that table while later queries queue behind it, so a millisecond change issued during a thirty-second analytics query stalls that table for thirty seconds. Adding a nullable column with a non-volatile default avoids a rewrite from version 11; a new index wants CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block and leaves an invalid index behind if it fails.
  4. Express the answer as a range whose endpoints each trace to a stated assumption, then name the cheapest experiment that collapses it, which is almost always running one real batch against the real table and multiplying.
  5. Commit to a checkpoint rather than a completion date: the day you report a measured number from that first batch. That is a promise you can keep under uncertainty, and it is what the asker actually needs in order to plan.

Follow-up

  • How do you verify the backfill genuinely finished, given rows written by production traffic while it ran?
  • Where does the backfill resume from after a worker is killed mid-batch, and what makes that resume point trustworthy?
  • Your first batch comes back ten times slower than assumed. What do you tell the person waiting on the estimate, and when?