cswork / libraryOpen practice workspace

Interview guides · AHEAD

AHEAD Machine Learning Engineer Interview Questions & Guide 2026

AHEAD Machine Learning Engineer interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.

The 60-second brief

As a Machine Learning Engineer (specifically operating as an MLOps Platform Engineer) at AHEAD, you are at the forefront of enabling enterprise digital transformation. AHEAD builds robust platforms for digital business by weaving together cloud infrastructure, automation, analytics, and modern software delivery. In this role, you are the critical bridge between cutting-edge artificial intelligence and enterprise-grade reliability.

Ask whether any round happens inside an existing repository instead of a blank file. Reading unfamiliar code, isolating a fault and making the smallest correct change is a different skill from writing a function from scratch, and it needs its own practice.

Focus areas

  • Choose indexes from the query's access path
  • Make every write idempotent under retry
  • Detect concurrent edits instead of losing writes

Overview

Your primary focus will be on the Agentic Platform, where you will own the deployment, Infrastructure as Code (IaC), observability, runtime management, and cost governance across all platform layers. Unlike traditional data science roles focused purely on model training, this position requires you to build the highly scalable, observable, and cost-efficient engines that allow Large Language Models (LLMs) and autonomous agents to operate safely in production.

This role is highly strategic. The platforms you build and manage will directly impact how enterprises leverage AI. By ensuring strict environment isolation, prompt versioning, and deep LLM observability, you empower and its clients to deliver on the promise of next-generation digital transformation without compromising on security, reliability, or budget.

The AHEAD interview process

AHEAD candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

The person on this call usually cannot evaluate your code and does not need to. They write a short paragraph, and that paragraph is what a hiring manager skims when deciding who to put on your loop. So the test is not whether your work was hard, it is whether a non-engineer can repeat it correctly. Name systems by what they did rather than by their internal codename, give each project a shape (what was breaking, what you changed, what happened after), and keep the whole walkthrough near ninety seconds. Depth that cannot survive a paraphrase reads as vagueness.

What to demonstrate

  • Whether a non-engineer can restate your projects without distorting them, since their paraphrase is what travels to the hiring manager, not your sentences
  • Whether each project has a shape rather than a stack list: the failure or constraint, the change you made, the result and how it was measured
  • Whether you can say what was yours inside a team project without either inflating it or disappearing into the plural

How to prepare

  • Rewrite each headline project as two sentences with no internal system names and no acronyms outside your company, then say them to someone outside engineering and have them repeat them back. Fix whatever came back wrong
  • Attach one measured number to each project: the baseline, the change, and the window it was measured over. Where nothing was ever measured, say that plainly rather than reaching for a plausible percentage
  • Time the background walkthrough against a clock. If it runs past two minutes, compress the earliest role to a single clause and spend the recovered time on the most recent one

Technical Deep-Dive

The same problem is scored by two different mechanisms depending on the format, and preparing for one does not cover the other. With a person watching, partial progress is visible and a hint is a correction you can absorb; silence is the expensive failure, because nobody can read a half-written function. With an automated grader there is no partial credit for what you were about to do, nobody to ask, and the worked examples in the prompt are the entire specification. Read them as a contract, down to whether an empty result should be an empty list or no output at all.

What to demonstrate

  • In a live session, whether your commentary tracks what your hands are doing, and whether a hint redirects you or gets defended against
  • In an automated one, whether you cover the cases the examples do not show, since the hidden cases are where the score moves
  • Whether you manage the clock on purpose: abandoning an approach that is not converging while there is still time to write something simpler that finishes

How to prepare

  • Have someone hand you a problem and feed you one deliberately wrong hint. Practise testing it against a concrete case instead of accepting or rejecting it on authority.
  • Do one timed run a week in a plain browser editor with autocomplete, linting and your own snippets switched off, which is closer to what these environments give you
  • For the automated format, write the harness before the solution: a main that feeds the worked examples plus an empty and a single-element case and prints expected against actual, so a wrong submission is caught by you first

Virtual Onsite Loop

A day like this is several different games in a row, and the expensive mistake is carrying the previous one into the next room. Coding rewards narrow precision and finishing inside a timer. Design rewards breadth, stated assumptions and naming what you are deliberately not building. Behavioural rewards specificity about people and decisions. Candidates who over-engineer a coding problem they were supposed to finish, or who start sketching class hierarchies before anyone has agreed what the system has to do, are usually still playing the last round. Between rooms, name out loud which game the next one is.

What to demonstrate

  • Whether the coding round ends with something that runs and has been traced against a degenerate input, rather than an extensible design that was never finished
  • Whether a design discussion opens by agreeing on traffic shape, read-to-write ratio and what is allowed to be stale, instead of proceeding from an architecture you arrived with
  • Whether a behavioural answer names a person, a disagreement and what you did about it, rather than describing the system the story happened inside
  • Whether the opening habits still appear late in the day: restating the problem, asking for constraints, saying the plan before typing

How to prepare

  • Book three mocks of different types back to back on one afternoon and ask each interviewer afterwards which round you answered in the wrong mode
  • Write a three-line opening script per round type — coding: restate, name the approach and its cost, then type; design: ask for scale, read-write mix and what must not break; behavioural: name the person, the stakes and the decision — and run it off a card so the switch is mechanical rather than remembered
  • Practise coding with a timer you do not extend, stopping when it stops, so the trained reflex is to finish a correct solution rather than to keep improving one

Where candidates lose points

Running a schema change as though the lock lasts as long as the statement

In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.

Assuming an isolation level prevents the anomaly you actually have

Isolation levels are named by the SQL standard but implemented differently, so any claim about one is only true of a named engine. PostgreSQL defaults to READ COMMITTED, where every statement takes a fresh snapshot, so two statements inside one transaction can legitimately disagree about the same row. Its REPEATABLE READ is snapshot isolation: it removes non-repeatable and phantom reads but permits write skew, where two transactions each read a set, each conclude their own write is safe, both commit, and the combined result violates a constraint that no single row expresses. Only SERIALIZABLE closes that, and it closes it by aborting a transaction with a serialization failure (SQLSTATE 40001), which means the guarantee is theoretical unless the application has a retry loop. InnoDB's REPEATABLE READ is a different mechanism again - plain SELECTs read a consistent snapshot while locking reads and writes see the latest committed row - so a read-modify-write inside one transaction can act on a value that the transaction's own earlier SELECT never returned.

Not asking what the system looks like if it dies halfway through

For any multi-step write, say what state remains if the process stops between step two and step three, and what brings it back: a single transaction, a saga with compensating actions, an outbox, or a reconciliation job. Partial failure is routine at any real call volume, so 'that shouldn't happen' is an answer with nothing behind it.

Comparing floating-point values for equality, or holding money in them

Binary floating point cannot represent 0.1 exactly, so repeated addition drifts and an equality check fails on values that are mathematically equal. Store currency as integer minor units or a decimal type, and compare floats against a tolerance you chose for a stated reason.

Questions & practice

What are the key considerations when writing a Dockerfile for a Python-based machine learning application?

Medium · machine learning fundamentals

Approach

  1. State the learning problem: the label, the unit of prediction and how the model is used.
  2. Pick the metric from the cost of each error type, not from habit.
  3. Name the simplest model that could work and what would make you move past it.

Follow-up

  • How would you know the model is overfitting?
  • Where could label leakage enter this setup?

What metrics are most important to monitor when running an LLM in production?

Medium · machine learning fundamentals

Approach

  1. Say how you would validate it, and where leakage could enter the split.
  2. State the learning problem: the label, the unit of prediction and how the model is used.
  3. Pick the metric from the cost of each error type, not from habit.

Follow-up

  • What changes if the classes are heavily imbalanced?
  • How would you know the model is overfitting?

Describe your approach to managing IAM roles and policies for an EKS cluster.

Medium · machine learning fundamentals

Approach

  1. Say how you would validate it, and where leakage could enter the split.
  2. Pick the metric from the cost of each error type, not from habit.
  3. Name the simplest model that could work and what would make you move past it.

Follow-up

  • What changes if the classes are heavily imbalanced?
  • How would you know the model is overfitting?

Describe a strategy for managing and versioning different iterations of LLM prompts.

Medium · machine learning fundamentals

Approach

  1. State the learning problem: the label, the unit of prediction and how the model is used.
  2. Pick the metric from the cost of each error type, not from habit.
  3. Name the simplest model that could work and what would make you move past it.

Follow-up

  • How would you know the model is overfitting?
  • Where could label leakage enter this setup?

Canonicalise a request body into a stable idempotency fingerprint

Medium · parsing · canonicalisation · hashing

idempotency_key.request_fingerprint is a SHA-256 over the method, path and canonicalised body, and a retry whose fingerprint differs must be rejected with 422 rather than served the stored response. Write the canonicaliser. Bodies are JSON up to 256 KB nested at most 32 levels; clients vary key order, whitespace and unicode escaping, and some send 64-bit ids as JSON numbers. Produce a deterministic byte string such that semantically identical bodies match and any semantic difference does not. State your complexity and name two normalisations you refuse to perform.

Approach

  1. Parse once into a tree, then re-serialise under fixed rules: object keys sorted, array order preserved, one escaping convention, no insignificant whitespace. Parsing is O(n) and sorting keys is O(k log k) per object, so O(n log n) overall with O(depth) stack, and the 32-level cap is enforced during parsing because hostile nesting is how a canonicaliser becomes a stack overflow.
  2. Sort keys by their UTF-8 bytes and say why the obvious implementation is wrong in some runtimes: a default string comparison that orders by UTF-16 code units places surrogate pairs, meaning code points from U+10000 up, below U+E000 to U+FFFF, which is not UTF-8 byte order, so two services written in different languages disagree on the same document.
  3. Do not re-encode numbers through a double. IEEE-754 binary64 represents integers exactly only up to 2^53, so normalising a 19-digit id through a float changes it, and 1 against 1.0 cannot be reconciled without deciding whether they are the same value. Preserve the literal token, and require ids as strings at the API boundary if you want them comparable.
  4. Reject duplicate keys rather than picking one. JSON permits them and parsers disagree, most keeping the last, so any choice you make ties the fingerprint to a parser detail that the code handling the request does not necessarily share.
  5. Frame the hash preimage so concatenation cannot collide: delimit or length-prefix the method, path and body, otherwise one request's fields can be rearranged into another request with the same byte stream and the same fingerprint.
  6. Name the refusals and their consequence: no case folding, no dropping of null-valued keys, no Unicode normalisation. Each makes two different requests fingerprint alike, and the resulting failure is the worst one this table has, since the second request is answered with the first one's stored response and its effect never happens.

Worked solution (25 min)

  1. Write the serialiser: recursive emit with a depth counter, objects sorted by UTF-8 key bytes, arrays in order, strings escaped by one fixed rule, numbers emitted as their original token.
  2. Run it over three bodies: the same object with keys reordered, the same object with \u0041 written as A, and one with a nested array reversed. The first two must produce identical bytes and the third must not.
  3. Take the id 9007199254740993, round-trip it through a double, show it returns as 9007199254740992, then state the rule that prevents this.
  4. Define the hash preimage explicitly with its delimiters, and construct a pair of (path, body) inputs that would collide without them.

Expected result: A canonicaliser that is O(n log n) in body size with an enforced depth cap, sorts keys by UTF-8 byte order, preserves array order, keeps number literals verbatim, rejects duplicate keys, and feeds a delimited preimage to SHA-256, together with a stated list of normalisations deliberately not performed and the failure each would cause.

Verify it yourself

  • Key reordering and whitespace changes produce identical fingerprints while reordering an array does not.
  • A body containing an integer above 2^53 fingerprints identically after a parse and re-serialise round trip.
  • Two requests differing only in a field the endpoint ignores produce different fingerprints, and you can say why that is the safe direction to err.

Follow-up

  • A client sends the same logical request with an extra field your API ignores. Same key, different fingerprint, so you return 422. Is that the right answer?
  • Where does the fingerprint get computed relative to request decompression and the body-size limit?
  • The endpoint takes 1,000 requests per second with 256 KB bodies. What does hashing cost, and does it belong at the edge or in the core service?

Hold a per-tenant active cap against concurrent creates

Hard · write skew · isolation levels · constraints

A tenant on the standard plan may hold at most 50 resources with status='active'. The create handler runs SELECT count(*) FROM resource WHERE tenant_id = $1 AND status = 'active', compares to 50, then inserts. Two creates arrive 3 ms apart on different instances and the tenant lands at 51. Name the anomaly, say whether PostgreSQL 16 READ COMMITTED or REPEATABLE READ prevents it and why, then give an implementation that holds the cap at READ COMMITTED with the exact statements. Finally, say what changes when the cap is 'at most one running export per tenant' on job_run.

Approach

  1. Name it: write skew. The two transactions read an overlapping set and write disjoint rows, so there is no row-level conflict for the engine to detect and each commit is individually legal.
  2. Rule out the levels precisely. READ COMMITTED takes a fresh snapshot per statement and takes no lock on the counted rows, so both see 49. PostgreSQL's REPEATABLE READ is snapshot isolation: it removes non-repeatable reads and phantoms within the snapshot but still admits write skew, because the anomaly is not a re-read of a changed row, it is a read of a set that a concurrent transaction invalidates. Only SERIALIZABLE closes it, by tracking the read dependency and aborting one transaction with SQLSTATE 40001 — a guarantee that exists only if the application re-runs the whole transaction from the read.
  3. Convert the set predicate into a single-row conflict: keep tenant.active_resource_count and run UPDATE tenant SET active_resource_count = active_resource_count + 1 WHERE tenant_id = $1 AND active_resource_count < 50 in the same transaction as the INSERT. Zero affected rows is the cap, returned as 409. The row lock serialises the decision at any isolation level, and contention is bounded to one tenant's row — which is also the fair-scheduling unit, unlike a global counter that would convoy every tenant behind one row.
  4. State the cost you just took on: a counter is a second source of truth that can drift, so every path that changes status must adjust it inside the same transaction, and a periodic reconciliation has to exist, with resource_revision as the authority for what the count should have been.
  5. For the job case the invariant is expressible per row, so let the database hold it: a partial unique index on job_run (tenant_id, job_type) WHERE status IN ('queued','running') makes a second running export unwritable and the loser takes 23505, mapped to 409. That is strictly better than a counter — no drift, no reconciliation — and it is available only because the cap is one rather than fifty.
  6. Add the retry discipline each route demands: under SERIALIZABLE both 40001 and deadlock 40P01 are retryable and the retry must re-execute the read, while under READ COMMITTED with the counter nothing retries, because the conflict is reported to the caller rather than raised as an error.

Follow-up

  • A resource moves from archived back to active. Which statements change, and what breaks if the counter update and the status change land in different transactions?
  • The cap becomes plan-dependent and a plan can change mid-month. Where does the number 50 live, and who reads it?
  • How do you detect after the fact that the counter drifted, without locking the table?

Write the update path that detects a concurrent edit

Medium · optimistic concurrency · transactional outbox · audit history

resource carries version INT NOT NULL DEFAULT 1. resource_revision holds revision_id, resource_id, version, actor_user_id, change_kind, patch JSONB, request_id, created_at with UNIQUE (resource_id, version). outbox_event holds aggregate_type, aggregate_id, aggregate_version, event_type, payload, status. A PUT carries the version the client read. Write the exact statements for the single transaction that applies the edit, records the revision and enqueues 'resource.updated', and give the handler's branch on zero affected rows. Then say what PostgreSQL 16 does under READ COMMITTED when two of these updates hit one row at once.

Approach

  1. One transaction, three writes, no network call inside it: UPDATE resource SET title = $3, version = version + 1, updated_at = now() WHERE resource_id = $1 AND tenant_id = $4 AND version = $2; then INSERT the resource_revision row at version $2 + 1; then INSERT the outbox_event row at the same aggregate_version. The event goes to a table rather than a broker because no transaction spans both.
  2. Branch on the affected-row count before doing anything else. Zero has three causes — stale version, wrong tenant, row gone — so re-read once and map to 409 carrying the current version, or 404 for an id outside the caller's tenant, which also stops the endpoint confirming that another tenant's id exists.
  3. State the engine behaviour instead of assuming it. Under READ COMMITTED the second UPDATE blocks on the row lock, and when the first commits PostgreSQL re-evaluates the WHERE clause against the newly committed row, so the version predicate now fails and the statement reports zero rows. Under REPEATABLE READ the identical collision raises SQLSTATE 40001 instead, so the handler must fold both shapes into one conflict response.
  4. Keep UNIQUE (resource_id, version) even though the predicate already serialises writers. It is what makes a lost update unwritable if any other path ever reaches the revision table, and it converts a logic bug into 23505 rather than into a silently missing history row.
  5. Refuse to auto-retry the whole PUT. A retry re-reads the winner's state and reapplies an intent formed against data that no longer exists — the silent overwrite the version token was added to detect. Return the conflict; merge field-wise only if the patches are provably disjoint.
  6. Note that now() is the transaction timestamp in PostgreSQL, so resource.updated_at, the revision's created_at and the outbox row share one instant, which is what later makes reconciliation between the three tables unambiguous.

Worked solution (25 min)

  1. Write the three statements plus the rowcount branch and confirm they sit in one BEGIN/COMMIT with no outbound call between them.
  2. Run two clients that both read version 7 and apply their updates 5 ms apart; assert one 200 and one 409.
  3. Assert resource.version = 8, exactly one resource_revision row at version 8, and one outbox_event row at aggregate_version 8.
  4. Repeat at REPEATABLE READ and record the different failure shape (SQLSTATE 40001) the handler must also map to 409.
  5. Delete the version predicate and re-run: both writes commit and the first edit disappears with no error raised anywhere.

Expected result: Exactly one of the two commits. version = 8 with one revision row and one outbox row at version 8, the loser gets 409 carrying version 8, and no state exists in which the resource moved but the revision or the outbox row is missing.

Verify it yourself

  • COUNT(*) of resource_revision for that resource equals resource.version, with no gap in the version sequence.
  • Kill the process between the UPDATE and the outbox INSERT — the transaction rolls back and no event describes a change that never happened.
  • Under a 50-way race, resource.version equals the number of 2xx responses rather than the number of requests.

Follow-up

  • A client sends the version it read ten minutes ago and the resource has moved three versions. What is in your 409 so it can resolve the conflict without a full re-fetch?
  • Two editors, two disjoint fields, no overlap. Does your answer still refuse the second write, and should it?
  • Every write now touches a second hot table. How do you keep the outbox insert and its partial index from becoming the write bottleneck at 1.2k writes/second?

How do you manage Terraform state in a collaborative, multi-developer environment?

Medium · ml system design and production

Approach

  1. Name what you would monitor after launch and what triggers a retrain.
  2. Fix the product goal and the online metric before choosing any model.
  3. Say where features come from at serving time and how they match training.

Follow-up

  • How would you roll the new model out safely?
  • How would you detect drift before the metric drops?

How do you enforce security and compliance standards within your IaC templates?

Medium · ml system design and production

Approach

  1. Separate the offline training path from the online serving path.
  2. Fix the product goal and the online metric before choosing any model.
  3. Name what you would monitor after launch and what triggers a retrain.

Follow-up

  • What happens when a feature is missing at serving time?
  • How would you detect drift before the metric drops?

Explain how you would design a highly available, multi-AZ architecture for a containerized application on AWS.

Medium · system design

Approach

  1. Name the read and write paths separately; they rarely have the same bottleneck.
  2. Name the failure you are designing for, then the recovery path.
  3. State the consistency you need, and where you are willing to be stale.

Follow-up

  • How does this behave when that dependency is down for an hour?
  • What would you drop to keep the system up under load?

Compare ECS Fargate and EKS. In what scenario would you advocate for one over the other?

Medium

Approach

  1. Clarify what is being asked and what a complete answer contains.
  2. State your assumptions explicitly before working the problem.
  3. Work from the requirement backwards to the design.

Follow-up

  • How would you know your answer was wrong?
  • What assumption would you test first?

Specify the outbound webhook contract a customer endpoint can verify

Medium · webhooks · hmac · at-least-once · ordering

Egress delivery sends resource.created, resource.updated and resource.archived to roughly 40k customer endpoints at about 1.5k deliveries/second, with a per-destination concurrency cap of 4 and a 10 second connect-plus-read timeout, at-least-once from the relay. The receiver is code you do not control, often behind a proxy that may re-encode JSON. Specify the payload envelope, how the receiver verifies authenticity and rejects replays, what ordering it may assume, how it deduplicates, which status codes it should return and what each means to us, and the retry, circuit and replay behaviour.

Approach

  1. Send the fact, not a pointer: event_id, event_type, aggregate_type, aggregate_id, aggregate_version, occurred_at and the payload copied from outbox_event. A receiver told only that resource 42 changed has to call back, and the call back returns a newer state than the event describes - that is precisely how a consumer applies changes out of order while believing it is following them.
  2. Sign the exact bytes on the wire: HMAC-SHA256 over the timestamp, a separator, and the raw body, in a header, compared in constant time before the body is parsed. Instruct receivers to verify the raw bytes, never a re-serialised copy, since a proxy that reorders keys or changes whitespace invalidates a signature computed over re-encoded JSON. Reject a timestamp outside a tolerance - five minutes is typical - and keep a short seen-set of event_ids, because a capture replayed inside the tolerance still verifies.
  3. Support key rotation by sending several signature values in one header during the overlap window and publishing that any one that verifies is sufficient. Without that rule a key rotation is a synchronised outage across every destination at once.
  4. State the guarantee as it actually is: at-least-once and unordered, including within a single aggregate_id. The outbox is ordered per aggregate at the source, but the relay holds up to 4 requests in flight per destination and backs off each retry independently, so a version 6 delivery that needs one retry lands after a version 7 that succeeded on its first attempt. Publishing per-aggregate ordering we do not enforce is worse than publishing none, because a receiver that trusts it overwrites new state with old. Receivers deduplicate on event_id and apply conditionally - UPDATE ... WHERE aggregate_version < :incoming, or the same comparison under a row lock on the aggregate - which discards stale redeliveries and stays correct when two deliveries for one aggregate are processed at the same time. A duplicate and an out-of-order delivery then have the same outcome: no-op.
  5. Define the response contract in the receiver's terms. 2xx means durably accepted, and accepting means writing it down - a receiver that does its processing inline inside our 10 second timeout will be retried while it is still working. 4xx is permanent and we stop. 429, 5xx and timeouts are retried with capped exponential backoff and full jitter within the per-destination concurrency cap, and a circuit opens after a run of consecutive failures and half-opens on a schedule, so a permanently dead endpoint costs one probe per interval instead of a growing queue.
  6. Give them a recovery path that does not involve us holding a queue for everyone: a visible parked state, and a replay endpoint keyed by event_id or time range, so a receiver that was down for two hours asks for the gap on its own schedule.

Worked solution (30 min)

  1. Write the envelope fields and mark which are needed for deduplication and which for discarding a stale delivery.
  2. Write the signature string construction and the receiver's verification steps in order, including the constant-time compare and the tolerance check.
  3. Write the receiver's idempotent apply: dedupe on event_id, then the version-conditional write that discards stale and concurrently delivered redeliveries.
  4. Write the status-to-action table from our side: which codes retry, which stop, and what opens the circuit.
  5. Describe the parked state and the replay call, including what the customer must supply.

Expected result: A self-contained signed envelope, verification over raw bytes with a replay window and a seen-set, an at-least-once guarantee that states delivery is unordered even within one aggregate, a version-conditional apply, a status table that distinguishes permanent from transient, and a customer-driven replay path.

Verify it yourself

  • Delivering the same event twice leaves the receiver's state identical to delivering it once.
  • Delivering version 7 and then version 6 for one aggregate leaves version 7 applied.
  • A destination that always times out costs one probe per circuit interval rather than a retry queue.

Follow-up

  • A customer insists an event never arrived. What do you check, in order, and which table settles the argument?
  • A receiver cannot store a version and asks us to deliver their events in order. What would the relay have to give up to offer that, and what does one failing event then do to the rest of their stream?

p99 jumped on one listing filter while p50 stayed flat

Medium · query planning · composite index · percentiles · tail latency

After a release that added an owner_user_id filter to the resource listing, p99 rose from 90 ms to 1.9 s while p50 stayed at 40 ms. Traffic and row counts are unchanged. resource carries the index (tenant_id, status, updated_at DESC, resource_id DESC). The new query filters tenant_id and owner_user_id, orders by updated_at DESC, resource_id DESC, and takes 20 rows. On PostgreSQL, explain the shape of the regression, prove it from a query plan, and give the index you would add.

Approach

  1. Start from the shape. A flat p50 with a moved p99 means a subset of requests changed cost, not all of them, so the first job is naming the subset. Bucket the endpoint's latency by the tenant's row count; the natural hypothesis is that large tenants are a small share of requests and all of the tail.
  2. Get the plan for the new query on a large tenant with EXPLAIN (ANALYZE, BUFFERS). Expect an index scan over the tenant's range, a filter discarding most of it, then a Sort feeding the Limit, possibly reporting Sort Method: external merge Disk. Read actual rows on the scan node, not estimated.
  3. Explain why the existing index cannot serve it. A composite B-tree is seekable only as a left prefix, and with no equality predicate on status the scan cannot treat updated_at as an ordering, because rows in the tenant's range are ordered by status first. Everything matching must be read and sorted before LIMIT 20 can apply, so a tenant with 400,000 rows pays 400,000 rows to return 20.
  4. Add (tenant_id, owner_user_id, updated_at DESC, resource_id DESC). Equality on the first two columns leaves the index ordered by updated_at within that pair, so the plan becomes an index scan that stops after 20 rows with no Sort node. PostgreSQL can scan a B-tree backwards, so the DESC markers matter only if the two sort columns ever disagree in direction; keeping them explicit documents the order the keyset cursor depends on.
  5. Price the fix. This is a fourth index on a table taking about 1.2k writes/second, and every insert and version bump maintains it. Justify it against the query it serves, and check whether it makes an existing index redundant, which here it does not, since the original still serves the status-filtered default listing.
  6. Re-measure per tenant-size bucket rather than in aggregate. A fleet-wide p99 can improve while the largest tenant is still on the old plan.

Follow-up

  • The endpoint paginates with OFFSET. What does page 500 cost with your index, and what does the keyset version cost?
  • How would you have caught this before release, given that a 10,000-row seed database produces the same plan shape at an unnoticeable cost?
  • If a fourth index were unacceptable on write grounds, what else could serve this query?

Suggested practice checklist

Day one measures instead of guessing, under a fixed rubric, and the remaining hours are allocated in proportion to the gaps before any studying begins. The allocation is deliberately not renegotiated midweek, because the area that feels worst on day three is usually the one that is moving.

Day 1 — Diagnostic, scored before you study anything

  • Sit a 110-minute diagnostic in four blocks: forty-five minutes on two coding problems, twenty-five on one design prompt taken to interface and data model, twenty of short-answer fundamentals, and twenty delivering two behavioural answers aloud.
  • Score each block from 0 to 3 on a fixed rubric where 3 is correct and fluent, 2 is correct but slow or prompted, 1 is partially correct and 0 is stuck, grading the artifact rather than how the attempt felt.
  • Allocate days two to five in proportion to 3 minus each block's score, write the allocation down, and commit to leaving it alone.

Deliverable: A scored rubric and a fixed hour allocation for the rest of the week.

Day 2 — Largest gap: find the boundary rather than the subject

  • Split the weakest area into named sub-skills and rate each separately. For coding those are restating the problem, choosing the structure, stating the invariant, turning the invariant into loop bounds, handling empty and single-element input, and accounting for complexity out loud.
  • Attempt three items positioned just above where the rating drops off, and for each write the first move you failed to make.
  • Re-attempt one of them from blank four hours later with nothing open.

Deliverable: A sub-skill map with the two blocking sub-skills circled.

Day 3 — Drill the blocking sub-skill by repeating the shape

  • Do eight short repetitions of the same shape rather than eight different problems, so what gets practised is the pattern and not the puzzle.
  • State the rule you now hold in one sentence, then test it against a case built to break it, a sliding window over an array containing negative values, or a cache-aside read path whose invalidation message is dropped.
  • Have someone else read your one-sentence rule and find the precondition you left out.

Deliverable: One rule statement with its preconditions attached and one counterexample that would have caught the incomplete version.

Day 4 — Second gap, plus maintenance on the strongest area

  • Run the same sub-skill decomposition on the second-largest gap in half the time.
  • Spend twenty-five timed minutes on the block you scored highest, choosing the hardest item you can still finish rather than a warm-up.
  • Write whether each area fails you on recall, on setup, or on execution, and set the fix accordingly: repetition for recall, a written checklist for setup, timed work for execution.

Deliverable: A second sub-skill map plus a one-line failure diagnosis for each area.

Day 5 — The gap that is not a skill

  • Record one technical and one behavioural answer, then count two things in the playback: seconds before your first clarifying question, and sentences you began without knowing where they would end.
  • Practise saying that you do not know, followed by how you would find out, without letting it soften into a guess, and practise stating a complexity or an estimate before being asked for it.
  • Redeliver one answer under a hard ninety-second cap, which forces structure ahead of detail.

Deliverable: Two recordings with a counted reduction in time-to-first-question.

Day 6 — Retest under day-one conditions

  • Sit the same 110-minute structure with new prompts of comparable difficulty and score it on the identical rubric.
  • For any block that did not move, change the method rather than adding hours: a block stuck at 1 usually means the practice was too varied, not too short.
  • Write down which single block you would still lose the offer on.

Deliverable: A second scored rubric placed beside the first, with one named remaining risk.

Day 7 — Full loop under interview conditions

  • Run a sixty-minute mock over the two blocks that moved least, with an interviewer briefed to interrupt and change direction mid-answer.
  • Write the recovery script for going blank: restate the question, state your assumption, name the first thing you would check.
  • Say every rule from the week aloud without reading it, and cut any you cannot state in a single sentence, since a rule you have to reconstruct mid-answer will not survive an interruption.

Deliverable: A one-page card holding the recovery script and only the rules you could state from memory.

Know where your numbers came from

Bring the two or three numbers the story rests on and know how they were collected. A p99 whose timer starts inside your handler excludes the time a request spent queued, so it can sit flat while users wait longer. Give the window, the percentile and what the measurement left out, or drop the number.

How do you handle database migrations or model weight updates during a CI/CD deployment?

Medium · behavioural and collaboration

Approach

  1. Give the blast radius: what could have broken, and what you measured.
  2. Pick a story where you made the decision, not one where you watched it.
  3. Close with what you would do differently, concretely.

Follow-up

  • What did you decide not to do, and why?
  • How did you know your change caused the improvement?

Tell callers you do not own that their integration breaks

Medium · deprecation · compatibility · stakeholders

A field in a write endpoint's response must change shape. You own the endpoint; you do not own the four internal callers or the outbound webhook consumers who read it. Describe a deprecation you were responsible for: what you shipped first, how you established who was actually reading the field, the window you gave and what set its length, what you did about the consumer who never moved, and how you decided removal was safe. Name the signal you used, not the announcement you sent.

Approach

  1. Establish the reader set empirically rather than from a wiki of owners: per-field usage counters keyed by principal, or access logs attributed to a consumer. State the blind spot of whichever you pick, since a consumer that reads the field only on a monthly job will not appear in a week of logs.
  2. Ship additive first. Populate the new field alongside the old one so no reader is forced to move, which is also what keeps a rolling deploy safe, because old and new instances answer the same requests at the same time and a rollback must still find the old shape present.
  3. Set the window from the slowest legitimate consumer's release cadence, not from your calendar, and decide separately what to do for a consumer with no release process at all, such as an external webhook endpoint you can only email.
  4. Convert silence into evidence before you rely on it: a short, low-traffic removal window that makes a still-dependent consumer fail visibly and loudly while you are watching, rather than at three in the morning after you have moved on.
  5. State the removal criterion as a measurement with a duration attached, such as observed reads at zero across a full billing cycle, and keep the change reversible for one release after removal.

Follow-up

  • How would you detect a consumer that reads the field only during a monthly export?
  • One caller refuses to move and has a commercial relationship behind it. What changes in your plan and what does not?
  • After removal, what makes the change irreversible, and how long before you cross that line?

Narrate an outage you owned from page to postmortem

Hard · incident response · blast radius · postmortems

Pick an incident you personally drove, ideally one where writes were affected rather than reads. In six to eight minutes: state the symptom as it first appeared on a dashboard, the blast radius you established before you knew the cause, the mitigation you applied and when, the mechanism you eventually proved, and the follow-up that would prevent a repeat. Bring numbers: error rate, tenants affected, minutes to mitigate, minutes to resolve. If you cannot name what you measured, choose a different incident.

Approach

  1. Open on the signal rather than the cause: which metric at which percentile moved, on which service, at what time, so the listener follows the same evidence you had rather than a conclusion you already reached.
  2. Separate mitigation from diagnosis out loud. State what you did to stop the bleeding (flag off, shed traffic, drain a lease, roll back a deploy) and say plainly that you did it before the mechanism was known, because those are two jobs with different deadlines.
  3. Establish blast radius in countable terms: how many tenants, how many writes, and crucially whether the effect was loss or only delay. An append-only revision table or a pending outbox row means the change survived and the projection was merely behind, which is a repair rather than a data-loss incident.
  4. Prove the mechanism instead of asserting it. Name the trace span that grew, the plan that flipped to a sequential scan, the lease that expired, plus one alternative you ruled out and the signal that stayed flat while you ruled it out.
  5. Close on the durable fix and its cost, distinguishing what landed that week from what needed an expand-and-contract migration across several deploys, and say which of the two you actually finished.

Follow-up

  • What would you do differently in the first five minutes, given the same dashboard and no more information?
  • Which follow-up action did you deliberately not take, and why was dropping it the right call?
  • How did you convince yourself the mitigation was safe to apply while the cause was still unknown?