Interview guides · AI Camp
AI Camp Machine Learning Engineer Interview Questions & Guide 2026
AI Camp Machine Learning Engineer interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.
The 60-second brief
As a Machine Learning Engineer at AI Camp, you are stepping into a highly impactful role at the intersection of artificial intelligence, product development, and education. AI Camp is dedicated to democratizing AI by building tools, platforms, and real-world projects that empower the next generation of technologists. In this position, you are not just optimizing algorithms in a vacuum; you are building the core infrastructure and intelligent features that make AI accessible, understandable, and highly functional for users across varying skill levels.
Browser-facing seats are not covered by algorithm practice. Scope in state ownership, what the page does on a slow or failed request, and how you would diagnose something that renders correctly but feels slow.
Focus areas
- Make every write idempotent under retry
- Bound every outbound call with a timeout
- Detect concurrent edits instead of losing writes
Overview
Your work will directly influence how products are built and how AI concepts are deployed in practical, real-world scenarios. You will collaborate closely with product managers, educators, and software engineers to design models that are both robust and scalable. Whether you are developing internal tooling to streamline data processing, creating generative AI features for educational platforms, or deploying computer vision models for industry partners, your contributions will have a visible and immediate impact on the business.
What makes this role uniquely exciting is the blend of technical rigor and human-centric design. You will face the challenge of taking complex, state-of-the-art machine learning concepts and translating them into reliable, production-ready systems. AI Camp values engineers who can think big about the future of AI while maintaining a pragmatic, hands-on approach to solving today's engineering bottlenecks.
The AI Camp interview process
AI Camp candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.
Recruiter Screen
Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.
What to demonstrate
- Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
- Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
- Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled
How to prepare
- Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
- Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
- Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market
Technical Assessment
Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.
What to demonstrate
- Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
- Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
- Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
- Whether a failing case is isolated and explained before any edit is made to the code
How to prepare
- From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
- Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
- Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.
Technical Discussions
Most of the time lost in this format is not lost to thinking. It goes to a standard-library call you half-remember, an off-by-one in a loop bound, and a debugging loop that mutates code at random until something passes. When output is wrong, stop re-reading the whole function: take the smallest input that reproduces it and walk the state through by hand, printing intermediates if the environment allows. Guessing at a fix without a failing case you understand is how a five-minute bug becomes twenty, and the clock does not pause while you do it.
What to demonstrate
- Whether you reach the right structure without a detour, and can write it from memory rather than only recall that one exists
- Whether overflow is considered where the language has fixed-width integers, since a signed 32-bit value stops at 2,147,483,647 and then wraps in Java, is undefined behaviour in C++, and does not arise in Python, whose integers grow instead
- Whether recursion depth is treated as a constraint on large inputs, given that CPython's default limit is 1000 frames and a deep recursion can exhaust the stack in any language where an iterative version would not
- Whether a failing case is isolated and explained before any edit is made to the code
How to prepare
- From an empty file and with no references open, implement the pieces you lean on most: a heap push and pop, an iterative DFS with an explicit stack, and a binary search whose midpoint is written lo + (hi - lo) / 2, which avoids the overflow that (lo + hi) / 2 can hit in a fixed-width integer type
- Time yourself on the ten library calls you look up most, such as sorting with a custom comparator, splitting and joining strings, and finding the next key at or above a value in an ordered map, until the lookup is gone
- Take a solution you know is broken and, before touching it, write one sentence naming the input, the expected value and the actual value. Repeat until you do it without deciding to.
Behavioral Interviews
This round is deciding whether a change you make without supervision can be allowed to reach production. It is scored on what you knew at the moment you decided, not on how it turned out, so a story that opens with the result and works backwards reads as luck retold as judgement. Say what the options were, what you did not know, what you did to shrink the unknown before committing, and what you accepted as the worst plausible case. The detail that separates answers is a bound: how many users, how much data, and for how long, if you had been wrong.
What to demonstrate
- Whether the reasoning you give was available at the time you decided rather than after the result came in, since a story whose deciding evidence arrived later describes an outcome and not a judgement
- Whether you can put units on the exposure (users, rows, minutes of degraded service) and whether the containment you chose actually bounded it: a canary bounds the request path it fronts, while a background job writing to a shared table reaches every user regardless of which version served their requests
- Whether the reversal path existed before you shipped or was improvised during the incident, and whether it restores state or only stops further damage
How to prepare
- For your three largest changes, write down the one thing you would have had to be wrong about for it to fail, and what your best estimate of it was on the day you shipped. If you never held an estimate, that is the gap the follow-up questions will find
- Write the undo procedure for one of those changes as it existed at the time, then mark which steps restore data and which only stop new damage. Turning a flag off or reverting a deploy ends the new writes; rows already written come back only from a copy you kept, and a dropped column comes back empty unless something outside the schema holds the values
- Rehearse one story from the decision point forward and stop before the outcome, then have someone ask what you would do next. If the story only works with the ending attached, it is an anecdote rather than a decision you can defend
Where candidates lose points
Letting a slow dependency consume unbounded concurrency
The failure that takes a service down is usually not an error but a delay. A dependency answering in thirty seconds instead of fifty milliseconds holds each request's worker or connection six hundred times longer, and since required concurrency is arrival rate times latency, a fleet sized for sixty in-flight requests now needs thirty-six thousand to sustain the same rate - so it queues, and requests whose clients have already abandoned them still occupy resources. Retries make it precisely worse: a policy of three attempts triples the load on a dependency at the exact moment it is least able to serve, which is how one slow dependency becomes an outage of everything sharing that pool. Containment is four specific things - a timeout on every outbound call shorter than the caller's remaining budget, a bounded pool per dependency so one cannot starve the others, backoff with full jitter rather than a fixed delay so retries do not resynchronise, and a circuit that stops sending once the failure rate makes an attempt pointless.
Paginating with LIMIT/OFFSET over a set that changes while the client is reading it
OFFSET n makes the database produce and discard n rows before returning anything, so the cost of a page grows with its depth rather than with its size and page 500 costs five hundred pages of work. The correctness problem is worse than the cost: if a row is inserted or reordered between two page fetches, rows shift across the offset boundary and are either skipped entirely or returned twice, and neither outcome leaves any trace in the response for the client to detect. Keyset pagination - WHERE (sort_key, id) < ($last_sort_key, $last_id) ORDER BY sort_key DESC, id DESC LIMIT n, backed by an index in exactly that order - reads only the rows it returns and is stable against concurrent inserts. It requires the tie-break column: a timestamp is not unique, and duplicate sort keys straddling a page boundary reintroduce the skip it was adopted to remove.
A cache with no invalidation story
Say how an entry goes stale, how long you can serve it stale, and what happens when many requests miss the same key at the same instant. One popular key expiring under load sends every concurrent request to the origin together; single-flight coalescing, jittered expiry, or serving stale while revalidating are the standard answers.
Optimising an axis nobody named
Ask which resource is actually scarce here: wall-clock latency, throughput, memory footprint, cost per request, or engineering time. Shaving a constant factor off an in-memory step is wasted effort when the same function makes a blocking remote call inside the loop.
Questions & practice
How do you stay updated with the rapidly changing landscape of AI and machine learning?
Medium · machine learning fundamentals
Approach
- Pick the metric from the cost of each error type, not from habit.
- State the learning problem: the label, the unit of prediction and how the model is used.
- Name the simplest model that could work and what would make you move past it.
Follow-up
- What changes if the classes are heavily imbalanced?
- How would you know the model is overfitting?
Walk me through the bias-variance tradeoff and how it impacts model complexity.
Medium · machine learning fundamentals
Approach
- Say how you would validate it, and where leakage could enter the split.
- Name the simplest model that could work and what would make you move past it.
- State the learning problem: the label, the unit of prediction and how the model is used.
Follow-up
- How would you know the model is overfitting?
- Where could label leakage enter this setup?
Explain how a Random Forest algorithm works under the hood.
Medium · machine learning fundamentals
Approach
- State the learning problem: the label, the unit of prediction and how the model is used.
- Say how you would validate it, and where leakage could enter the split.
- Pick the metric from the cost of each error type, not from habit.
Follow-up
- How would you know the model is overfitting?
- Where could label leakage enter this setup?
Can you explain the difference between L1 and L2 regularization, and when you would use each?
Medium · machine learning fundamentals
Approach
- Name the simplest model that could work and what would make you move past it.
- State the learning problem: the label, the unit of prediction and how the model is used.
- Pick the metric from the cost of each error type, not from habit.
Follow-up
- What changes if the classes are heavily imbalanced?
- Where could label leakage enter this setup?
Given a Pandas DataFrame, how would you group the data by a specific column and calculate the mean of another column?
Medium · coding and algorithms
Approach
- Choose the data structure from the access pattern, not from familiarity.
- State the target complexity and say which constraint rules the naive version out.
- Name the brute-force solution and its complexity before improving on it.
Follow-up
- What is the worst case, and how likely is it on real data?
- How does this change if the input no longer fits in memory?
How would you optimize a Python script that is running too slowly on a large dataset?
Medium · coding and algorithms
Approach
- Restate the input: its shape, its size, and what is guaranteed about it.
- State the target complexity and say which constraint rules the naive version out.
- Walk one small example through your approach before writing the whole thing.
Follow-up
- Which test case would catch an off-by-one here?
- How does this change if the input no longer fits in memory?
Track a rolling failure rate per destination for circuit decisions
Easy · sliding window · ring buffer · circuit breaker
The egress service delivers about 1,500 webhooks per second across roughly 40,000 destinations, each call bounded by a 10 second timeout. Maintain, per destination, the failure rate over the trailing 60 seconds so a caller can ask before dispatch whether the circuit should open. Attempts arrive as (destination_id, finished_at_ms, outcome). Requirement: amortised O(1) per attempt, with total memory bounded by the destination count rather than by traffic. Give the structure, its exact memory, and the rule that stops a destination with three attempts from opening a circuit.
Approach
- Name the exact-deque version and then reject it as the default. Holding timestamps and advancing a tail pointer past anything older than now minus 60 seconds is a correct two-pointer window at amortised O(1) per attempt, but its memory tracks in-window traffic, so one destination in a retry storm holds hundreds of thousands of entries while thousands of quiet destinations hold none.
- Use a ring of 60 one-second buckets per destination, each bucket a pair of counters for attempts and failures. On an attempt, advance the ring by the elapsed whole seconds, zeroing at most min(elapsed, 60) buckets, then increment the head. That is amortised O(1) with a fixed footprint per destination.
- State the footprint: 60 buckets times two 4-byte counters is 480 bytes of payload per destination, so 40,000 destinations is roughly 20 to 25 MB with per-entry overhead, bounded by the catalogue rather than by the rate. The cost is granularity, since the oldest bucket ages out in whole seconds, which is far tighter than the decision needs.
- Require a minimum sample before the circuit may open. A destination with three attempts and three failures reads as 100 percent and is not evidence; a floor of roughly 20 attempts in the window makes the ratio meaningful, and below that floor use a run of consecutive failures as the trigger instead.
- Expire idle destinations, or memory grows with every destination ever seen rather than with the live set. Hold the rings in a bounded LRU keyed on destination_id and treat a miss as no history, which is the correct default for an endpoint that has been silent for a minute.
- Keep the half-open probe out of the window arithmetic. After the circuit opens, one probe per interval decides whether to close it, and folding that single success into a window that still holds a 100 percent failure history would reopen the destination on one data point.
Worked solution (20 min)
- Define the bucket struct and the advance step: take floor(finished_at_ms / 1000), compare with the ring's current second, zero min(delta, 60) buckets forward, then write into the new head.
- Trace a destination that receives 5 attempts, goes silent for 90 seconds, then receives one more, and confirm the rate is computed from one attempt rather than six.
- Compute total memory for 40,000 destinations at 60 buckets of two 4-byte counters, and state what changes if the window widens to 300 seconds.
- Write the open rule as a single predicate combining the minimum-attempt floor with the rate threshold.
Expected result: A per-destination ring of 60 one-second (attempts, failures) buckets advanced lazily for amortised O(1) cost, roughly 480 bytes of counters per destination and about 20 to 25 MB for 40,000 of them, an LRU bound on live rings, and an open rule requiring both a minimum attempt count in the window and a rate above threshold.
Verify it yourself
- After 60 seconds of silence a destination's window holds zero attempts and cannot open a circuit.
- Memory measured with one hot destination and 39,999 idle ones is indistinguishable from the reverse, which the deque version would fail.
- Replaying the same attempt stream with the clock shifted by an hour yields the same final rate.
Follow-up
- The fleet is 30 instances and each sees roughly a thirtieth of a destination's traffic. Where does the rate actually live, and what does a per-instance answer get wrong?
- A destination answers in 9.5 seconds and succeeds. It is not failing but it is consuming your per-destination concurrency. What signal should open the circuit here?
- How would you make the window survive a process restart, and is it worth the cost?
Keep soft-deleted accounts from blocking re-registration
Easy · soft delete · unique constraints · engine differences
app_user holds user_id, tenant_id, email CITEXT, password_hash (NULL for SSO principals), email_verified_at, auth_version, status ('invited','active','suspended','deactivated'), created_at, updated_at, deleted_at. Two live accounts for one address inside a tenant must be impossible, but an address freed by a soft delete must be reusable, and the same tenant may delete and re-register it repeatedly. Write the uniqueness DDL for PostgreSQL 16, then the equivalent for MySQL 8 where partial indexes do not exist, and say what each permits once three deleted rows already hold that address.
Approach
- Start from what is actually unique: not (tenant_id, email), but (tenant_id, email) among live rows. PostgreSQL says that directly — CREATE UNIQUE INDEX app_user_live_email ON app_user (tenant_id, email) WHERE deleted_at IS NULL. A full constraint over the same two columns burns the address permanently the first time someone deletes an account.
- Keep case-insensitivity in the type or the index, never in the application: CITEXT as given, or UNIQUE (tenant_id, lower(email)) as an expression index where the extension is unavailable. A case-sensitive unique column is exactly how two accounts for one human appear.
- For MySQL 8 the predicate has to move inside the key: add a discriminator column that is a constant 0 while the row is live and is set to user_id on delete, with UNIQUE (tenant_id, email, deleted_marker). Live rows share the constant and still collide; deleted rows differ from each other and stop colliding.
- State the NULL variant and its dependency: leaving the marker NULL for deleted rows also works, because a unique index treats NULLs as distinct — true in MySQL, and true in PostgreSQL only under the default NULLS DISTINCT, which PostgreSQL 15 lets you reverse. Check the polarity against the three existing deleted rows: constant-on-live is what preserves the collision you want, and reversing it silently admits duplicate live accounts.
- Say what a soft delete must do besides setting deleted_at: increment auth_version so existing tokens stop validating, leave resource.owner_user_id and resource_revision.actor_user_id intact, and accept that the address is retained — erasure is a different requirement answered by scrubbing the column, not by a DELETE that would break those references.
Worked solution (20 min)
- Create the PostgreSQL partial unique index, insert a live row, soft delete it, and insert the same address again.
- Repeat the delete-and-reinsert cycle three times and confirm three deleted rows coexist with exactly one live row.
- Write the MySQL form with the discriminator, then deliberately reverse the polarity so live rows carry NULL, and show two live duplicates commit.
- Attempt a second live insert on both engines and map the resulting 23505 / ER_DUP_ENTRY to the 409 the handler should return.
Expected result: PostgreSQL rejects only the second live row and admits unlimited deleted ones. MySQL behaves identically with UNIQUE (tenant_id, email, deleted_marker) where live = 0; with the polarity reversed, two live rows commit and nothing afterwards can tell them apart.
Verify it yourself
- Group live rows by (tenant_id, lower(email)) after a randomised insert/delete run — no group exceeds one.
- Inserting 'User@x.test' after 'user@x.test' is rejected; if it is accepted, the case handling is in application code rather than in the index.
- EXPLAIN a login lookup and confirm the partial index is chosen — it is only usable by queries that carry the deleted_at IS NULL predicate, so a query without it falls back to a scan.
Follow-up
- A deleted account re-registers with the same address the next day. Do the old resource rows follow the new user_id, and how does the API keep the two principals apart?
- How do you honour an erasure request while resource_revision.actor_user_id still references this table?
- What changes if a user may hold membership in two tenants?
Explain why the owner filter ignores the listing index
Medium · composite index · query plans · sort elimination
The only index on resource is (tenant_id, status, updated_at DESC, resource_id DESC). A new endpoint returns one user's resources across all statuses, newest created first: WHERE tenant_id = $1 AND owner_user_id = $2 ORDER BY created_at DESC LIMIT 20. On a tenant with 2M rows it takes 900 ms and EXPLAIN shows a sort above a large scan. Explain precisely why the existing index cannot serve it, give the index that can, and state which of these the new index still will not help: owner_user_id alone across tenants; the same query ordered by updated_at. PostgreSQL 16.
Approach
- Separate the two jobs an index does. For filtering, a composite btree is seekable only on a left prefix, so with no predicate on status the scan can at best range over tenant_id and test owner_user_id per row; PostgreSQL 16 has no btree skip scan to jump the unconstrained column.
- For ordering, the index is sorted by (status, updated_at) within a tenant and not by created_at, so the LIMIT cannot stop early: every matching row is read and then sorted. That is the 'Sort Method: top-N heapsort' line, and it is why the plan reads 2M rows to answer with 20.
- Derive the replacement from the access path — equality, equality, then the ordering column: CREATE INDEX CONCURRENTLY ON resource (tenant_id, owner_user_id, created_at DESC). The scan seeks to the (tenant, owner) range and walks 20 entries in order, so the Sort node disappears along with the row-read.
- Treat INCLUDE (title, status) as conditional, not free. An index-only scan still visits the heap for any row whose page is not marked all-visible, so on a table taking 1.2k writes/second the win depends on autovacuum keeping the visibility map current, and the wider index costs more on every insert.
- Answer the two negatives explicitly. owner_user_id alone is not a left prefix of the new index, so it degrades to a full scan of the index at best. Ordered by updated_at, the query still seeks on the (tenant, owner) pair but must sort, because only created_at is ordered within that pair.
- Measure both sides with EXPLAIN (ANALYZE, BUFFERS) and compare estimated against actual rows at the lowest node — a 2M-versus-200 misestimate there is usually what chose the plan, and adding an index will not fix a statistics problem.
Follow-up
- 90% of rows are status='active'. Would a partial index WHERE status = 'active' change your answer, and for which of the three queries?
- A dashboard runs this for 40 owners in one page load. What changes about the design?
- How do you roll this index out on a table taking 1.2k writes/second, and what does it cost on every insert from then on?
Rebuild the search projection while it serves nine thousand queries
Medium · projections · eventual consistency · rebuild · watermark
The read-model service answers about 9k queries/second at a 120 ms p99 from a projection built off the event log, normally under 2 seconds behind. A mapping change forces a full rebuild from resource_revision, during which apply lag rises to minutes. Writes continue at 1.2k/second and events at 4k/second. Design the rebuild: how the new index is populated and cut over, how position is tracked per log partition, what the API returns alongside results so a client can tell a stale answer from a current one, and the criterion for cutting over.
Approach
- Build into a second index and swap an alias rather than mutating the live one. The rebuild is then reversible by pointing the alias back, so a bad mapping costs a wasted rebuild instead of an outage. The price is peak storage for two full copies and double apply load during catch-up, and both numbers should be stated up front rather than discovered when disk fills.
- Track position per log partition, not globally. Order is guaranteed only within an aggregate's partition, so progress is a vector, and the only number safe to publish is taken from the least advanced partition - the slowest one is what bounds completeness. Publishing the most advanced partition's position declares the projection current while another partition sits twenty minutes behind.
- Make apply idempotent so the backfill and the live tail can overlap without a freeze. Each document records the aggregate_version it reflects, and any event at or below that version is discarded. This is why the event carries the full fact and not a delta: a delta cannot be discarded safely, and a consumer that calls back to read current state applies a state newer than the event it is processing, which is how a projection ends up with changes applied out of order.
- Turn lag into a contract instead of a surprise. Return the watermark with every result set, and return the version produced by a write so the client can compare the two. A client that wrote version 7 and receives results at a watermark older than its own commit can show that its change is still landing, rather than rendering the previous value as current. Blocking the read until the projection catches up would convert a staleness problem into an availability problem at 9k queries/second, and choosing not to do that is the trade.
- Define the cutover numerically. Writes are untouched by the rebuild - they commit to the primary and land in the outbox - so the only coupling is apply throughput. If catch-up applies slower than the 4k events/second arriving, it never converges. The cutover criterion is that lag is measurably decreasing and below a stated threshold, not that the backfill loop reached the end of its range.
Follow-up
- The rebuilt index disagrees with the primary tables for 300 documents. Which is authoritative, and how do you decide without freezing writes?
- The rebuild doubles load on the log and pushes the projection p99 from 120 ms to 400 ms. What do you throttle, and which signal sets how much?
- Clients start polling until the watermark passes their write. What does that do at 9k queries/second, and what do you offer instead?
Evolve the resource contract without breaking integrations you cannot upgrade
Hard · versioning · backward compatibility · deprecation
GET /v1/resources/{id} returns status from a four-value enum ('draft','active','archived','deleted'), a numeric version, and the body inline. Consumers are a browser app you deploy and roughly 300 server-side integrations, some untouched for two years, that switch exhaustively on status and parse ids as JSON numbers. You must add a 'pending_review' status, move bodies over 256 KB to a body_ref pointer, and expose per-field change history from resource_revision. Specify the compatibility policy, the wire changes, how both generations are served, and the evidence that lets you remove the old shape.
Approach
- Write the policy first and date it: fields are added, never retyped or repurposed; consumers ignore unknown fields; an unknown enum value maps to a documented fallback; nothing is removed until telemetry shows no caller reads it. Then say the uncomfortable part out loud - v1 shipped without the unknown-value rule, so 300 running integrations have no fallback, and no server change can install one into code that is already deployed.
- That single fact forces per-request negotiation rather than a server-side default. Keep one internal model and select a serialiser from an explicit version in the request, and default a caller that sends nothing to the oldest supported version. Defaulting to the newest is the change that breaks every integration that never asked for anything, on the day you ship.
- Downgrade 'pending_review' for old callers to the nearest state they already handle, 'draft', and state the loss explicitly: those integrations cannot see review state and will treat the resource as editable. If that is unacceptable for one integration, the remedy is moving it to the new version, not a cleverer projection - there is no mapping that invents a state the client has no code for.
- Make the body change additive. Old callers keep
bodyinline; the new shape addsbody_refand a size field, and resources over the limit are served to old callers by resolving the pointer server-side or by refusing with a documented code, chosen once and published. Never repurposebodyto carry the pointer: a client that renders it shows a storage key to a user, and that failure is silent, where a missing field would have been loud. While you are here, serialise BIGINT ids as strings in the new shape - a browser parsing JSON numbers gets IEEE-754 doubles, exact only to 2^53 - and treat that as its own breaking change requiring the same negotiation, not a quiet fix. - Add change history as a separate sub-resource, GET /v1/resources/{id}/revisions, keyset-paginated over (resource_id, version) rather than as an array inside the resource. A field added to a hot response is paid for by every caller including those that never read it, and an unbounded array inside a cached object destroys the size assumptions the cache was configured with.
- Retire on evidence rather than on a date alone: count requests per negotiated version per credential, publish a Sunset header (RFC 8594) with the removal date and a link to the migration, contact the credentials still on the old version, then answer 410 Gone once it is removed. Keep each version's serialiser under snapshot tests so a refactor cannot change v1's bytes by accident.
Worked solution (40 min)
- Write and date the compatibility policy as five rules, then mark which of them v1 callers cannot honour.
- Choose the negotiation mechanism and the default for an unversioned request, and justify the default in one sentence.
- Write the downgrade table: new state to old state, new body shape to old body shape, and what is lost in each direction.
- Specify the revisions sub-resource: its URL, its cursor, and why it is not a field on the resource.
- Write the sunset plan: the per-version per-credential metric, the header, the lead time, and the terminal status code.
Expected result: A dated additive policy, version negotiation over one internal model with the oldest version as the default, a downgrade table with its losses named, history as a paginated sub-resource, and a removal gated on per-credential version telemetry rather than a calendar entry.
Verify it yourself
- A two-year-old integration receives a resource that is really in 'pending_review' and neither crashes nor renders a blank state.
- Deleting the legacy body field from the internal model breaks exactly one serialiser and no business logic.
- Every v1 response is byte-stable under snapshot tests after an unrelated refactor.
Follow-up
- An old integration submits a status transition while the resource is really in 'pending_review'. What does the write path accept, and what does it reject?
- Two years on you want to delete the v1 serialiser. What evidence makes that safe, and who must be contacted before it happens?
- How would you test against a two-year-old integration rather than against today's source?
Every query on one table stalls for forty seconds mid-deploy
Medium · lock contention · ddl · migrations · postgres
During a release on PostgreSQL, every query touching resource times out for about 40 seconds and then recovers with no intervention. The release ran one migration, ALTER TABLE resource ADD COLUMN archived_reason TEXT, and the migration log shows it completing in 6 ms. Unrelated tables showed no change in error rate. Explain how a 6 ms statement caused a 40-second stall, give the ordered checks you would run on a live system to confirm it, and give the migration procedure that prevents a repeat.
Approach
- Separate the statement's duration from the lock's duration. ADD COLUMN with no default is a catalogue-only change and genuinely runs in milliseconds, but it requires ACCESS EXCLUSIVE, and it cannot acquire that until every transaction already touching the table has finished.
- Account for the queueing, which is the part that surprises people. A lock request that is waiting blocks later requests for conflicting modes behind it rather than letting them overtake, so one long-open transaction holds the DDL and the DDL holds all the traffic. The stall length is set by the longest open transaction, not by the size of the change.
- Confirm on a live system in this order: pg_stat_activity for that table ordered by xact_start, looking for the oldest transaction and specifically for state = idle in transaction; then pg_locks where granted = false to find the waiter; then join them on pid to name blocker and blocked. pg_blocking_pids() does that join for you and is the fastest single call.
- Prevent rather than merely time it better. Set lock_timeout to a second or two on the migration session so the DDL abandons the queue after a bounded wait and is retried, instead of holding it for as long as the oldest transaction lives. Be exact about what that buys: queries arriving during the wait still queue behind the pending ACCESS EXCLUSIVE request, so each attempt costs them up to one lock_timeout of added latency. The outage goes from 40 seconds to about one second per attempt, not to zero. Also run migrations away from deploy-time peaks, and put a statement timeout and an idle-in-transaction timeout on the analytics role that opens the long transactions.
- Know the lock each change takes, since the mitigation differs by change. A column with a non-volatile default is a metadata-only change from PostgreSQL 11 and still needs the brief ACCESS EXCLUSIVE; an index needs CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block and leaves an INVALID index to drop if it fails; a check or foreign key is added NOT VALID and then VALIDATE CONSTRAINT as a separate statement under a weaker lock.
Follow-up
- The same release also wants NOT NULL on that column. What is the sequence that gets there without a long lock?
- Your lock_timeout retry fails ten times in a row because the analytics transaction is always open. What do you change?
- How does this differ on MySQL with InnoDB online DDL, and what is the equivalent of the waiting-lock queue there?
Suggested practice checklist
For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.
Day 1 — Measure before reasoning
- Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
- Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
- Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.
Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.
Day 2 — References, copies, and the bugs they produce
- Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
- Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
- Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.
Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.
Day 3 — Types, once, in a language that checks them
- Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
- Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
- Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.
Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.
Day 4 — Concurrency, starting with what actually runs at the same time
- Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
- Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
- Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.
Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.
Day 5 — Debugging as a procedure rather than an instinct
- Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
- Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
- Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.
Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.
Day 6 — Tests that catch the bug you are about to write
- Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
- Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
- Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.
Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.
Day 7 — Debug something broken, out loud
- Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
- Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
- Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.
Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.
Outage answers are graded on instruments
Nobody is scoring your stamina at three in the morning. What carries weight is which signal told you something was wrong, what you measured before touching anything, what you rolled back versus what you fixed forward, and why you picked one. 'We restarted it and it went away' is a story about not knowing.
Tell me about a time you had to work with a difficult dataset. How did you manage it?
Medium · behavioural and collaboration
Approach
- Pick a story where you made the decision, not one where you watched it.
- Close with what you would do differently, concretely.
- Name the disagreement and how you resolved it with evidence.
Follow-up
- How did you know your change caused the improvement?
- What did you decide not to do, and why?
How do you handle missing or corrupted data in a dataset before training a model?
Medium · behavioural and collaboration
Approach
- Close with what you would do differently, concretely.
- State the situation in two sentences and spend the rest on the reasoning.
- Give the blast radius: what could have broken, and what you measured.
Follow-up
- How did you know your change caused the improvement?
- What would you do differently if you ran that again?
Narrate an outage you owned from page to postmortem
Hard · incident response · blast radius · postmortems
Pick an incident you personally drove, ideally one where writes were affected rather than reads. In six to eight minutes: state the symptom as it first appeared on a dashboard, the blast radius you established before you knew the cause, the mitigation you applied and when, the mechanism you eventually proved, and the follow-up that would prevent a repeat. Bring numbers: error rate, tenants affected, minutes to mitigate, minutes to resolve. If you cannot name what you measured, choose a different incident.
Approach
- Open on the signal rather than the cause: which metric at which percentile moved, on which service, at what time, so the listener follows the same evidence you had rather than a conclusion you already reached.
- Separate mitigation from diagnosis out loud. State what you did to stop the bleeding (flag off, shed traffic, drain a lease, roll back a deploy) and say plainly that you did it before the mechanism was known, because those are two jobs with different deadlines.
- Establish blast radius in countable terms: how many tenants, how many writes, and crucially whether the effect was loss or only delay. An append-only revision table or a pending outbox row means the change survived and the projection was merely behind, which is a repair rather than a data-loss incident.
- Prove the mechanism instead of asserting it. Name the trace span that grew, the plan that flipped to a sequential scan, the lease that expired, plus one alternative you ruled out and the signal that stayed flat while you ruled it out.
- Close on the durable fix and its cost, distinguishing what landed that week from what needed an expand-and-contract migration across several deploys, and say which of the two you actually finished.
Follow-up
- What would you do differently in the first five minutes, given the same dashboard and no more information?
- Which follow-up action did you deliberately not take, and why was dropping it the right call?
- How did you convince yourself the mitigation was safe to apply while the cause was still unknown?