Blackstraw.ai Machine Learning Engineer Interview Questions & Guide 2026

The 60-second brief

As a Machine Learning Engineer at Blackstraw.ai, you are at the intersection of high-scale data engineering and advanced predictive modeling. You are tasked with transforming complex, raw data streams into actionable intelligence that drives the company’s core AI-driven products. Your role is pivotal in bridging the gap between research-grade models and production-ready, scalable software.

Allocate preparation against your weakest link rather than your favourite topic. A strong algorithm habit usually comes with weak out-loud explanation of tradeoffs, and years of shipping usually come with rusty from-scratch implementation under a clock.

Focus areas

• Trace a symptom to a mechanism under load
• Detect concurrent edits instead of losing writes
• Paginate large result sets with keyset cursors

Overview

You will work on optimizing model performance, ensuring high availability, and architecting robust data pipelines that handle significant throughput. Because Blackstraw.ai prioritizes efficiency and reliability, you must be comfortable managing the trade-offs between computational cost and predictive accuracy. This role offers the opportunity to influence the technical roadmap of AI solutions that provide real-world business value.

Your ability to explain the "why" behind your technical choices is as important as the code you write. Expect to be challenged on your architectural decisions during technical discussions.

The Blackstraw.ai interview process

Blackstraw.ai candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Technical Screen

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

• Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
• Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
• Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
• Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

• For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
• For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
• Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

System Design Interview

An unlabelled round is first an information problem, and the cheapest information is free. Whoever schedules it can usually tell you how long it runs, who will be in the room and what they work on, whether you will be writing code and in what environment, and whether anything is being sent beforehand. Ask in writing so the answer is on record, then prepare for the two or three formats those answers still leave open instead of betting on one. What separates a strong candidate is not guessing right; it is having an opening that works whichever one it turns out to be.

What to demonstrate

• Whether you can start work from an ambiguous brief, since tolerating a vague scope without stalling is the same thing the job asks for
• Whether the questions you asked beforehand were ones that change your preparation, such as duration, medium and who is joining, rather than ones whose answers you could not have acted on
• Whether you adapt when the round turns out to be something other than what you were told, instead of spending the first ten minutes visibly recalibrating

How to prepare

• Send one short scheduling message asking four things: how long, who is joining and what they work on, whether you will be writing code and where, and whether to prepare anything in advance. Treat a vague reply as real information, since it means the round is loosely structured and you will be shaping it yourself.
• Write one opening that works in any of the formats still open: restate in your own words what you have been asked to do, then ask which of two directions is more useful to them. Say it aloud until it stops sounding recited.
• Set up for the two most likely formats before the call starts, with a blank editor in the language you would choose and a shared document you can type into, so a format surprise costs you nothing in the first minutes

Practical Application Assessment

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

• Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
• Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
• Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
• Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

• For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
• For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
• Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Where candidates lose points

Shipping a migration and the code that depends on it as a single change

During any rolling deploy, and for as long as a rollback remains possible, old and new code execute against the same schema at the same time. A migration that drops or renames a column breaks every instance that has not restarted yet, and code that requires a column the migration has not applied breaks every instance that restarted early. The discipline is expand then contract: add the new column nullable, write both shapes, backfill in batches, move reads across once the backfill is verified, and only then stop writing the old shape and drop it - four deploys, usually spread over days. It feels disproportionate until the first rollback, at which point it is the only reason the previous version still runs.

Letting a slow dependency consume unbounded concurrency

The failure that takes a service down is usually not an error but a delay. A dependency answering in thirty seconds instead of fifty milliseconds holds each request's worker or connection six hundred times longer, and since required concurrency is arrival rate times latency, a fleet sized for sixty in-flight requests now needs thirty-six thousand to sustain the same rate - so it queues, and requests whose clients have already abandoned them still occupy resources. Retries make it precisely worse: a policy of three attempts triples the load on a dependency at the exact moment it is least able to serve, which is how one slow dependency becomes an outage of everything sharing that pool. Containment is four specific things - a timeout on every outbound call shorter than the caller's remaining budget, a bounded pool per dependency so one cannot starve the others, backoff with full jitter rather than a fixed delay so retries do not resynchronise, and a circuit that stops sending once the failure rate makes an attempt pointless.

Arguing past a hint

When the interviewer asks what happens for a particular input or floats a different data structure, stop and take it seriously; it is almost always a correction rather than idle curiosity. Talking over it converts a recoverable wrong turn into a data point about how you handle review.

Optimising an axis nobody named

Ask which resource is actually scarce here: wall-clock latency, throughput, memory footprint, cost per request, or engineering time. Shaving a constant factor off an in-memory step is wasted effort when the same function makes a blocking remote call inside the loop.

Questions & practice

What are the advantages of using FastAPI for serving machine learning models?

Medium · machine learning fundamentals

Approach

1. Say how you would validate it, and where leakage could enter the split.
2. State the learning problem: the label, the unit of prediction and how the model is used.
3. Name the simplest model that could work and what would make you move past it.

Follow-up

• Where could label leakage enter this setup?
• How would you know the model is overfitting?

How do you implement asynchronous execution using async and await?

Medium · coding and algorithms

Approach

1. Name the brute-force solution and its complexity before improving on it.
2. State the target complexity and say which constraint rules the naive version out.
3. Walk one small example through your approach before writing the whole thing.

Follow-up

• Which test case would catch an off-by-one here?
• What is the worst case, and how likely is it on real data?

Explain the role and impact of the Global Interpreter Lock (GIL) in Python.

Medium · coding and algorithms

Approach

1. Walk one small example through your approach before writing the whole thing.
2. Name the brute-force solution and its complexity before improving on it.
3. State the target complexity and say which constraint rules the naive version out.

Follow-up

• What is the worst case, and how likely is it on real data?
• Which test case would catch an off-by-one here?

How does Python’s Garbage Collection mechanism work, and when might you need to intervene?

Medium · coding and algorithms

Approach

1. Choose the data structure from the access pattern, not from familiarity.
2. Name the brute-force solution and its complexity before improving on it.
3. State the target complexity and say which constraint rules the naive version out.

Follow-up

• What is the worst case, and how likely is it on real data?
• Which test case would catch an off-by-one here?

Identify the heaviest tenants in a five-minute window under memory pressure

Medium · top-k · heavy hitters · streaming

The edge service handles about 3,000 requests per second across roughly 50,000 tenants, peaking near 9,000. Expose the 50 heaviest tenants by request count over the trailing five minutes so limits can be tightened before one tenant's backfill starves the fleet. You may not retain five minutes of raw records. Give the exact solution and its memory, then the bounded-memory approximation with its error stated as a formula, and say which you would ship and at what tenant cardinality that choice changes.

Approach

1. Do the exact version first, because it is affordable at this cardinality: a ring of 300 one-second counters per tenant, advanced lazily, is 1,200 bytes of counters per tenant and roughly 60 to 90 MB for 50,000 tenants with overhead. Carry a running total and subtract the bucket you overwrite so a window read is O(1) rather than 300 adds.
2. Extract the top 50 with a size-k min-heap over the tenant sums: O(d log k) for d tenants, against O(d log d) to sort them all. Maintaining the heap continuously instead of on query requires a tenant-to-heap-index map, because incrementing a count already inside the heap means sifting from a known position, and without that map you rebuild the heap on every request.
3. State the approximation precisely rather than gesturing at sketches. Misra-Gries with m counters retains every item whose true count exceeds N/(m+1), and each retained count underestimates the truth by at most N/(m+1). With m = 1,000 and N = 900,000 requests in the window the error is roughly 900 requests, which is fine for spotting a tenant sending 50,000 and useless for ranking two tenants 200 apart.
4. Say what breaks when the window slides: Misra-Gries and Space-Saving are insert-only and cannot be decremented as records age out. The workable construction is one summary per sub-window, say ten seconds, with 30 summaries merged at query time, and the merged error is the sum of the per-summary errors, so the bound degrades linearly in the number of sub-windows.
5. Choose and defend it: at 50,000 tenants the exact rings cost under 100 MB in a process that already holds more, so ship exact. Keep the sketch for the case that actually motivates it, a per-principal or per-IP key where cardinality runs to millions and is not bounded by anything you control.
6. Raise the fleet problem before it is asked: each of 20 to 40 instances sees only its share, and the top 50 of one shard is not the top 50 of the fleet. Either aggregate counts centrally or accept that a per-instance threshold multiplied by instance count is the limit you are really enforcing.

Worked solution (25 min)

1. Size the exact structure: 300 one-second counters per tenant across 50,000 tenants, plus the running-total trick that makes a window read O(1).
2. Write the top-k extraction with a size-50 min-heap and compare its complexity against sorting all 50,000 sums.
3. Substitute N = 900,000 and m = 1,000 into N/(m+1) and state in requests what the sketch can and cannot distinguish.
4. Write the sub-window merge for the sliding case and state the resulting bound for 30 merged summaries.

Expected result: An exact per-tenant ring of 300 one-second counters at roughly 60 to 90 MB for 50,000 tenants with O(1) window reads, top-50 extraction by a size-k min-heap in O(d log k), a Misra-Gries bound of N/(m+1) with the numbers substituted, the sub-window merge needed to slide it, and a decision to ship exact at this cardinality with the sketch reserved for unbounded keys.

Verify it yourself

• The exact window sum for one tenant equals a replay of its raw records over the same five minutes.
• The heap result matches a full sort of the tenant sums on a sample.
• The sketch's count for a known heavy tenant is at most the true count and no less than the true count minus N/(m+1).

Follow-up

• The heaviest tenant is heavy because of one export job rather than user traffic. Should the limiter treat those as the same tenant?
• Two tenants sit tied at the boundary of the top 50. Does your answer flap, and does the flapping matter?
• You switch to per-principal keys and cardinality goes to 10 million. Walk through what changes.

How do you structure asynchronous requests in a production API?

Medium · data, sql and pipelines

Approach

1. Say which index the query would use, and what makes it unusable.
2. State the isolation you are assuming and the anomaly it still allows.
3. Name the grain you start from and join outward from it.

Follow-up

• What happens to this when the table is ten times larger?
• How would you run this migration without downtime?

Write the update path that detects a concurrent edit

Medium · optimistic concurrency · transactional outbox · audit history

resource carries version INT NOT NULL DEFAULT 1. resource_revision holds revision_id, resource_id, version, actor_user_id, change_kind, patch JSONB, request_id, created_at with UNIQUE (resource_id, version). outbox_event holds aggregate_type, aggregate_id, aggregate_version, event_type, payload, status. A PUT carries the version the client read. Write the exact statements for the single transaction that applies the edit, records the revision and enqueues 'resource.updated', and give the handler's branch on zero affected rows. Then say what PostgreSQL 16 does under READ COMMITTED when two of these updates hit one row at once.

Approach

1. One transaction, three writes, no network call inside it: UPDATE resource SET title = $3, version = version + 1, updated_at = now() WHERE resource_id = $1 AND tenant_id = $4 AND version = $2; then INSERT the resource_revision row at version $2 + 1; then INSERT the outbox_event row at the same aggregate_version. The event goes to a table rather than a broker because no transaction spans both.
2. Branch on the affected-row count before doing anything else. Zero has three causes — stale version, wrong tenant, row gone — so re-read once and map to 409 carrying the current version, or 404 for an id outside the caller's tenant, which also stops the endpoint confirming that another tenant's id exists.
3. State the engine behaviour instead of assuming it. Under READ COMMITTED the second UPDATE blocks on the row lock, and when the first commits PostgreSQL re-evaluates the WHERE clause against the newly committed row, so the version predicate now fails and the statement reports zero rows. Under REPEATABLE READ the identical collision raises SQLSTATE 40001 instead, so the handler must fold both shapes into one conflict response.
4. Keep UNIQUE (resource_id, version) even though the predicate already serialises writers. It is what makes a lost update unwritable if any other path ever reaches the revision table, and it converts a logic bug into 23505 rather than into a silently missing history row.
5. Refuse to auto-retry the whole PUT. A retry re-reads the winner's state and reapplies an intent formed against data that no longer exists — the silent overwrite the version token was added to detect. Return the conflict; merge field-wise only if the patches are provably disjoint.
6. Note that now() is the transaction timestamp in PostgreSQL, so resource.updated_at, the revision's created_at and the outbox row share one instant, which is what later makes reconciliation between the three tables unambiguous.

Worked solution (25 min)

1. Write the three statements plus the rowcount branch and confirm they sit in one BEGIN/COMMIT with no outbound call between them.
2. Run two clients that both read version 7 and apply their updates 5 ms apart; assert one 200 and one 409.
3. Assert resource.version = 8, exactly one resource_revision row at version 8, and one outbox_event row at aggregate_version 8.
4. Repeat at REPEATABLE READ and record the different failure shape (SQLSTATE 40001) the handler must also map to 409.
5. Delete the version predicate and re-run: both writes commit and the first edit disappears with no error raised anywhere.

Expected result: Exactly one of the two commits. version = 8 with one revision row and one outbox row at version 8, the loser gets 409 carrying version 8, and no state exists in which the resource moved but the revision or the outbox row is missing.

Verify it yourself

• COUNT(*) of resource_revision for that resource equals resource.version, with no gap in the version sequence.
• Kill the process between the UPDATE and the outbox INSERT — the transaction rolls back and no event describes a change that never happened.
• Under a 50-way race, resource.version equals the number of 2xx responses rather than the number of requests.

Follow-up

• A client sends the version it read ten minutes ago and the resource has moved three versions. What is in your 409 so it can resolve the conflict without a full re-fetch?
• Two editors, two disjoint fields, no overlap. Does your answer still refuse the second write, and should it?
• Every write now touches a second hot table. How do you keep the outbox insert and its partial index from becoming the write bottleneck at 1.2k writes/second?

How would you design a data pipeline to ensure low-latency inference?

Medium · ml system design and production

Approach

1. Separate the offline training path from the online serving path.
2. Name what you would monitor after launch and what triggers a retrain.
3. Say where features come from at serving time and how they match training.

Follow-up

• How would you detect drift before the metric drops?
• What happens when a feature is missing at serving time?

What is the fundamental difference between DROP, DELETE, and TRUNCATE?

Medium · system design

Approach

1. Name the failure you are designing for, then the recovery path.
2. State the consistency you need, and where you are willing to be stale.
3. Choose a partition key and say what query it makes expensive.

Follow-up

• What would you drop to keep the system up under load?
• How does this behave when that dependency is down for an hour?

Explain how you would manage containerized deployments using AWS EKS and EC2.

Medium · system design

Approach

1. Choose a partition key and say what query it makes expensive.
2. State the consistency you need, and where you are willing to be stale.
3. Name the failure you are designing for, then the recovery path.

Follow-up

• What breaks first when traffic grows ten times?
• What would you drop to keep the system up under load?

Publish rate-limit and deadline semantics the edge actually enforces

Easy · rate limiting · backpressure · retries

The edge API serves about 3k requests/second steady and 9k at peak against a 400 ms p99 budget, with an explicit bounded concurrency limit per instance. Limits exist per principal and per tenant. Callers are a partner integration running nightly bulk loads and a browser app. Specify the counting algorithm and window, which limit a request is charged against, the headers a well-behaved client reads, the status and body when a limit is hit, how that differs from the response when an instance is shedding load, and what each caller does with each.

Approach

1. Choose the counter and name its failure mode. Fixed windows admit nearly twice the limit across a boundary - a full burst at the end of one window and another at the start of the next. A token bucket states sustained rate and burst separately, which is exactly what a nightly bulk load needs. A sliding-window counter is more faithful and costs more state per key. State the choice and the burst it permits.
2. Charge each request against both keys and reject on the stricter. The tenant limit protects the shared primary, which absorbs roughly 1.2k writes/second in total; the per-principal limit stops one credential inside a tenant from consuming that tenant's whole allowance. The tenant is the fairness unit for the same reason it is the leading column of every index.
3. Advertise limit, remaining and reset for the binding key on every response, not only on rejections, so a client can pace before it is refused. Pick one naming scheme - the RateLimit-* draft fields or an X-prefixed set - document the units, and never change them afterwards.
4. Separate two rejections that look identical to a naive client. 429 means this caller exceeded its own share and Retry-After is a real schedule it should obey. 503 means the instance is at its concurrency bound and shedding, which is a statement about the server; a fleet-wide 503 retried on a fixed delay resynchronises every client into one stampede, so full jitter is mandatory there and the delay is the client's guess, not ours.
5. Make shedding cheap and early - before the token is verified against the database, before any downstream call - because a rejection that costs as much as the work relieves nothing. Drop requests whose client deadline has already elapsed rather than serving them; the caller has stopped listening and the work is pure cost.
6. Write the caller behaviours down: the bulk loader paces against remaining and treats a 429 as a defect in its own pacing; the browser surfaces the wait and must never retry a 429 inside a render loop, which turns one limited user into a self-inflicted flood.

Worked solution (20 min)

1. Write the bucket parameters for both keys: sustained rate, burst size, and the refill interval, with the arithmetic that ties them to the 3k/9k figures.
2. Draft the three response headers and one example 429 body carrying a code, the limit that bound, and Retry-After.
3. Write the 429-versus-503 decision as a two-line rule an on-call engineer can apply to a log line.
4. State where in the request pipeline the rejection happens and which work it skips.

Expected result: A token bucket per principal and per tenant with named burst, headers on every response, 429 with Retry-After for share exhaustion, 503 with jittered client backoff for concurrency shedding, and rejection placed before authentication's database read.

Verify it yourself

• A client that respects remaining never sees a 429 under steady traffic at its configured rate.
• The cost of a shed request is bounded and does not include a database round trip.
• Peak throughput times the limiter's per-request state cost is affordable at 9k requests/second.

Follow-up

• One tenant stays under its limit and still degrades everyone else during a backfill. What changes - the limiter, the worker concurrency caps, or both?
• How are counters kept correct across 20 to 40 stateless instances, and what does your answer cost per request?

Every query on one table stalls for forty seconds mid-deploy

Medium · lock contention · ddl · migrations · postgres

During a release on PostgreSQL, every query touching resource times out for about 40 seconds and then recovers with no intervention. The release ran one migration, ALTER TABLE resource ADD COLUMN archived_reason TEXT, and the migration log shows it completing in 6 ms. Unrelated tables showed no change in error rate. Explain how a 6 ms statement caused a 40-second stall, give the ordered checks you would run on a live system to confirm it, and give the migration procedure that prevents a repeat.

Approach

1. Separate the statement's duration from the lock's duration. ADD COLUMN with no default is a catalogue-only change and genuinely runs in milliseconds, but it requires ACCESS EXCLUSIVE, and it cannot acquire that until every transaction already touching the table has finished.
2. Account for the queueing, which is the part that surprises people. A lock request that is waiting blocks later requests for conflicting modes behind it rather than letting them overtake, so one long-open transaction holds the DDL and the DDL holds all the traffic. The stall length is set by the longest open transaction, not by the size of the change.
3. Confirm on a live system in this order: pg_stat_activity for that table ordered by xact_start, looking for the oldest transaction and specifically for state = idle in transaction; then pg_locks where granted = false to find the waiter; then join them on pid to name blocker and blocked. pg_blocking_pids() does that join for you and is the fastest single call.
4. Prevent rather than merely time it better. Set lock_timeout to a second or two on the migration session so the DDL abandons the queue after a bounded wait and is retried, instead of holding it for as long as the oldest transaction lives. Be exact about what that buys: queries arriving during the wait still queue behind the pending ACCESS EXCLUSIVE request, so each attempt costs them up to one lock_timeout of added latency. The outage goes from 40 seconds to about one second per attempt, not to zero. Also run migrations away from deploy-time peaks, and put a statement timeout and an idle-in-transaction timeout on the analytics role that opens the long transactions.
5. Know the lock each change takes, since the mitigation differs by change. A column with a non-volatile default is a metadata-only change from PostgreSQL 11 and still needs the brief ACCESS EXCLUSIVE; an index needs CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block and leaves an INVALID index to drop if it fails; a check or foreign key is added NOT VALID and then VALIDATE CONSTRAINT as a separate statement under a weaker lock.

Follow-up

• The same release also wants NOT NULL on that column. What is the sequence that gets there without a long lock?
• Your lock_timeout retry fails ten times in a row because the analytics transaction is always open. What do you change?
• How does this differ on MySQL with InnoDB online DDL, and what is the equivalent of the waiting-lock queue there?

Suggested practice checklist

For someone fluent in a dynamic language who has shipped real work but has never had to say what the runtime is doing underneath. The week is built on measuring and deliberately breaking things, because the questions that expose this background are the ones where the interviewer asks why a second time.

Day 1 — Measure before reasoning

• Take a slow piece of your own code, write down in advance where you believe the time goes, then profile it and record how wrong the guess was. The cost is usually an allocation you did not notice or an accidental quadratic membership test.
• Replace one list membership test inside a loop with a set and measure at a thousand, ten thousand and a hundred thousand elements, confirming the shape of the curve rather than only that it got faster.
• Write down the three quantities you can now measure instead of assert: wall time, peak memory, and call count for the function you suspected.

Deliverable: A before-and-after profile of real code plus a written note on the size of the gap between the guess and the measurement.

Day 2 — References, copies, and the bugs they produce

• Write the function with a mutable default argument, call it three times, and explain the accumulating result: the default is evaluated once when the function is defined, so every call shares one object.
• Build a nested structure, take a shallow copy, mutate an inner element, and show that both views changed, because a shallow copy duplicates the container and not the elements. Then fix it with a deep copy and state the cost you just accepted.
• Write two functions, one mutating its argument in place and one rebinding the local name, and predict the caller's view of each before running it. That single distinction produces most of the bugs that pass their tests.

Deliverable: Three small programs whose output you predicted correctly before running, each with a one-line statement of the rule underneath.

Day 3 — Types, once, in a language that checks them

• Port one module you have already written, roughly a hundred lines, into a statically typed language, and record every place the compiler demanded an answer your original had left implicit: a value that can be absent, a numeric width, a case never handled.
• Write the same signature in both languages and state what the static one guarantees before the program runs and what it does not, since it will not save you from a wrong algorithm or an index out of range.
• Write the difference between an interface satisfied by declaration and one satisfied structurally, with one case each where the other approach would miss the mistake.

Deliverable: One module in two languages plus a list of the questions the type checker forced you to answer.

Day 4 — Concurrency, starting with what actually runs at the same time

• Run the same CPU-bound function across four threads and four processes and measure both. Under the default CPython build the threaded version will not speed up, because only one thread executes bytecode at a time; the process version will. Check which build you are on first, since free-threaded builds remove that lock and change the result.
• Then run a blocking I/O workload across four threads and measure it speeding up, because the interpreter releases that lock around blocking calls, which is why treating threads as useless is wrong as a general claim.
• Build the lost update: two threads each incrementing a shared counter a hundred thousand times, and show a final value below the expected sum, because an increment is a load, an add and a store and the thread can be suspended between them. Fix it with a lock and then measure what the lock costs.

Deliverable: Three measurements, threads against processes on CPU work, threads on I/O work, and a demonstrated lost update, each with the mechanism written underneath.

Day 5 — Debugging as a procedure rather than an instinct

• Work one real failure as a bisection: find a revision or an input size where it is good and one where it is bad, halve repeatedly, and state the two assumptions bisection needs, that the property changes exactly once across the range and that the test is reliable.
• Minimise one failing input to the smallest version that still fails, and record how many rounds it took.
• Keep a hypothesis log for one bug in three columns, what I believe, what would disprove it, what I observed, and stop yourself the first time you are about to change two things at once.

Deliverable: One bug worked to root cause with a written hypothesis log and a minimised reproducing input.

Day 6 — Tests that catch the bug you are about to write

• Implement an LRU cache with a capacity bound, then write the three test cases that would catch an off-by-one in eviction: insert exactly capacity items and assert nothing was evicted, insert one more and assert the least recently used key is the one gone, and read an old key just before that insert so the eviction victim changes.
• Add a property test comparing your implementation against a deliberately slow reference, an ordered list scanned linearly, over a few thousand random operation sequences, because a slow reference finds the cases you would not have thought to write.
• Write one numeric test that fails under exact equality and passes with a tolerance, and state why the tolerance has to be relative rather than absolute once the magnitudes grow.

Deliverable: An LRU implementation with three boundary tests, one property test against a slow reference, and one tolerance-based numeric test.

Day 7 — Debug something broken, out loud

• Have someone plant three defects in a two-hundred-line program, an off-by-one, a shared mutable state bug, and a wrong error-handling path, then find them while narrating, under a fixed rule: state the hypothesis before touching anything.
• Time each one and record which tool found it, reading, a printed value, a debugger, or a test, because the question asked in interviews is how you would find it rather than what it was.
• Write the sentence you will use when you do not yet know the cause, one that names the next measurement instead of offering a guess.

Deliverable: A recorded debugging session with time-to-find per defect and the method that found each.

The failure that was yours

Keep one story where the bad call was yours rather than a dependency's or a manager's. Name the check that would have caught it, whether you added that check afterwards, and whether it has fired since. Answers that route blame outward end the conversation early; answers that end in a guardrail someone still relies on tend to open it up.

How do you handle large datasets in pandas that exceed local memory?

Medium · behavioural and collaboration

Approach

1. Give the blast radius: what could have broken, and what you measured.
2. Pick a story where you made the decision, not one where you watched it.
3. Close with what you would do differently, concretely.

Follow-up

• What would you do differently if you ran that again?
• How did you know your change caused the improvement?

Estimate work you have never done and defend the range

Hard · estimation · backfills · expand-contract

You are asked to estimate a change you have never attempted: add a column to a 100-million-row table, populate it, move reads across, and drop the old shape. Give a range with the assumptions that generate it, including batch size, the signal your backfill throttles on, and wall-clock hours, and name the three unknowns that would move the number most. Then describe a real estimate you gave under comparable ignorance: how you expressed its uncertainty, what you committed to, and how wrong you turned out to be.

Approach

1. Decompose into independently deployable steps before estimating anything: add the column nullable, write both shapes, backfill in batches, verify, move reads, stop writing the old shape, drop it. That is four deploys spread over days, and the calendar estimate is dominated by them rather than by the loop's runtime.
2. Do the arithmetic aloud for the part that has arithmetic in it: batch size times number of batches times per-batch duration, at a write rate the primary can absorb alongside roughly 1.2k writes per second of production traffic. The loop is throttled by replication lag and lock waits, not by how fast it can issue statements.
3. Price the schema step by its lock rather than its statement duration. In PostgreSQL an ALTER TABLE taking ACCESS EXCLUSIVE waits for every open transaction on that table while later queries queue behind it, so a millisecond change issued during a thirty-second analytics query stalls that table for thirty seconds. Adding a nullable column with a non-volatile default avoids a rewrite from version 11; a new index wants CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block and leaves an invalid index behind if it fails.
4. Express the answer as a range whose endpoints each trace to a stated assumption, then name the cheapest experiment that collapses it, which is almost always running one real batch against the real table and multiplying.
5. Commit to a checkpoint rather than a completion date: the day you report a measured number from that first batch. That is a promise you can keep under uncertainty, and it is what the asker actually needs in order to plan.

Follow-up

• How do you verify the backfill genuinely finished, given rows written by production traffic while it ran?
• Where does the backfill resume from after a worker is killed mid-batch, and what makes that resume point trustworthy?
• Your first batch comes back ten times slower than assumed. What do you tell the person waiting on the estimate, and when?

Narrate an outage you owned from page to postmortem

Hard · incident response · blast radius · postmortems

Pick an incident you personally drove, ideally one where writes were affected rather than reads. In six to eight minutes: state the symptom as it first appeared on a dashboard, the blast radius you established before you knew the cause, the mitigation you applied and when, the mechanism you eventually proved, and the follow-up that would prevent a repeat. Bring numbers: error rate, tenants affected, minutes to mitigate, minutes to resolve. If you cannot name what you measured, choose a different incident.

Approach

1. Open on the signal rather than the cause: which metric at which percentile moved, on which service, at what time, so the listener follows the same evidence you had rather than a conclusion you already reached.
2. Separate mitigation from diagnosis out loud. State what you did to stop the bleeding (flag off, shed traffic, drain a lease, roll back a deploy) and say plainly that you did it before the mechanism was known, because those are two jobs with different deadlines.
3. Establish blast radius in countable terms: how many tenants, how many writes, and crucially whether the effect was loss or only delay. An append-only revision table or a pending outbox row means the change survived and the projection was merely behind, which is a repair rather than a data-loss incident.
4. Prove the mechanism instead of asserting it. Name the trace span that grew, the plan that flipped to a sequential scan, the lease that expired, plus one alternative you ruled out and the signal that stayed flat while you ruled it out.
5. Close on the durable fix and its cost, distinguishing what landed that week from what needed an expand-and-contract migration across several deploys, and say which of the two you actually finished.

Follow-up

• What would you do differently in the first five minutes, given the same dashboard and no more information?
• Which follow-up action did you deliberately not take, and why was dropping it the right call?
• How did you convince yourself the mitigation was safe to apply while the cause was still unknown?