Interview guides · Samsung Ads
Samsung Ads Machine Learning Engineer Interview Questions & Guide 2026
Samsung Ads Machine Learning Engineer interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.
The 60-second brief
A Machine Learning Engineer at Samsung Ads plays a critical role in shaping the future of connected television (CTV) and smart media advertising. Operating at the intersection of big data, advanced modeling, and real-time systems, you will build and scale the algorithms that power personalized ad experiences for millions of users globally. The work here directly impacts how audiences engage with content on Samsung Smart TVs and partner devices, making your contributions highly visible and strategically vital to the business.
Seniority moves the scope further than the words in the title do. An earlier-career loop mostly checks that you implement something correctly and can reason about its cost, while a senior loop checks that you can pick between two defensible designs and say what you gave up.
Focus areas
- Detect concurrent edits instead of losing writes
- Bound every outbound call with a timeout
- Paginate large result sets with keyset cursors
Overview
The engineering challenges at Samsung Ads are distinct due to the sheer scale of the data ecosystem. You will work with massive, high-velocity data streams, including Automatic Content Recognition (ACR) telemetry, to develop models for real-time bidding (RTB), audience segmentation, and predictive targeting. Solving these problems requires a deep understanding of distributed systems, high-throughput model deployment, and state-of-the-art deep learning architectures.
Joining this team means you will collaborate closely with data scientists, software developers, and product managers to transition complex models from research to production. Your ability to write clean, production-grade code while maintaining rigorous scientific standards will directly influence monetization strategies and user satisfaction across the entire ecosystem.
The Samsung Ads interview process
Samsung Ads candidates report 4 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.
Recruiter Screen
The title covers product work, platform work, infrastructure, mobile and frontend, and those are different jobs with different loops behind them. A screening call is the cheapest place to find out which one the seat is, and asking reads as experienced rather than fussy. The questions that separate them: what the team is on call for, what the last three projects were, and whether any round happens inside an existing repository instead of a blank file. Then say which of that you have done and which you have not. Claiming the whole posting is the fastest way to be found out one round later.
What to demonstrate
- Whether you can locate your experience inside one flavour of the role honestly instead of claiming the entire requirements list
- Whether you name what you have not done, which an experienced screener reads as a level signal and can plan the loop around
- Whether what you want next matches what the seat is: someone who wants greenfield work landing on a team that mostly operates an existing system is a hire that leaves within the year
How to prepare
- Mark every line of the posting as done, adjacent or new, and write one sentence for each adjacent line naming the closest thing you actually built
- Split your last two years into rough percentages across feature work, operating and debugging live systems, and design or review, so a question about scope gets numbers rather than adjectives
- Bring three questions that discriminate between seats: what the team is paged for, how much of the work is changing existing code versus standing up something new, and what shipped in the last quarter
Technical Assessment
Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.
What to demonstrate
- Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
- Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
- Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
- Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply
How to prepare
- For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
- For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
- Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last
Core Technical Rounds
Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.
What to demonstrate
- Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
- Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
- Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
- Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply
How to prepare
- For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
- For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
- Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last
Hiring Manager Conversations
Part of what is being decided is what your manager's week looks like once you are on the team: how they find out that something you shipped is broken, and whether they hear it from you or from a customer. The material that moves this round is therefore not the launch, it is the week after. Describe how you knew the change was working, which number you watched and for how long, and what you would have reverted to. A project that ends at the word shipped leaves the manager guessing about the part they care about most.
What to demonstrate
- How a change of yours was verified in production, and whether that check existed before the deploy or was assembled afterwards once something looked wrong
- Whether the rollback story is specific, including the case where a revert is not enough: once a migration has dropped a column or rewritten data, going back is its own change with its own risk
- Whether you can describe an incident you caused, how it was found, and how much time passed between it starting and anyone noticing
- Whether bad news in your account travels early and from you, or consistently arrives via somebody else
How to prepare
- For your last significant change, write down what you watched after the deploy, for how long, and the number that would have made you revert. If nothing was watched, say that plainly instead of inventing a dashboard.
- Work out the rollback answer for a change that touched stored data, and be able to say what made it reversible or what you would have had to do instead. That distinction is a level signal on its own.
- Write the two-minute version of an incident you owned: what broke, how it surfaced, what you did in the first ten minutes, and the change that stopped it recurring. Get the timeline straight, because this is the story most likely to be interrupted with questions.
Where candidates lose points
Choosing an index from the columns a query mentions rather than from how it filters and orders
A composite B-tree index on (a, b, c) can be seeked only as a left prefix: equality on a, then equality on b, then a range or an ordering on c. A query that filters on b alone cannot seek into it at all and at best gets a full scan of the index; a query that filters a and ranges on b gets no benefit from c, because the index is only sorted by c within a fixed (a, b) pair. The practical consequence is that one index per column is close to useless for multi-predicate queries while a single correctly ordered composite index turns a scan into a lookup. The ordering half is what gets missed: if the index cannot satisfy the ORDER BY, the database must read every matching row and sort before the limit can apply, so a LIMIT 20 over a million matching rows still reads a million rows.
Assuming an isolation level prevents the anomaly you actually have
Isolation levels are named by the SQL standard but implemented differently, so any claim about one is only true of a named engine. PostgreSQL defaults to READ COMMITTED, where every statement takes a fresh snapshot, so two statements inside one transaction can legitimately disagree about the same row. Its REPEATABLE READ is snapshot isolation: it removes non-repeatable and phantom reads but permits write skew, where two transactions each read a set, each conclude their own write is safe, both commit, and the combined result violates a constraint that no single row expresses. Only SERIALIZABLE closes that, and it closes it by aborting a transaction with a serialization failure (SQLSTATE 40001), which means the guarantee is theoretical unless the application has a retry loop. InnoDB's REPEATABLE READ is a different mechanism again - plain SELECTs read a consistent snapshot while locking reads and writes see the latest committed row - so a read-modify-write inside one transaction can act on a value that the transaction's own earlier SELECT never returned.
Saying 'eventually consistent' without naming the anomaly a user would see
Describe the concrete symptom you are choosing to accept: the author reloads and their own comment is missing for two seconds, or two devices show different balances for a minute. The class of consistency model is a technical label; the tolerable anomaly is the actual product decision.
Writing code before the input contract is pinned down
Before the first line, state the types, the size bounds, whether duplicates, negatives or an empty input are possible, whether the input is sorted, whether you may mutate it, and what the function returns when nothing matches. Every one of those answers changes the code, and discovering one at minute twenty costs a rewrite you no longer have time for.
Questions & practice
How do you address extreme class imbalance when training a model to predict rare events like ad clicks or conversions?
Medium · machine learning fundamentals
Approach
- State the learning problem: the label, the unit of prediction and how the model is used.
- Name the simplest model that could work and what would make you move past it.
- Pick the metric from the cost of each error type, not from habit.
Follow-up
- Where could label leakage enter this setup?
- How would you know the model is overfitting?
Explain the mathematical difference between L1 and L2 regularization, and describe how they impact feature selection in high-dimensional ad datasets.
Medium · machine learning fundamentals
Approach
- Say how you would validate it, and where leakage could enter the split.
- Pick the metric from the cost of each error type, not from habit.
- Name the simplest model that could work and what would make you move past it.
Follow-up
- What changes if the classes are heavily imbalanced?
- How would you know the model is overfitting?
What are the trade-offs between using offline evaluation metrics like AUC-ROC versus online A/B testing metrics for recommendation systems?
Medium · machine learning fundamentals
Approach
- Say how you would validate it, and where leakage could enter the split.
- Name the simplest model that could work and what would make you move past it.
- State the learning problem: the label, the unit of prediction and how the model is used.
Follow-up
- Where could label leakage enter this setup?
- How would you know the model is overfitting?
Explain the mechanics of backpropagation in a deep neural network and how you would mitigate the vanishing gradient problem.
Medium · machine learning fundamentals
Approach
- Pick the metric from the cost of each error type, not from habit.
- State the learning problem: the label, the unit of prediction and how the model is used.
- Name the simplest model that could work and what would make you move past it.
Follow-up
- How would you know the model is overfitting?
- Where could label leakage enter this setup?
Solve a medium-level array manipulation problem to optimize budget allocation across multiple ad campaigns.
Medium · coding and algorithms
Approach
- Walk one small example through your approach before writing the whole thing.
- Restate the input: its shape, its size, and what is guaranteed about it.
- Name the brute-force solution and its complexity before improving on it.
Follow-up
- What is the worst case, and how likely is it on real data?
- How does this change if the input no longer fits in memory?
Given a directed graph representing user interactions, write an algorithm to detect if there are any cyclical patterns in the ad delivery path.
Medium · coding and algorithms
Approach
- Walk one small example through your approach before writing the whole thing.
- State the target complexity and say which constraint rules the naive version out.
- Name the brute-force solution and its complexity before improving on it.
Follow-up
- What is the worst case, and how likely is it on real data?
- How does this change if the input no longer fits in memory?
Canonicalise a request body into a stable idempotency fingerprint
Medium · parsing · canonicalisation · hashing
idempotency_key.request_fingerprint is a SHA-256 over the method, path and canonicalised body, and a retry whose fingerprint differs must be rejected with 422 rather than served the stored response. Write the canonicaliser. Bodies are JSON up to 256 KB nested at most 32 levels; clients vary key order, whitespace and unicode escaping, and some send 64-bit ids as JSON numbers. Produce a deterministic byte string such that semantically identical bodies match and any semantic difference does not. State your complexity and name two normalisations you refuse to perform.
Approach
- Parse once into a tree, then re-serialise under fixed rules: object keys sorted, array order preserved, one escaping convention, no insignificant whitespace. Parsing is O(n) and sorting keys is O(k log k) per object, so O(n log n) overall with O(depth) stack, and the 32-level cap is enforced during parsing because hostile nesting is how a canonicaliser becomes a stack overflow.
- Sort keys by their UTF-8 bytes and say why the obvious implementation is wrong in some runtimes: a default string comparison that orders by UTF-16 code units places surrogate pairs, meaning code points from U+10000 up, below U+E000 to U+FFFF, which is not UTF-8 byte order, so two services written in different languages disagree on the same document.
- Do not re-encode numbers through a double. IEEE-754 binary64 represents integers exactly only up to 2^53, so normalising a 19-digit id through a float changes it, and 1 against 1.0 cannot be reconciled without deciding whether they are the same value. Preserve the literal token, and require ids as strings at the API boundary if you want them comparable.
- Reject duplicate keys rather than picking one. JSON permits them and parsers disagree, most keeping the last, so any choice you make ties the fingerprint to a parser detail that the code handling the request does not necessarily share.
- Frame the hash preimage so concatenation cannot collide: delimit or length-prefix the method, path and body, otherwise one request's fields can be rearranged into another request with the same byte stream and the same fingerprint.
- Name the refusals and their consequence: no case folding, no dropping of null-valued keys, no Unicode normalisation. Each makes two different requests fingerprint alike, and the resulting failure is the worst one this table has, since the second request is answered with the first one's stored response and its effect never happens.
Worked solution (25 min)
- Write the serialiser: recursive emit with a depth counter, objects sorted by UTF-8 key bytes, arrays in order, strings escaped by one fixed rule, numbers emitted as their original token.
- Run it over three bodies: the same object with keys reordered, the same object with \u0041 written as A, and one with a nested array reversed. The first two must produce identical bytes and the third must not.
- Take the id 9007199254740993, round-trip it through a double, show it returns as 9007199254740992, then state the rule that prevents this.
- Define the hash preimage explicitly with its delimiters, and construct a pair of (path, body) inputs that would collide without them.
Expected result: A canonicaliser that is O(n log n) in body size with an enforced depth cap, sorts keys by UTF-8 byte order, preserves array order, keeps number literals verbatim, rejects duplicate keys, and feeds a delimited preimage to SHA-256, together with a stated list of normalisations deliberately not performed and the failure each would cause.
Verify it yourself
- Key reordering and whitespace changes produce identical fingerprints while reordering an array does not.
- A body containing an integer above 2^53 fingerprints identically after a parse and re-serialise round trip.
- Two requests differing only in a field the endpoint ignores produce different fingerprints, and you can say why that is the safe direction to err.
Follow-up
- A client sends the same logical request with an extra field your API ignores. Same key, different fingerprint, so you return 422. Is that the right answer?
- Where does the fingerprint get computed relative to request decompression and the body-size limit?
- The endpoint takes 1,000 requests per second with 256 KB bodies. What does hashing cost, and does it belong at the edge or in the core service?
Keep soft-deleted accounts from blocking re-registration
Easy · soft delete · unique constraints · engine differences
app_user holds user_id, tenant_id, email CITEXT, password_hash (NULL for SSO principals), email_verified_at, auth_version, status ('invited','active','suspended','deactivated'), created_at, updated_at, deleted_at. Two live accounts for one address inside a tenant must be impossible, but an address freed by a soft delete must be reusable, and the same tenant may delete and re-register it repeatedly. Write the uniqueness DDL for PostgreSQL 16, then the equivalent for MySQL 8 where partial indexes do not exist, and say what each permits once three deleted rows already hold that address.
Approach
- Start from what is actually unique: not (tenant_id, email), but (tenant_id, email) among live rows. PostgreSQL says that directly — CREATE UNIQUE INDEX app_user_live_email ON app_user (tenant_id, email) WHERE deleted_at IS NULL. A full constraint over the same two columns burns the address permanently the first time someone deletes an account.
- Keep case-insensitivity in the type or the index, never in the application: CITEXT as given, or UNIQUE (tenant_id, lower(email)) as an expression index where the extension is unavailable. A case-sensitive unique column is exactly how two accounts for one human appear.
- For MySQL 8 the predicate has to move inside the key: add a discriminator column that is a constant 0 while the row is live and is set to user_id on delete, with UNIQUE (tenant_id, email, deleted_marker). Live rows share the constant and still collide; deleted rows differ from each other and stop colliding.
- State the NULL variant and its dependency: leaving the marker NULL for deleted rows also works, because a unique index treats NULLs as distinct — true in MySQL, and true in PostgreSQL only under the default NULLS DISTINCT, which PostgreSQL 15 lets you reverse. Check the polarity against the three existing deleted rows: constant-on-live is what preserves the collision you want, and reversing it silently admits duplicate live accounts.
- Say what a soft delete must do besides setting deleted_at: increment auth_version so existing tokens stop validating, leave resource.owner_user_id and resource_revision.actor_user_id intact, and accept that the address is retained — erasure is a different requirement answered by scrubbing the column, not by a DELETE that would break those references.
Worked solution (20 min)
- Create the PostgreSQL partial unique index, insert a live row, soft delete it, and insert the same address again.
- Repeat the delete-and-reinsert cycle three times and confirm three deleted rows coexist with exactly one live row.
- Write the MySQL form with the discriminator, then deliberately reverse the polarity so live rows carry NULL, and show two live duplicates commit.
- Attempt a second live insert on both engines and map the resulting 23505 / ER_DUP_ENTRY to the 409 the handler should return.
Expected result: PostgreSQL rejects only the second live row and admits unlimited deleted ones. MySQL behaves identically with UNIQUE (tenant_id, email, deleted_marker) where live = 0; with the polarity reversed, two live rows commit and nothing afterwards can tell them apart.
Verify it yourself
- Group live rows by (tenant_id, lower(email)) after a randomised insert/delete run — no group exceeds one.
- Inserting 'User@x.test' after 'user@x.test' is rejected; if it is accepted, the case handling is in application code rather than in the index.
- EXPLAIN a login lookup and confirm the partial index is chosen — it is only usable by queries that carry the deleted_at IS NULL predicate, so a query without it falls back to a scan.
Follow-up
- A deleted account re-registers with the same address the next day. Do the old resource rows follow the new user_id, and how does the API keep the two principals apart?
- How do you honour an erasure request while resource_revision.actor_user_id still references this table?
- What changes if a user may hold membership in two tenants?
Explain why the owner filter ignores the listing index
Medium · composite index · query plans · sort elimination
The only index on resource is (tenant_id, status, updated_at DESC, resource_id DESC). A new endpoint returns one user's resources across all statuses, newest created first: WHERE tenant_id = $1 AND owner_user_id = $2 ORDER BY created_at DESC LIMIT 20. On a tenant with 2M rows it takes 900 ms and EXPLAIN shows a sort above a large scan. Explain precisely why the existing index cannot serve it, give the index that can, and state which of these the new index still will not help: owner_user_id alone across tenants; the same query ordered by updated_at. PostgreSQL 16.
Approach
- Separate the two jobs an index does. For filtering, a composite btree is seekable only on a left prefix, so with no predicate on status the scan can at best range over tenant_id and test owner_user_id per row; PostgreSQL 16 has no btree skip scan to jump the unconstrained column.
- For ordering, the index is sorted by (status, updated_at) within a tenant and not by created_at, so the LIMIT cannot stop early: every matching row is read and then sorted. That is the 'Sort Method: top-N heapsort' line, and it is why the plan reads 2M rows to answer with 20.
- Derive the replacement from the access path — equality, equality, then the ordering column: CREATE INDEX CONCURRENTLY ON resource (tenant_id, owner_user_id, created_at DESC). The scan seeks to the (tenant, owner) range and walks 20 entries in order, so the Sort node disappears along with the row-read.
- Treat INCLUDE (title, status) as conditional, not free. An index-only scan still visits the heap for any row whose page is not marked all-visible, so on a table taking 1.2k writes/second the win depends on autovacuum keeping the visibility map current, and the wider index costs more on every insert.
- Answer the two negatives explicitly. owner_user_id alone is not a left prefix of the new index, so it degrades to a full scan of the index at best. Ordered by updated_at, the query still seeks on the (tenant, owner) pair but must sort, because only created_at is ordered within that pair.
- Measure both sides with EXPLAIN (ANALYZE, BUFFERS) and compare estimated against actual rows at the lowest node — a 2M-versus-200 misestimate there is usually what chose the plan, and adding an index will not fix a statistics problem.
Follow-up
- 90% of rows are status='active'. Would a partial index WHERE status = 'active' change your answer, and for which of the three queries?
- A dashboard runs this for 40 owners in one page load. What changes about the design?
- How do you roll this index out on a table taking 1.2k writes/second, and what does it cost on every insert from then on?
Design an efficient data structure to track the top-k most frequently viewed ad campaigns in real time.
Medium · system design
Approach
- Name the failure you are designing for, then the recovery path.
- Fix the scope first: who calls this, how often, and what they do when it fails.
- State the consistency you need, and where you are willing to be stale.
Follow-up
- How does this behave when that dependency is down for an hour?
- What would you drop to keep the system up under load?
Cache the tenant listing feed with a bounded staleness window
Easy · caching · cache invalidation · keyset pagination
GET /v1/resources returns one tenant's resources ordered by updated_at DESC, 20 per page, at 14k requests/second peak against a 120 ms p99. The table carries the index (tenant_id, status, updated_at DESC, resource_id DESC) and writes land on the primary at 1.2k/second. Design the read path: the pagination contract, the cache key and value, what a write invalidates, and the staleness a user can observe. State the request rate that actually reaches the database, and the one repopulation race that deleting on write does not close.
Approach
- Settle the pagination contract first, because it decides what is cacheable. OFFSET makes the database produce and discard the skipped rows, so page 500 costs five hundred pages of work, and rows inserted between two fetches shift across the boundary and are skipped or repeated with nothing in the response to reveal it. The cursor is the row value of the last row returned: WHERE tenant_id = $1 AND status = $2 AND (updated_at, resource_id) < ($3, $4) ORDER BY updated_at DESC, resource_id DESC LIMIT 21. That is a row-value comparison, not updated_at < $3 AND resource_id < $4, which is a different and wrong predicate.
- Confirm the index actually serves it: equality on the two leading columns, then a range on the pair that follows in exactly the index's sort order, so the plan is an index scan that touches 21 entries with no sort node. Requesting 21 to return 20 is how has_more is answered without a count. resource_id is not decoration - updated_at is not unique, and without the tie-break two rows sharing a timestamp at a page boundary are the skip that keyset pagination was adopted to remove.
- Key the cache on every value the predicate reads: tenant_id, status, cursor and page size. A key that omits tenant_id is a cross-tenant disclosure, and no test running against a single tenant's data will show it.
- Be honest that a write does not invalidate one key. An update moves its row to the head of the ordering, so it invalidates the first page and every cursor page whose range spans the row's old and new position, which is not enumerable. Cache the first page per (tenant_id, status) - that is where the traffic is - with a short TTL, invalidate it on write, and serve deep cursor pages uncached from a replica, since each is already a 21-row index scan and they are rare.
- Name the residual race and the real bound. A reader that loaded rows before the write can populate the cache after the invalidation deleted the key, so the delete is not a staleness bound; the TTL is. Choose the TTL as the staleness a listing can tolerate, and jitter expiries so a busy tenant's keys do not all expire together and stampede the replica. Then do the arithmetic: at an 85% hit rate, 14k requests/second is about 2.1k database reads/second across two replicas, and that is the number capacity planning uses.
Worked solution (20 min)
- Write the keyset predicate and match it column by column against the index, marking which columns are equality, which is the range, and which satisfies the ORDER BY.
- Compare rows examined for page 1 and page 500 under OFFSET and under keyset, and state both numbers.
- Write the cache key template and the first-page invalidation the write path performs.
- Write the interleaving in which a stale value is written into the cache after the invalidation, and identify what bounds it.
Expected result: Keyset pagination on the row value (updated_at, resource_id) served as a 21-entry index scan with no sort at any page depth. The first page per (tenant_id, status) is cached under a short jittered TTL and deleted on write; deeper cursor pages go to a replica uncached. Observable staleness is the TTL, not the invalidation, because a concurrent reader can repopulate the key afterwards. At an 85% hit rate roughly 2.1k reads/second reach the two replicas.
Verify it yourself
- The query plan shows an index scan with no Sort node, and rows examined equal to the limit, at page 1 and at page 500.
- Two rows sharing an updated_at value across a page boundary are returned exactly once across the two pages.
- Every cache key contains tenant_id.
- Removing the TTL leaves staleness unbounded even though every write still invalidates.
Follow-up
- The tenant writes and immediately lists. What does it see, and what is the smallest change that makes its own write visible without sending all 14k requests/second to the primary?
- A tenant has 4 million resources and a client walks every page nightly. What does that do to the cache hit rate, and should that traffic share this path at all?
- Sort order becomes configurable - by title, by created_at. What happens to the index set and to the cache key space?
Every query on one table stalls for forty seconds mid-deploy
Medium · lock contention · ddl · migrations · postgres
During a release on PostgreSQL, every query touching resource times out for about 40 seconds and then recovers with no intervention. The release ran one migration, ALTER TABLE resource ADD COLUMN archived_reason TEXT, and the migration log shows it completing in 6 ms. Unrelated tables showed no change in error rate. Explain how a 6 ms statement caused a 40-second stall, give the ordered checks you would run on a live system to confirm it, and give the migration procedure that prevents a repeat.
Approach
- Separate the statement's duration from the lock's duration. ADD COLUMN with no default is a catalogue-only change and genuinely runs in milliseconds, but it requires ACCESS EXCLUSIVE, and it cannot acquire that until every transaction already touching the table has finished.
- Account for the queueing, which is the part that surprises people. A lock request that is waiting blocks later requests for conflicting modes behind it rather than letting them overtake, so one long-open transaction holds the DDL and the DDL holds all the traffic. The stall length is set by the longest open transaction, not by the size of the change.
- Confirm on a live system in this order: pg_stat_activity for that table ordered by xact_start, looking for the oldest transaction and specifically for state = idle in transaction; then pg_locks where granted = false to find the waiter; then join them on pid to name blocker and blocked. pg_blocking_pids() does that join for you and is the fastest single call.
- Prevent rather than merely time it better. Set lock_timeout to a second or two on the migration session so the DDL abandons the queue after a bounded wait and is retried, instead of holding it for as long as the oldest transaction lives. Be exact about what that buys: queries arriving during the wait still queue behind the pending ACCESS EXCLUSIVE request, so each attempt costs them up to one lock_timeout of added latency. The outage goes from 40 seconds to about one second per attempt, not to zero. Also run migrations away from deploy-time peaks, and put a statement timeout and an idle-in-transaction timeout on the analytics role that opens the long transactions.
- Know the lock each change takes, since the mitigation differs by change. A column with a non-volatile default is a metadata-only change from PostgreSQL 11 and still needs the brief ACCESS EXCLUSIVE; an index needs CREATE INDEX CONCURRENTLY, which cannot run inside a transaction block and leaves an INVALID index to drop if it fails; a check or foreign key is added NOT VALID and then VALIDATE CONSTRAINT as a separate statement under a weaker lock.
Follow-up
- The same release also wants NOT NULL on that column. What is the sequence that gets there without a long lock?
- Your lock_timeout retry fails ten times in a row because the analytics transaction is always open. What do you change?
- How does this differ on MySQL with InnoDB online DDL, and what is the equivalent of the waiting-lock queue there?
Suggested practice checklist
Roughly ninety minutes on weeknights with one longer weekend block. The plan cuts scope rather than compressing everything, on the assumption that one thing finished per night beats four half-started.
Day 1 — Fix the scope and take a cold baseline
- Read the role description and write the three things the loop will almost certainly test, then write an explicit not-doing list and keep it visible all week.
- Take one twenty-five-minute coding problem and one fifteen-minute design prompt cold, and write the single sentence naming what blocked each, because those two sentences decide where the remaining evenings go.
- Set the week's rule: one thing finished every night, including the night you only have forty minutes.
Deliverable: A one-page scope with a not-doing list and two cold attempts, each carrying one sentence on what blocked it.
Day 2 — One pattern, written three times from blank
- Choose the single pattern most likely to appear in your loop and write it three times from an empty file rather than editing the previous attempt.
- On the third pass, write the invariant as a comment before the loop body and the complexity before the first line of code.
- Stop at ninety minutes even if the third version is imperfect, and write the one thing you would fix given another hour.
Deliverable: Three independent implementations of the same pattern plus a note on what changed between them.
Day 3 — One design, only to the depth you can defend
- Take one system shape and go only as far as requirements, interface and data model, refusing to draw a box you could not survive a follow-up about.
- Attach one number to each non-functional requirement, deriving it rather than asserting it, and write the assumption the number rests on.
- Write the one tradeoff you are choosing against and the observation that would make you reverse it.
Deliverable: One design at interface-and-schema depth with derived numbers and one written reversible tradeoff.
Day 4 — Only the fundamentals you will have to defend
- Write, in under two hundred words each, the answers to the two questions that follow almost any implementation: why this structure and not the obvious alternative, and what happens to this code at a hundred times the input.
- Write what an index actually costs: faster lookups on the indexed columns against a write that now maintains a second structure, plus the cases where the planner declines to use it anyway, low selectivity, or a predicate wrapping the column in a function.
- Delete any answer you cannot deliver aloud in under a minute, since an answer that needs reading is not an answer you have.
Deliverable: Three written answers, each under two hundred words and each timed aloud.
Day 5 — Your own work, timed
- Write a ninety-second and a four-minute version of your main project and time both aloud rather than reading them.
- Prepare the two follow-ups that always come: what you would do differently, and how you knew it worked.
- Put one number in the first sentence and be ready to say exactly where it came from and what it excludes.
Deliverable: Two timed narratives with one defensible number in the opening line.
Day 6 — The one full rehearsal, in the weekend block
- Run a sixty-minute mock covering a coding round and a design round in one sitting with no break, because sustained attention is the thing evenings have not trained.
- Immediately afterwards, and before hearing any feedback, write the three moments you lost the thread.
- Spend the rest of the block only on those three moments, and on nothing you merely feel shaky about.
Deliverable: Mock notes naming three failure moments with a specific fix written under each.
Day 7 — Taper
- Write the twenty-minute warm-up you will actually do on the morning: one problem you can already solve from a blank file, one design you can narrate, and nothing you have never seen.
- Re-read only your own notes from this week and open no new material.
- Write the logistics down: the editor or shared document you will be working in, whether execution and lookups are permitted, and the sentence you will use when you do not know something.
Deliverable: A one-page card holding the design structure, the project numbers, and the logistics.
Risky changes and the reversal path
For anything that touched live traffic, be ready to say how you would have undone it: a flag, a staged rollout, dual writes with the old path still authoritative. Once the old column is dropped or the source rows are overwritten there is no reverse, so name what you kept a copy of and for how long.
Tell callers you do not own that their integration breaks
Medium · deprecation · compatibility · stakeholders
A field in a write endpoint's response must change shape. You own the endpoint; you do not own the four internal callers or the outbound webhook consumers who read it. Describe a deprecation you were responsible for: what you shipped first, how you established who was actually reading the field, the window you gave and what set its length, what you did about the consumer who never moved, and how you decided removal was safe. Name the signal you used, not the announcement you sent.
Approach
- Establish the reader set empirically rather than from a wiki of owners: per-field usage counters keyed by principal, or access logs attributed to a consumer. State the blind spot of whichever you pick, since a consumer that reads the field only on a monthly job will not appear in a week of logs.
- Ship additive first. Populate the new field alongside the old one so no reader is forced to move, which is also what keeps a rolling deploy safe, because old and new instances answer the same requests at the same time and a rollback must still find the old shape present.
- Set the window from the slowest legitimate consumer's release cadence, not from your calendar, and decide separately what to do for a consumer with no release process at all, such as an external webhook endpoint you can only email.
- Convert silence into evidence before you rely on it: a short, low-traffic removal window that makes a still-dependent consumer fail visibly and loudly while you are watching, rather than at three in the morning after you have moved on.
- State the removal criterion as a measurement with a duration attached, such as observed reads at zero across a full billing cycle, and keep the change reversible for one release after removal.
Follow-up
- How would you detect a consumer that reads the field only during a monthly export?
- One caller refuses to move and has a commercial relationship behind it. What changes in your plan and what does not?
- After removal, what makes the change irreversible, and how long before you cross that line?
Reverse your own decision and price the reversal
Medium · reversibility · measurement · migrations
Describe a technical decision you made and later reversed. Pick one that cost something: a service you split and merged back, a cache you added and removed, an index you created that pushed the planner onto a worse plan, a projection you rebuilt from scratch. State what you believed when you decided, the measurement that changed your mind, how long the wrong version ran in production, and what the reversal cost in migrations, dual writes, and a deprecation window for callers you did not own.
Approach
- State the original rationale without irony, in the version you would still defend given what was known then. If it is not defensible, the story is about carelessness rather than judgement, and a different example serves you better.
- Give the measurement that moved with a before and after: the p99 that did not improve, the cache hit rate that sat at 40%, the plan that flipped to a sequential scan once the table passed a size you can name.
- Cost the reversal in steps, not adjectives: expand-and-contract deploys, the dual-write window, the callers who had to be notified, the rows already written in the wrong shape that had to be backfilled or abandoned.
- Distinguish reversal from rewrite by naming what you kept. Most good reversals preserve the schema or the interface and undo one decision inside it, which is also why they were affordable.
- Finish on the process change: the smallest experiment that would have produced the same measurement in a day, and why you did not run it the first time.
Follow-up
- What in that decision was irreversible, and did you know it was irreversible when you made it?
- How did you tell the people who had already built on top of the original decision?
- What do you now measure before committing to a change of this size?
Narrate an outage you owned from page to postmortem
Hard · incident response · blast radius · postmortems
Pick an incident you personally drove, ideally one where writes were affected rather than reads. In six to eight minutes: state the symptom as it first appeared on a dashboard, the blast radius you established before you knew the cause, the mitigation you applied and when, the mechanism you eventually proved, and the follow-up that would prevent a repeat. Bring numbers: error rate, tenants affected, minutes to mitigate, minutes to resolve. If you cannot name what you measured, choose a different incident.
Approach
- Open on the signal rather than the cause: which metric at which percentile moved, on which service, at what time, so the listener follows the same evidence you had rather than a conclusion you already reached.
- Separate mitigation from diagnosis out loud. State what you did to stop the bleeding (flag off, shed traffic, drain a lease, roll back a deploy) and say plainly that you did it before the mechanism was known, because those are two jobs with different deadlines.
- Establish blast radius in countable terms: how many tenants, how many writes, and crucially whether the effect was loss or only delay. An append-only revision table or a pending outbox row means the change survived and the projection was merely behind, which is a repair rather than a data-loss incident.
- Prove the mechanism instead of asserting it. Name the trace span that grew, the plan that flipped to a sequential scan, the lease that expired, plus one alternative you ruled out and the signal that stayed flat while you ruled it out.
- Close on the durable fix and its cost, distinguishing what landed that week from what needed an expand-and-contract migration across several deploys, and say which of the two you actually finished.
Follow-up
- What would you do differently in the first five minutes, given the same dashboard and no more information?
- Which follow-up action did you deliberately not take, and why was dropping it the right call?
- How did you convince yourself the mitigation was safe to apply while the cause was still unknown?