Interview guides · Canonical
Canonical Machine Learning Engineer Interview Questions & Guide 2026
Canonical Machine Learning Engineer interview guide: reported rounds, practice questions with worked solutions and a 7-day prep plan.
The 60-second brief
At Canonical, a Machine Learning Engineer does not simply build and train isolated models; you build the robust, secure, and highly scalable infrastructure that powers machine learning at an enterprise grade. Canonical is the company behind Ubuntu, the operating system that runs a vast portion of the world's public cloud workloads and developer desktops. In this role, your mission is to enable organizations to deploy, manage, and scale machine learning workflows seamlessly across multi-cloud, hybrid, and edge environments.
For a design discussion, a catalogue of architectures is worth less than the ability to turn a vague requirement into a data model and an API contract. A box diagram with no schema under it collapses at the first follow-up question.
Focus areas
- Choose indexes from the query's access path
- Trace a symptom to a mechanism under load
- Bound every outbound call with a timeout
Overview
You will work closely with the open-source community and enterprise partners to design and deliver secure, reliable, and standardized MLOps platforms. This involves packaging, integrating, and optimizing complex open-source technologies like Kubeflow, MLflow, PyTorch, and TensorFlow into Canonical's software ecosystem, such as Charmed Kubeflow. Your work directly impacts how thousands of companies transition their artificial intelligence initiatives from experimental notebooks to secure, production-grade automated pipelines.
This position demands a rare combination of deep software engineering discipline, systems-level thinking, and a comprehensive understanding of the machine learning lifecycle. You will tackle complex challenges surrounding container orchestration, high-performance computing, and cross-platform compatibility. It is an intellectually rigorous environment where your engineering decisions will influence the open-source ecosystem and define the standard for enterprise MLOps.
The Canonical interview process
Canonical candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.
Written Assessment
What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.
What to demonstrate
- Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
- Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
- Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly
How to prepare
- Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
- Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
- Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live
Technical Screening
What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.
What to demonstrate
- Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
- Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
- Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly
How to prepare
- Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
- Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
- Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live
Technical Interviews
What this round decides is narrow: whether you can produce code that runs and is correct on inputs nobody showed you. An elegant solution that does not compile scores below a plain one that does, so write a correct brute force first, say out loud that you know its cost, and improve it with the working version still on screen. What separates strong answers is who finds the broken case. Trace your own code against an empty input, a single element, and duplicate keys before you say you are finished, because being told is far more expensive than noticing.
What to demonstrate
- Whether degenerate inputs get checked without being asked for: an empty collection, one element, every element equal, and the extreme value the input type allows
- Whether the complexity you state matches the code you actually wrote, including a sort or a copy sitting inside a loop
- Whether the finished answer is verified against the worked examples before you call it done, rather than assumed correct because the code reads correctly
How to prepare
- Take five problems you have already solved and, without running anything, write down what each returns for empty input, a single element, and all-duplicates. Then run them and count how many you predicted wrong.
- Drill the brute force as its own skill: on ten problems, write only the obviously-correct slow version and time how long it takes to get it passing. If that is more than a few minutes, that is what to practise, not the optimal version.
- Add a fixed last step before you submit anything, reading only the loop bounds and the initial value of each accumulator, which is where most off-by-one errors live
Where candidates lose points
Choosing an index from the columns a query mentions rather than from how it filters and orders
A composite B-tree index on (a, b, c) can be seeked only as a left prefix: equality on a, then equality on b, then a range or an ordering on c. A query that filters on b alone cannot seek into it at all and at best gets a full scan of the index; a query that filters a and ranges on b gets no benefit from c, because the index is only sorted by c within a fixed (a, b) pair. The practical consequence is that one index per column is close to useless for multi-predicate queries while a single correctly ordered composite index turns a scan into a lookup. The ordering half is what gets missed: if the index cannot satisfy the ORDER BY, the database must read every matching row and sort before the limit can apply, so a LIMIT 20 over a million matching rows still reads a million rows.
Shipping a migration and the code that depends on it as a single change
During any rolling deploy, and for as long as a rollback remains possible, old and new code execute against the same schema at the same time. A migration that drops or renames a column breaks every instance that has not restarted yet, and code that requires a column the migration has not applied breaks every instance that restarted early. The discipline is expand then contract: add the new column nullable, write both shapes, backfill in batches, move reads across once the backfill is verified, and only then stop writing the old shape and drop it - four deploys, usually spread over days. It feels disproportionate until the first rollback, at which point it is the only reason the previous version still runs.
Treating a network call as though it were a local function call
A remote call can be slow, fail, or return after you stopped waiting, so name the timeout, the retry policy, and what the caller sees while the dependency is down. A call with no timeout turns one slow dependency into an exhausted thread or connection pool in every service upstream of it.
Arguing past a hint
When the interviewer asks what happens for a particular input or floats a different data structure, stop and take it seriously; it is almost always a correction rather than idle curiosity. Talking over it converts a recoverable wrong turn into a data point about how you handle review.
Questions & practice
What are your ideas about Canonical's business model, open-source strategy, and our current positioning in the enterprise market?
Medium · machine learning fundamentals
Approach
- State the learning problem: the label, the unit of prediction and how the model is used.
- Name the simplest model that could work and what would make you move past it.
- Say how you would validate it, and where leakage could enter the split.
Follow-up
- What changes if the classes are heavily imbalanced?
- Where could label leakage enter this setup?
Optimize a search algorithm to locate specific system log patterns across high-volume, distributed server clusters under strict memory constraints.
Medium · machine learning fundamentals
Approach
- Name the simplest model that could work and what would make you move past it.
- Say how you would validate it, and where leakage could enter the split.
- Pick the metric from the cost of each error type, not from habit.
Follow-up
- Where could label leakage enter this setup?
- How would you know the model is overfitting?
Explain the process of containerizing a complex machine learning pipeline and ensuring its security and minimal image size.
Medium · machine learning fundamentals
Approach
- Name the simplest model that could work and what would make you move past it.
- Say how you would validate it, and where leakage could enter the split.
- State the learning problem: the label, the unit of prediction and how the model is used.
Follow-up
- What changes if the classes are heavily imbalanced?
- Where could label leakage enter this setup?
How would you design a highly available, multi-tenant Kubeflow deployment on a Kubernetes cluster?
Medium · machine learning fundamentals
Approach
- Say how you would validate it, and where leakage could enter the split.
- Pick the metric from the cost of each error type, not from habit.
- Name the simplest model that could work and what would make you move past it.
Follow-up
- What changes if the classes are heavily imbalanced?
- How would you know the model is overfitting?
Given an array of integers representing resource demands, find the maximum sum of a contiguous subarray using an optimal time complexity.
Medium · coding and algorithms
Approach
- Name the brute-force solution and its complexity before improving on it.
- State the target complexity and say which constraint rules the naive version out.
- Walk one small example through your approach before writing the whole thing.
Follow-up
- What is the worst case, and how likely is it on real data?
- Which test case would catch an off-by-one here?
Implement an algorithm to detect cycles in a directed graph representing package dependencies, and return a valid compilation order.
Medium · coding and algorithms
Approach
- State the target complexity and say which constraint rules the naive version out.
- Restate the input: its shape, its size, and what is guaranteed about it.
- Walk one small example through your approach before writing the whole thing.
Follow-up
- Which test case would catch an off-by-one here?
- What is the worst case, and how likely is it on real data?
Canonicalise a request body into a stable idempotency fingerprint
Medium · parsing · canonicalisation · hashing
idempotency_key.request_fingerprint is a SHA-256 over the method, path and canonicalised body, and a retry whose fingerprint differs must be rejected with 422 rather than served the stored response. Write the canonicaliser. Bodies are JSON up to 256 KB nested at most 32 levels; clients vary key order, whitespace and unicode escaping, and some send 64-bit ids as JSON numbers. Produce a deterministic byte string such that semantically identical bodies match and any semantic difference does not. State your complexity and name two normalisations you refuse to perform.
Approach
- Parse once into a tree, then re-serialise under fixed rules: object keys sorted, array order preserved, one escaping convention, no insignificant whitespace. Parsing is O(n) and sorting keys is O(k log k) per object, so O(n log n) overall with O(depth) stack, and the 32-level cap is enforced during parsing because hostile nesting is how a canonicaliser becomes a stack overflow.
- Sort keys by their UTF-8 bytes and say why the obvious implementation is wrong in some runtimes: a default string comparison that orders by UTF-16 code units places surrogate pairs, meaning code points from U+10000 up, below U+E000 to U+FFFF, which is not UTF-8 byte order, so two services written in different languages disagree on the same document.
- Do not re-encode numbers through a double. IEEE-754 binary64 represents integers exactly only up to 2^53, so normalising a 19-digit id through a float changes it, and 1 against 1.0 cannot be reconciled without deciding whether they are the same value. Preserve the literal token, and require ids as strings at the API boundary if you want them comparable.
- Reject duplicate keys rather than picking one. JSON permits them and parsers disagree, most keeping the last, so any choice you make ties the fingerprint to a parser detail that the code handling the request does not necessarily share.
- Frame the hash preimage so concatenation cannot collide: delimit or length-prefix the method, path and body, otherwise one request's fields can be rearranged into another request with the same byte stream and the same fingerprint.
- Name the refusals and their consequence: no case folding, no dropping of null-valued keys, no Unicode normalisation. Each makes two different requests fingerprint alike, and the resulting failure is the worst one this table has, since the second request is answered with the first one's stored response and its effect never happens.
Worked solution (25 min)
- Write the serialiser: recursive emit with a depth counter, objects sorted by UTF-8 key bytes, arrays in order, strings escaped by one fixed rule, numbers emitted as their original token.
- Run it over three bodies: the same object with keys reordered, the same object with \u0041 written as A, and one with a nested array reversed. The first two must produce identical bytes and the third must not.
- Take the id 9007199254740993, round-trip it through a double, show it returns as 9007199254740992, then state the rule that prevents this.
- Define the hash preimage explicitly with its delimiters, and construct a pair of (path, body) inputs that would collide without them.
Expected result: A canonicaliser that is O(n log n) in body size with an enforced depth cap, sorts keys by UTF-8 byte order, preserves array order, keeps number literals verbatim, rejects duplicate keys, and feeds a delimited preimage to SHA-256, together with a stated list of normalisations deliberately not performed and the failure each would cause.
Verify it yourself
- Key reordering and whitespace changes produce identical fingerprints while reordering an array does not.
- A body containing an integer above 2^53 fingerprints identically after a parse and re-serialise round trip.
- Two requests differing only in a field the endpoint ignores produce different fingerprints, and you can say why that is the safe direction to err.
Follow-up
- A client sends the same logical request with an extra field your API ignores. Same key, different fingerprint, so you return 422. Is that the right answer?
- Where does the fingerprint get computed relative to request decompression and the body-size limit?
- The endpoint takes 1,000 requests per second with 256 KB bodies. What does hashing cost, and does it belong at the edge or in the core service?
Replace offset paging on the resource feed with keyset
Easy · keyset pagination · composite index · sort order
resource holds resource_id, tenant_id, owner_user_id, title, body_ref, version, status ('draft','active','archived','deleted'), created_at, updated_at, deleted_at, with an index on (tenant_id, status, updated_at DESC, resource_id DESC). The listing endpoint returns active resources for one tenant, newest update first, 50 per page, today with LIMIT 50 OFFSET n. Tenants reach page 400 and rows are created while they read. Write the keyset query, define what the cursor carries and how it is encoded, and say which part of the index each predicate uses. Assume PostgreSQL 16.
Approach
- Name the two failures separately. OFFSET 20000 makes the server produce and discard 20,000 rows, so page cost grows with depth rather than with page size. Independently, any write that changes how many rows sort above the offset moves the window between two fetches, and the direction decides which anomaly you get: an insert lands at the head of updated_at DESC and pushes already-returned rows down past the boundary, so they are returned a second time; a delete above the offset, or a row whose updated_at is bumped above the cursor, pulls rows up and one is never returned at all. Nothing in the response reveals either.
- Write the seek: WHERE tenant_id = $1 AND status = 'active' AND (updated_at, resource_id) < ($2, $3) ORDER BY updated_at DESC, resource_id DESC LIMIT 50. The row-value comparison is one index range rather than a disjunction, and both columns are NOT NULL, which is what makes that comparison well defined.
- Map each predicate onto the index: tenant_id and status are equality on the leading columns, (updated_at, resource_id) is the range, and the ORDER BY matches the index order so no Sort node appears and the scan stops after 50 rows. The DESC in the definition only matters for mixed directions — a plain ascending btree on the same columns is read backwards for this query.
- Put both sort columns in the cursor and nothing the client can tamper with into another tenant: base64 of (updated_at, resource_id), validated server-side, with tenant_id taken from the principal.
- State the residual honestly. Keyset is stable against concurrent inserts and deletes, but not against a row whose updated_at changes mid-scroll — that row moves in the ordering and can be seen twice. If the feed must be a snapshot, order by an immutable key or bound the page set with updated_at <= the cursor's start value.
- Keep a total out of the page path. A tenant-wide COUNT(*) is the scan keyset just removed; fetch LIMIT 51 and return has_more instead.
Worked solution (25 min)
- Seed one tenant with 500k active resources, 2% of them sharing an identical updated_at.
- Time LIMIT 50 OFFSET 0 against OFFSET 20000 and record rows-read from EXPLAIN (ANALYZE, BUFFERS) for each.
- Page the whole set with the keyset query while an insert-only writer adds 100 rows/second, collecting resource_ids, and repeat the run with OFFSET.
- Repeat both runs under a second writer profile that also deletes 20 rows/second from pages already returned and bumps updated_at on 20 more, and diff each collected id set against the rows that existed for the whole run.
- Remove resource_id from the cursor so the seek degrades to updated_at < $2, and re-run the tie-heavy section of the feed.
Expected result: Keyset reads about 50 index entries per page and its latency is flat from page 1 to page 10,000; under both writer profiles the collected ids contain no duplicate and omit no row that existed for the whole scroll. OFFSET at page 400 reads 20,050 rows to return 50. Under the insert-only writer it repeats rows and skips none — an insert can only push a row down past the boundary — and only the second profile, whose deletes and updated_at bumps move rows up past the cursor, also makes it skip. Without the tie-break, the seek is updated_at < $2, which drops every still-unread row sharing the boundary timestamp.
Verify it yourself
- EXPLAIN shows an Index Scan with no Sort node — a Sort means the ORDER BY is not satisfied by the index.
- Buffers read per page stays constant as page depth grows.
- Ids collected across the scroll are unique, and complete for rows untouched during the run.
Follow-up
- The client asks for 'jump to page 400'. What do you offer instead, and what does the honest version cost?
- Sort order becomes user-selectable across four columns. How many indexes is that, and which would you refuse to add?
- What does the cursor do when the row it points at has since been deleted?
Denormalise tenant onto revisions and backfill it live
Hard · denormalisation · online migration · backfill
resource_revision (revision_id, resource_id, version, actor_user_id, change_kind, patch, request_id, created_at) has 400M rows and no tenant column; tenant_id lives only on resource. Two reads need it: a tenant-scoped audit feed ordered by created_at DESC, and an offboarding purge. Both join back to resource today. Justify adding tenant_id to resource_revision against those two reads, name the anomaly the copy introduces and the constraint that prevents it, then give the ordered migration for a live table taking 1.2k writes/second — the lock each step takes, how the backfill is batched, and where each step stops being reversible. PostgreSQL 16.
Approach
- Justify from the access path rather than from taste. Without the column, the audit feed either scans resource_revision by created_at and discards other tenants' rows, or resolves the tenant's resource_ids first and probes with them — both proportional to the tenant's whole history rather than to one page. With (tenant_id, created_at DESC, revision_id DESC) it is a seek that stops at 50 rows, and the purge becomes a ranged delete instead of a join.
- Name the cost exactly: a second copy of a fact can disagree with the first. Make the disagreement unwritable rather than documented — add UNIQUE (resource_id, tenant_id) on resource so it can serve as a foreign-key target, then FOREIGN KEY (resource_id, tenant_id) REFERENCES resource (resource_id, tenant_id) on the revision table. A revision can then only ever carry its parent's tenant.
- Step one, expand: ALTER TABLE resource_revision ADD COLUMN tenant_id BIGINT NULL, with no default, so it is a catalogue change and no rewrite. It still needs ACCESS EXCLUSIVE for an instant, and that instant queues behind the longest open transaction on the table while every later query queues behind it — set lock_timeout to 2s and retry rather than wait.
- Step two, dual-write: deploy the writer that populates tenant_id on every new revision while reads still use the join. Reversible by redeploying the previous build, because nothing reads the column yet.
- Step three, backfill: batch by primary key rather than by created_at so the cursor is dense and resumable — UPDATE resource_revision rr SET tenant_id = r.tenant_id FROM resource r WHERE r.resource_id = rr.resource_id AND rr.revision_id > $1 AND rr.revision_id <= $1 + 5000 AND rr.tenant_id IS NULL — committing per batch and persisting the cursor. Throttle on replica replay lag and on dead-tuple count, since each batch writes 5,000 new row versions. Run the backfill before the index exists so those updates can stay HOT.
- Step four, index then enforce then contract: CREATE INDEX CONCURRENTLY (cannot run inside a transaction block, scans the table twice, waits on open transactions, and leaves an INVALID index to drop concurrently if it fails); ADD CONSTRAINT ... CHECK (tenant_id IS NOT NULL) NOT VALID, then VALIDATE CONSTRAINT, which takes only SHARE UPDATE EXCLUSIVE, after which SET NOT NULL uses the validated check instead of re-scanning on PostgreSQL 12 and later. Only then move the audit reads onto the column and, in a later deploy, delete the join path.
Follow-up
- The backfill is half finished and a rollback is required. What state is the table in, and what does the previous build do with a half-populated column?
- How do you verify the backfill actually finished, given rows are still being inserted while it runs?
- A resource must now be movable between tenants. What does that do to the composite foreign key and to the revisions already written?
How do you manage and automate model deployment and monitoring to handle issues like data drift and concept drift in a production environment?
Medium · ml system design and production
Approach
- Name what you would monitor after launch and what triggers a retrain.
- Say where features come from at serving time and how they match training.
- Separate the offline training path from the online serving path.
Follow-up
- How would you detect drift before the metric drops?
- What happens when a feature is missing at serving time?
Why do you want to work specifically on open-source infrastructure and MLOps tools rather than focusing solely on training proprietary machine learning models?
Medium · ml system design and production
Approach
- Fix the product goal and the online metric before choosing any model.
- Name what you would monitor after launch and what triggers a retrain.
- Separate the offline training path from the online serving path.
Follow-up
- How would you detect drift before the metric drops?
- What happens when a feature is missing at serving time?
Detail your university experience, highlighting your major projects, thesis work, and how your academic focus prepared you for a career in systems engineering.
Medium
Approach
- Clarify what is being asked and what a complete answer contains.
- Say what you would check first and why it is the highest-information step.
- Work from the requirement backwards to the design.
Follow-up
- What assumption would you test first?
- How would you know your answer was wrong?
Cache the tenant listing feed with a bounded staleness window
Easy · caching · cache invalidation · keyset pagination
GET /v1/resources returns one tenant's resources ordered by updated_at DESC, 20 per page, at 14k requests/second peak against a 120 ms p99. The table carries the index (tenant_id, status, updated_at DESC, resource_id DESC) and writes land on the primary at 1.2k/second. Design the read path: the pagination contract, the cache key and value, what a write invalidates, and the staleness a user can observe. State the request rate that actually reaches the database, and the one repopulation race that deleting on write does not close.
Approach
- Settle the pagination contract first, because it decides what is cacheable. OFFSET makes the database produce and discard the skipped rows, so page 500 costs five hundred pages of work, and rows inserted between two fetches shift across the boundary and are skipped or repeated with nothing in the response to reveal it. The cursor is the row value of the last row returned: WHERE tenant_id = $1 AND status = $2 AND (updated_at, resource_id) < ($3, $4) ORDER BY updated_at DESC, resource_id DESC LIMIT 21. That is a row-value comparison, not updated_at < $3 AND resource_id < $4, which is a different and wrong predicate.
- Confirm the index actually serves it: equality on the two leading columns, then a range on the pair that follows in exactly the index's sort order, so the plan is an index scan that touches 21 entries with no sort node. Requesting 21 to return 20 is how has_more is answered without a count. resource_id is not decoration - updated_at is not unique, and without the tie-break two rows sharing a timestamp at a page boundary are the skip that keyset pagination was adopted to remove.
- Key the cache on every value the predicate reads: tenant_id, status, cursor and page size. A key that omits tenant_id is a cross-tenant disclosure, and no test running against a single tenant's data will show it.
- Be honest that a write does not invalidate one key. An update moves its row to the head of the ordering, so it invalidates the first page and every cursor page whose range spans the row's old and new position, which is not enumerable. Cache the first page per (tenant_id, status) - that is where the traffic is - with a short TTL, invalidate it on write, and serve deep cursor pages uncached from a replica, since each is already a 21-row index scan and they are rare.
- Name the residual race and the real bound. A reader that loaded rows before the write can populate the cache after the invalidation deleted the key, so the delete is not a staleness bound; the TTL is. Choose the TTL as the staleness a listing can tolerate, and jitter expiries so a busy tenant's keys do not all expire together and stampede the replica. Then do the arithmetic: at an 85% hit rate, 14k requests/second is about 2.1k database reads/second across two replicas, and that is the number capacity planning uses.
Worked solution (20 min)
- Write the keyset predicate and match it column by column against the index, marking which columns are equality, which is the range, and which satisfies the ORDER BY.
- Compare rows examined for page 1 and page 500 under OFFSET and under keyset, and state both numbers.
- Write the cache key template and the first-page invalidation the write path performs.
- Write the interleaving in which a stale value is written into the cache after the invalidation, and identify what bounds it.
Expected result: Keyset pagination on the row value (updated_at, resource_id) served as a 21-entry index scan with no sort at any page depth. The first page per (tenant_id, status) is cached under a short jittered TTL and deleted on write; deeper cursor pages go to a replica uncached. Observable staleness is the TTL, not the invalidation, because a concurrent reader can repopulate the key afterwards. At an 85% hit rate roughly 2.1k reads/second reach the two replicas.
Verify it yourself
- The query plan shows an index scan with no Sort node, and rows examined equal to the limit, at page 1 and at page 500.
- Two rows sharing an updated_at value across a page boundary are returned exactly once across the two pages.
- Every cache key contains tenant_id.
- Removing the TTL leaves staleness unbounded even though every write still invalidates.
Follow-up
- The tenant writes and immediately lists. What does it see, and what is the smallest change that makes its own write visible without sending all 14k requests/second to the primary?
- A tenant has 4 million resources and a client walks every page nightly. What does that do to the cache hit rate, and should that traffic share this path at all?
- Sort order becomes configurable - by title, by created_at. What happens to the index set and to the cache key space?
Listing latency scales with page size, not with filters
Easy · n+1 · latency · query batching
The tenant listing endpoint reads resource filtered by tenant_id and status, ordered by updated_at DESC, and returns each row plus the owner's display name from app_user and the actor of that resource's latest resource_revision. p99 is 55 ms at 10 rows per page and 1.4 s at 200. Database telemetry shows 401 statements per request, each under 1 ms, and nothing in the slow-query log. Diagnose the cause and give the fix, stating the statement count per request and the p99 you expect afterwards.
Approach
- Read the counters before forming a theory. 401 statements for 200 rows is one driver query plus two per row, and sub-millisecond execution with an empty slow-query log rules out a bad plan. The time is round trips, which is why it is invisible in every per-query metric and scales with rows returned rather than with filter selectivity.
- Name the two per-row statements from their normalised text: a single-row app_user lookup by user_id, and a resource_revision lookup by resource_id ordered by version DESC LIMIT 1. Confirm by dropping those two response fields and watching the statement count fall to one. That locates the calls in the serialisation layer, not the repository.
- Check that the arithmetic accounts for the whole gap. Measure one round trip to the replica in isolation; 400 trips at roughly 3 ms of network plus 0.2 ms of execution is about 1.3 s on top of a 55 ms baseline, which matches. If the multiplication had fallen short, the N+1 would only be part of the story and you would keep looking.
- Batch both lookups. Collect owner_user_ids and resource_ids from the driver query, then issue WHERE tenant_id = $1 AND user_id = ANY($2) for the users, and PostgreSQL's SELECT DISTINCT ON (resource_id) ... WHERE resource_id = ANY($2) ORDER BY resource_id, version DESC for the latest revision, which the UNIQUE (resource_id, version) index serves directly. On an engine without DISTINCT ON, use a lateral join or a row_number window. Three statements per request at any page size.
- Keep the tenant predicate in the batched query. The per-row version was implicitly scoped because its ids came from tenant-scoped rows; a batched user_id = ANY(...) with no tenant_id is an unscoped read that behaves correctly only as long as the id list is trustworthy.
- Re-measure at 10, 50 and 200 rows and confirm the statement count is constant. Latency should now track bytes returned.
Follow-up
- The page size is capped at 200 today. What breaks first if it is raised to 2,000, and is it still this bug?
- How do you stop the next N+1 from reaching production, given that no individual query is slow and the endpoint's tests pass?
- The latest-revision actor is only used to render an avatar. Make the case for denormalising it onto resource, and name the write anomaly that introduces.
Suggested practice checklist
Roughly ninety minutes on weeknights with one longer weekend block. The plan cuts scope rather than compressing everything, on the assumption that one thing finished per night beats four half-started.
Day 1 — Fix the scope and take a cold baseline
- Read the role description and write the three things the loop will almost certainly test, then write an explicit not-doing list and keep it visible all week.
- Take one twenty-five-minute coding problem and one fifteen-minute design prompt cold, and write the single sentence naming what blocked each, because those two sentences decide where the remaining evenings go.
- Set the week's rule: one thing finished every night, including the night you only have forty minutes.
Deliverable: A one-page scope with a not-doing list and two cold attempts, each carrying one sentence on what blocked it.
Day 2 — One pattern, written three times from blank
- Choose the single pattern most likely to appear in your loop and write it three times from an empty file rather than editing the previous attempt.
- On the third pass, write the invariant as a comment before the loop body and the complexity before the first line of code.
- Stop at ninety minutes even if the third version is imperfect, and write the one thing you would fix given another hour.
Deliverable: Three independent implementations of the same pattern plus a note on what changed between them.
Day 3 — One design, only to the depth you can defend
- Take one system shape and go only as far as requirements, interface and data model, refusing to draw a box you could not survive a follow-up about.
- Attach one number to each non-functional requirement, deriving it rather than asserting it, and write the assumption the number rests on.
- Write the one tradeoff you are choosing against and the observation that would make you reverse it.
Deliverable: One design at interface-and-schema depth with derived numbers and one written reversible tradeoff.
Day 4 — Only the fundamentals you will have to defend
- Write, in under two hundred words each, the answers to the two questions that follow almost any implementation: why this structure and not the obvious alternative, and what happens to this code at a hundred times the input.
- Write what an index actually costs: faster lookups on the indexed columns against a write that now maintains a second structure, plus the cases where the planner declines to use it anyway, low selectivity, or a predicate wrapping the column in a function.
- Delete any answer you cannot deliver aloud in under a minute, since an answer that needs reading is not an answer you have.
Deliverable: Three written answers, each under two hundred words and each timed aloud.
Day 5 — Your own work, timed
- Write a ninety-second and a four-minute version of your main project and time both aloud rather than reading them.
- Prepare the two follow-ups that always come: what you would do differently, and how you knew it worked.
- Put one number in the first sentence and be ready to say exactly where it came from and what it excludes.
Deliverable: Two timed narratives with one defensible number in the opening line.
Day 6 — The one full rehearsal, in the weekend block
- Run a sixty-minute mock covering a coding round and a design round in one sitting with no break, because sustained attention is the thing evenings have not trained.
- Immediately afterwards, and before hearing any feedback, write the three moments you lost the thread.
- Spend the rest of the block only on those three moments, and on nothing you merely feel shaky about.
Deliverable: Mock notes naming three failure moments with a specific fix written under each.
Day 7 — Taper
- Write the twenty-minute warm-up you will actually do on the morning: one problem you can already solve from a blank file, one design you can narrate, and nothing you have never seen.
- Re-read only your own notes from this week and open no new material.
- Write the logistics down: the editor or shared document you will be working in, whether execution and lookups are permitted, and the sentence you will use when you do not know something.
Deliverable: A one-page card holding the design structure, the project numbers, and the logistics.
Review and mentoring: what you let through
Counting review comments or mentees proves nothing. The useful version is a specific change you approved with a reservation you stated, or one you blocked and the delay that cost. Say which standard you were holding and why it was worth the friction. A mentoring story needs the thing the other person can now do without you.
Describe your experience with Linux systems administration, specifically focusing on kernel optimization and resource allocation for GPU-accelerated workloads.
Medium · behavioural and collaboration
Approach
- Give the blast radius: what could have broken, and what you measured.
- Pick a story where you made the decision, not one where you watched it.
- State the situation in two sentences and spend the rest on the reasoning.
Follow-up
- What would you do differently if you ran that again?
- How did you know your change caused the improvement?
Argue against a design, lose, and commit anyway
Medium · disagreement · service boundaries · decision records
Describe a design you argued against and lost. State the failure you predicted as a named mechanism, not a feeling about complexity: two services that would need one transaction, a projection with no rebuild path, a write path with no idempotency key. Say what evidence you brought, what the decision maker weighed instead, and what you did after the decision was made: what you instrumented, what you wrote down, and whether the prediction came true. Five minutes.
Approach
- State the prediction in falsifiable form up front: the mechanism, the condition that triggers it, and the observable outcome. A prediction that cannot be checked also cannot be credited to you later.
- Show the evidence you had at the time and label each piece honestly as measured, analogous, or intuition. Keeping the intuition is fine; disguising it as data is the thing that erodes your standing in the next argument.
- Represent the opposing case at full strength, including the constraint you did not control: a fixed date, a team boundary, or the fact that the decision was cheap to reverse and yours was not.
- Make disagree-and-commit concrete. Name the artefact you left behind so the prediction could be settled without you: the alert and its threshold, the counter on the dashboard, the decision note that recorded the trade-off and the condition that would revisit it.
- Report the outcome without editing it. If the design held and your predicted mechanism never fired, say so and say what you had mis-weighted, which is more persuasive than a vindication story.
Follow-up
- What threshold on that alert would have proved you right, and did anyone ever look at it?
- If the same proposal arrived tomorrow with the same deadline, would you argue it the same way?
- How did you behave toward the design once it shipped and started failing in a different way than you predicted?
Ship under a deadline and bound the debt you chose
Medium · pagination · technical debt · tradeoffs
You have four days to ship a tenant-facing listing endpoint. The version you would defend uses keyset pagination over (tenant_id, status, updated_at DESC, resource_id DESC); the version you can finish uses LIMIT/OFFSET with no matching index. Describe a deadline call you actually made of this shape: what you shipped, what you knowingly deferred, how you bounded the damage with a mechanism rather than an intention, and the specific numeric condition that would force the follow-up. Name who you told and where you wrote it down.
Approach
- Name the deferred failure precisely instead of calling it slow. OFFSET n makes the database produce and discard n rows, so cost grows with page depth; without an index matching the sort, every matching row is read and sorted before the limit applies; and rows inserted between two page fetches shift across the boundary so items are skipped or repeated with nothing in the response to signal it.
- Bound the blast radius with something mechanical rather than a promise: cap maximum page depth, cap page size, restrict the endpoint to one internal caller, or keep it behind a flag. State which failure each cap removes and which it leaves standing.
- Attach a number to the trigger and wire it to an alarm: the first tenant crossing N resources, or the endpoint's p99 crossing its share of the 400 ms budget, so the debt announces itself instead of waiting to be remembered.
- Write it where the next engineer looks, which is the code and the ticket, not a chat message: what was deferred, why, the cap, and the trigger.
- Report what actually happened in your real example, including the case where the trigger never fired and the debt was correctly never repaid.
Follow-up
- At what page depth does the offset version breach your latency budget, given your page size and row counts?
- What breaks first when you switch to keyset pagination later, and what does a client holding an old page token see?
- Who would have overruled you if you had asked for two more days, and did you ask?