Motorola Machine Learning Engineer Interview Questions & Guide 2026

The 60-second brief

As a Machine Learning Engineer at Motorola, you will play a pivotal role in shaping the future of mission-critical communications, public safety technology, and advanced video security solutions. Motorola relies on machine learning to power real-time analytics, predictive dispatching, threat detection, and advanced audio-visual processing. Your work directly impacts the safety and efficiency of first responders, enterprise operations, and millions of citizens worldwide who rely on resilient, intelligent communication networks.

Find out what you will be typing into. A shared plain-text editor with no autocomplete, compiler or test runner changes what you have to hold in your head, and practising inside your own configured environment hides exactly that gap.

Focus areas

• Trace a symptom to a mechanism under load
• Make every write idempotent under retry
• Detect concurrent edits instead of losing writes

Overview

In this position, you will bridge the gap between advanced research and production-grade software. You will build, optimize, and deploy robust machine learning models that run seamlessly across cloud infrastructure and resource-constrained edge devices, such as smart radios, body-worn cameras, and security systems. This requires not only deep theoretical knowledge of artificial intelligence but also exceptional software engineering discipline to ensure these systems operate with high reliability and low latency.

By joining Motorola, you are stepping into an environment where your algorithmic contributions have tangible, real-world consequences. Whether you are working on computer vision for public safety cameras or natural language processing for emergency dispatch systems, your engineering decisions will help build safer communities and more responsive systems globally.

The Motorola interview process

Motorola candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

HR Screening

Before anything technical happens, someone has to decide which rung of the ladder your loop is calibrated to, and that decision sets the bar for every round after it. It comes from how you describe scope, not from your title, because titles do not convert cleanly between companies. The weak version of the answer is team size and years. The strong version names the largest change you shipped where nobody reviewed the design, what would have broken if you had been wrong, and what you were paged for. Get the level said out loud on this call, because the range and the loop both follow from it.

What to demonstrate

• Whether the scope in your own account maps onto a level the team actually has an opening at, so a mismatch ends the process cheaply rather than after four interviewers have spent a day
• Whether your title needs re-mapping: the same word describes very different amounts of independent decision-making at a twenty-person company and a ten-thousand-person one
• Whether your compensation expectation can be filled at that level in the structure the role pays in, which is why the number gets asked for before any engineer is scheduled

How to prepare

• Write down two changes from the last two years: the largest one you designed with nobody reviewing the design, and the largest one where someone more senior did. Lead with the first when scope comes up, and be ready to say which parts of the second were yours
• Ask which level the loop is calibrated to and what changes at the level above it, then plan your weeks from that answer rather than from the posting
• Settle a total-compensation range beforehand with the split named, base against bonus against equity and its vesting period, so a question about numbers gets a number instead of the word market

Technical Rounds

Input bounds are the part of the prompt most often skimmed, and they usually contain the answer. They tell you which complexity class is admissible, which narrows the search before you have thought about the problem itself. As a rough planning figure, a compiled language does on the order of 10^8 simple operations per second and an interpreted one roughly an order of magnitude less. So n up to about twenty admits enumerating subsets, a few thousand admits a quadratic pass, and a million admits neither: you need near-linear, or linear with a log factor. If the bounds are missing, ask for them.

What to demonstrate

• Whether the approach is justified by the stated input size rather than by whichever pattern you recognised first
• Whether you ask about the properties that change the algorithm: whether the input arrives sorted, whether duplicates occur, whether values are bounded integers, whether it all fits in memory
• Whether you can name the bottleneck in your own solution and what would remove it, even when you deliberately leave it in place
• Whether a claimed speedup is real, since memoising a recursion only helps when subproblems genuinely overlap and the state can be keyed cheaply

How to prepare

• For each algorithm you rely on, write down the largest n it handles in roughly a second, then check two of those figures by timing them in the language you will actually type in
• For two weeks, write one line naming your target complexity and the bound that justifies it before you write any code, then compare that line with what you ended up submitting
• Practise the conversion backwards: given a required O(n log n), list the mechanisms that get you there (sorting, a heap, an ordered map, divide and conquer) and choose by what the problem needs to query, not by what you used last

Behavioral Assessment

This round is deciding whether a change you make without supervision can be allowed to reach production. It is scored on what you knew at the moment you decided, not on how it turned out, so a story that opens with the result and works backwards reads as luck retold as judgement. Say what the options were, what you did not know, what you did to shrink the unknown before committing, and what you accepted as the worst plausible case. The detail that separates answers is a bound: how many users, how much data, and for how long, if you had been wrong.

What to demonstrate

• Whether the reasoning you give was available at the time you decided rather than after the result came in, since a story whose deciding evidence arrived later describes an outcome and not a judgement
• Whether you can put units on the exposure (users, rows, minutes of degraded service) and whether the containment you chose actually bounded it: a canary bounds the request path it fronts, while a background job writing to a shared table reaches every user regardless of which version served their requests
• Whether the reversal path existed before you shipped or was improvised during the incident, and whether it restores state or only stops further damage

How to prepare

• For your three largest changes, write down the one thing you would have had to be wrong about for it to fail, and what your best estimate of it was on the day you shipped. If you never held an estimate, that is the gap the follow-up questions will find
• Write the undo procedure for one of those changes as it existed at the time, then mark which steps restore data and which only stop new damage. Turning a flag off or reverting a deploy ends the new writes; rows already written come back only from a copy you kept, and a dropped column comes back empty unless something outside the schema holds the values
• Rehearse one story from the decision point forward and stop before the outcome, then have someone ask what you would do next. If the story only works with the ending attached, it is an anecdote rather than a decision you can defend

Where candidates lose points

Running a schema change as though the lock lasts as long as the statement

In PostgreSQL an ALTER TABLE that needs an ACCESS EXCLUSIVE lock must first wait for every open transaction touching that table, and while it waits, later queries needing a conflicting lock queue behind it rather than overtaking it. A DDL statement that would execute in milliseconds, issued while a thirty-second analytics query is open, therefore stalls all traffic on that table for thirty seconds: the outage length is set by the longest open transaction, not by the change. The defences are specific and worth knowing by name - set lock_timeout low and retry rather than queue, add columns without a volatile default so no table rewrite occurs (from version 11 a non-volatile default is a metadata-only change), build indexes with CREATE INDEX CONCURRENTLY while accepting that it cannot run inside a transaction block and leaves an invalid index behind if it fails, and add constraints as NOT VALID followed by a separate VALIDATE CONSTRAINT, which takes a weaker lock.

Choosing an index from the columns a query mentions rather than from how it filters and orders

A composite B-tree index on (a, b, c) can be seeked only as a left prefix: equality on a, then equality on b, then a range or an ordering on c. A query that filters on b alone cannot seek into it at all and at best gets a full scan of the index; a query that filters a and ranges on b gets no benefit from c, because the index is only sorted by c within a fixed (a, b) pair. The practical consequence is that one index per column is close to useless for multi-predicate queries while a single correctly ordered composite index turns a scan into a lookup. The ordering half is what gets missed: if the index cannot satisfy the ORDER BY, the database must read every matching row and sort before the limit can apply, so a LIMIT 20 over a million matching rows still reads a million rows.

Naming no test cases at all

State what you would test before being asked: empty input, a single element, all elements equal, the maximum permitted size, and the input that exercises the branch you just wrote. It costs thirty seconds and is much of what separates someone who has shipped code from someone who has only solved puzzles.

A queue or buffer with no bound

Every producer-consumer boundary needs a capacity and a policy for reaching it: block the producer, shed load, or drop the oldest entry. Unbounded buffering converts a temporary slowdown into memory exhaustion and hides the backpressure signal that would have revealed the consumer was falling behind.

Questions & practice

Describe the difference between L1 and L2 regularization and how they affect model weights.

Medium · machine learning fundamentals

Approach

1. Say how you would validate it, and where leakage could enter the split.
2. Pick the metric from the cost of each error type, not from habit.
3. Name the simplest model that could work and what would make you move past it.

Follow-up

• What changes if the classes are heavily imbalanced?
• How would you know the model is overfitting?

Describe how you would implement a memory-efficient cache for frequently accessed model predictions.

Medium · machine learning fundamentals

Approach

1. Say how you would validate it, and where leakage could enter the split.
2. State the learning problem: the label, the unit of prediction and how the model is used.
3. Pick the metric from the cost of each error type, not from habit.

Follow-up

• What changes if the classes are heavily imbalanced?
• Where could label leakage enter this setup?

What techniques would you use to optimize a deep learning model to run on a resource-constrained edge device?

Medium · machine learning fundamentals

Approach

1. Name the simplest model that could work and what would make you move past it.
2. Say how you would validate it, and where leakage could enter the split.
3. Pick the metric from the cost of each error type, not from habit.

Follow-up

• Where could label leakage enter this setup?
• How would you know the model is overfitting?

How do you handle highly imbalanced datasets when training a classification model?

Medium · machine learning fundamentals

Approach

1. State the learning problem: the label, the unit of prediction and how the model is used.
2. Name the simplest model that could work and what would make you move past it.
3. Say how you would validate it, and where leakage could enter the split.

Follow-up

• How would you know the model is overfitting?
• What changes if the classes are heavily imbalanced?

Implement a function to detect cycles in a directed graph.

Medium · coding and algorithms

Approach

1. Choose the data structure from the access pattern, not from familiarity.
2. Restate the input: its shape, its size, and what is guaranteed about it.
3. Name the brute-force solution and its complexity before improving on it.

Follow-up

• Which test case would catch an off-by-one here?
• How does this change if the input no longer fits in memory?

Given an array of integers, find the contiguous subarray with the largest sum.

Medium · coding and algorithms

Approach

1. Walk one small example through your approach before writing the whole thing.
2. State the target complexity and say which constraint rules the naive version out.
3. Name the brute-force solution and its complexity before improving on it.

Follow-up

• Which test case would catch an off-by-one here?
• What is the worst case, and how likely is it on real data?

Track a rolling failure rate per destination for circuit decisions

Easy · sliding window · ring buffer · circuit breaker

The egress service delivers about 1,500 webhooks per second across roughly 40,000 destinations, each call bounded by a 10 second timeout. Maintain, per destination, the failure rate over the trailing 60 seconds so a caller can ask before dispatch whether the circuit should open. Attempts arrive as (destination_id, finished_at_ms, outcome). Requirement: amortised O(1) per attempt, with total memory bounded by the destination count rather than by traffic. Give the structure, its exact memory, and the rule that stops a destination with three attempts from opening a circuit.

Approach

1. Name the exact-deque version and then reject it as the default. Holding timestamps and advancing a tail pointer past anything older than now minus 60 seconds is a correct two-pointer window at amortised O(1) per attempt, but its memory tracks in-window traffic, so one destination in a retry storm holds hundreds of thousands of entries while thousands of quiet destinations hold none.
2. Use a ring of 60 one-second buckets per destination, each bucket a pair of counters for attempts and failures. On an attempt, advance the ring by the elapsed whole seconds, zeroing at most min(elapsed, 60) buckets, then increment the head. That is amortised O(1) with a fixed footprint per destination.
3. State the footprint: 60 buckets times two 4-byte counters is 480 bytes of payload per destination, so 40,000 destinations is roughly 20 to 25 MB with per-entry overhead, bounded by the catalogue rather than by the rate. The cost is granularity, since the oldest bucket ages out in whole seconds, which is far tighter than the decision needs.
4. Require a minimum sample before the circuit may open. A destination with three attempts and three failures reads as 100 percent and is not evidence; a floor of roughly 20 attempts in the window makes the ratio meaningful, and below that floor use a run of consecutive failures as the trigger instead.
5. Expire idle destinations, or memory grows with every destination ever seen rather than with the live set. Hold the rings in a bounded LRU keyed on destination_id and treat a miss as no history, which is the correct default for an endpoint that has been silent for a minute.
6. Keep the half-open probe out of the window arithmetic. After the circuit opens, one probe per interval decides whether to close it, and folding that single success into a window that still holds a 100 percent failure history would reopen the destination on one data point.

Worked solution (20 min)

1. Define the bucket struct and the advance step: take floor(finished_at_ms / 1000), compare with the ring's current second, zero min(delta, 60) buckets forward, then write into the new head.
2. Trace a destination that receives 5 attempts, goes silent for 90 seconds, then receives one more, and confirm the rate is computed from one attempt rather than six.
3. Compute total memory for 40,000 destinations at 60 buckets of two 4-byte counters, and state what changes if the window widens to 300 seconds.
4. Write the open rule as a single predicate combining the minimum-attempt floor with the rate threshold.

Expected result: A per-destination ring of 60 one-second (attempts, failures) buckets advanced lazily for amortised O(1) cost, roughly 480 bytes of counters per destination and about 20 to 25 MB for 40,000 of them, an LRU bound on live rings, and an open rule requiring both a minimum attempt count in the window and a rate above threshold.

Verify it yourself

• After 60 seconds of silence a destination's window holds zero attempts and cannot open a circuit.
• Memory measured with one hot destination and 39,999 idle ones is indistinguishable from the reverse, which the deque version would fail.
• Replaying the same attempt stream with the clock shifted by an hour yields the same final rate.

Follow-up

• The fleet is 30 instances and each sees roughly a thirtieth of a destination's traffic. Where does the rate actually live, and what does a per-instance answer get wrong?
• A destination answers in 9.5 seconds and succeeds. It is not failing but it is consuming your per-destination concurrency. What signal should open the circuit here?
• How would you make the window survive a process restart, and is it worth the cost?

Hold a per-tenant active cap against concurrent creates

Hard · write skew · isolation levels · constraints

A tenant on the standard plan may hold at most 50 resources with status='active'. The create handler runs SELECT count(*) FROM resource WHERE tenant_id = $1 AND status = 'active', compares to 50, then inserts. Two creates arrive 3 ms apart on different instances and the tenant lands at 51. Name the anomaly, say whether PostgreSQL 16 READ COMMITTED or REPEATABLE READ prevents it and why, then give an implementation that holds the cap at READ COMMITTED with the exact statements. Finally, say what changes when the cap is 'at most one running export per tenant' on job_run.

Approach

1. Name it: write skew. The two transactions read an overlapping set and write disjoint rows, so there is no row-level conflict for the engine to detect and each commit is individually legal.
2. Rule out the levels precisely. READ COMMITTED takes a fresh snapshot per statement and takes no lock on the counted rows, so both see 49. PostgreSQL's REPEATABLE READ is snapshot isolation: it removes non-repeatable reads and phantoms within the snapshot but still admits write skew, because the anomaly is not a re-read of a changed row, it is a read of a set that a concurrent transaction invalidates. Only SERIALIZABLE closes it, by tracking the read dependency and aborting one transaction with SQLSTATE 40001 — a guarantee that exists only if the application re-runs the whole transaction from the read.
3. Convert the set predicate into a single-row conflict: keep tenant.active_resource_count and run UPDATE tenant SET active_resource_count = active_resource_count + 1 WHERE tenant_id = $1 AND active_resource_count < 50 in the same transaction as the INSERT. Zero affected rows is the cap, returned as 409. The row lock serialises the decision at any isolation level, and contention is bounded to one tenant's row — which is also the fair-scheduling unit, unlike a global counter that would convoy every tenant behind one row.
4. State the cost you just took on: a counter is a second source of truth that can drift, so every path that changes status must adjust it inside the same transaction, and a periodic reconciliation has to exist, with resource_revision as the authority for what the count should have been.
5. For the job case the invariant is expressible per row, so let the database hold it: a partial unique index on job_run (tenant_id, job_type) WHERE status IN ('queued','running') makes a second running export unwritable and the loser takes 23505, mapped to 409. That is strictly better than a counter — no drift, no reconciliation — and it is available only because the cap is one rather than fifty.
6. Add the retry discipline each route demands: under SERIALIZABLE both 40001 and deadlock 40P01 are retryable and the retry must re-execute the read, while under READ COMMITTED with the counter nothing retries, because the conflict is reported to the caller rather than raised as an error.

Worked solution (35 min)

1. Reproduce with two sessions that both count 49, both insert and both commit, at READ COMMITTED and then at REPEATABLE READ; record the final active count for each.
2. Repeat both sessions at SERIALIZABLE and record which SQLSTATE the loser receives and at which statement it is raised.
3. Implement the counter form and run a 20-way concurrent create against a tenant sitting at 45 active resources.
4. Implement the partial unique index for the job case and race 20 enqueues of the same export.

Expected result: READ COMMITTED and REPEATABLE READ both finish at 51. SERIALIZABLE finishes at 50 with one transaction aborted at 40001 — usually at commit, though SSI may raise it at an earlier statement. The counter form finishes at exactly 50: of the twenty creates, the five that take the tenant from 45 to 50 return 201 and the remaining fifteen match zero rows and return 409, with no error raised. The job index admits one running export and rejects the rest with 23505.

Verify it yourself

• count(*) WHERE status='active' never exceeds the cap across 50 randomised concurrent runs.
• Remove the active_resource_count < 50 predicate and the same test must start failing; if it still passes, the race is not being exercised.
• At SERIALIZABLE, confirm every participating session is SERIALIZABLE — one session at a lower level and the guarantee is gone.
• After each run, the counter equals the real count of active rows.

Follow-up

• A resource moves from archived back to active. Which statements change, and what breaks if the counter update and the status change land in different transactions?
• The cap becomes plan-dependent and a plan can change mid-month. Where does the number 50 live, and who reads it?
• How do you detect after the fact that the counter drifted, without locking the table?

Write the update path that detects a concurrent edit

Medium · optimistic concurrency · transactional outbox · audit history

resource carries version INT NOT NULL DEFAULT 1. resource_revision holds revision_id, resource_id, version, actor_user_id, change_kind, patch JSONB, request_id, created_at with UNIQUE (resource_id, version). outbox_event holds aggregate_type, aggregate_id, aggregate_version, event_type, payload, status. A PUT carries the version the client read. Write the exact statements for the single transaction that applies the edit, records the revision and enqueues 'resource.updated', and give the handler's branch on zero affected rows. Then say what PostgreSQL 16 does under READ COMMITTED when two of these updates hit one row at once.

Approach

1. One transaction, three writes, no network call inside it: UPDATE resource SET title = $3, version = version + 1, updated_at = now() WHERE resource_id = $1 AND tenant_id = $4 AND version = $2; then INSERT the resource_revision row at version $2 + 1; then INSERT the outbox_event row at the same aggregate_version. The event goes to a table rather than a broker because no transaction spans both.
2. Branch on the affected-row count before doing anything else. Zero has three causes — stale version, wrong tenant, row gone — so re-read once and map to 409 carrying the current version, or 404 for an id outside the caller's tenant, which also stops the endpoint confirming that another tenant's id exists.
3. State the engine behaviour instead of assuming it. Under READ COMMITTED the second UPDATE blocks on the row lock, and when the first commits PostgreSQL re-evaluates the WHERE clause against the newly committed row, so the version predicate now fails and the statement reports zero rows. Under REPEATABLE READ the identical collision raises SQLSTATE 40001 instead, so the handler must fold both shapes into one conflict response.
4. Keep UNIQUE (resource_id, version) even though the predicate already serialises writers. It is what makes a lost update unwritable if any other path ever reaches the revision table, and it converts a logic bug into 23505 rather than into a silently missing history row.
5. Refuse to auto-retry the whole PUT. A retry re-reads the winner's state and reapplies an intent formed against data that no longer exists — the silent overwrite the version token was added to detect. Return the conflict; merge field-wise only if the patches are provably disjoint.
6. Note that now() is the transaction timestamp in PostgreSQL, so resource.updated_at, the revision's created_at and the outbox row share one instant, which is what later makes reconciliation between the three tables unambiguous.

Follow-up

• A client sends the version it read ten minutes ago and the resource has moved three versions. What is in your 409 so it can resolve the conflict without a full re-fetch?
• Two editors, two disjoint fields, no overlap. Does your answer still refuse the second write, and should it?
• Every write now touches a second hot table. How do you keep the outbox insert and its partial index from becoming the write bottleneck at 1.2k writes/second?

How do you detect and mitigate data drift in a production environment?

Medium · ml system design and production

Approach

1. Say where features come from at serving time and how they match training.
2. Separate the offline training path from the online serving path.
3. Fix the product goal and the online metric before choosing any model.

Follow-up

• What happens when a feature is missing at serving time?
• How would you roll the new model out safely?

What are the differences between multi-threading and multi-processing in Python, and how do they impact ML pipeline performance?

Medium · ml system design and production

Approach

1. Separate the offline training path from the online serving path.
2. Fix the product goal and the online metric before choosing any model.
3. Name what you would monitor after launch and what triggers a retrain.

Follow-up

• How would you roll the new model out safely?
• How would you detect drift before the metric drops?

How would you design an efficient data structure to store and query real-time GPS coordinates of active devices?

Medium · system design

Approach

1. State the consistency you need, and where you are willing to be stale.
2. Choose a partition key and say what query it makes expensive.
3. Name the failure you are designing for, then the recovery path.

Follow-up

• What would you drop to keep the system up under load?
• How does this behave when that dependency is down for an hour?

Rebuild the search projection while it serves nine thousand queries

Medium · projections · eventual consistency · rebuild · watermark

The read-model service answers about 9k queries/second at a 120 ms p99 from a projection built off the event log, normally under 2 seconds behind. A mapping change forces a full rebuild from resource_revision, during which apply lag rises to minutes. Writes continue at 1.2k/second and events at 4k/second. Design the rebuild: how the new index is populated and cut over, how position is tracked per log partition, what the API returns alongside results so a client can tell a stale answer from a current one, and the criterion for cutting over.

Approach

1. Build into a second index and swap an alias rather than mutating the live one. The rebuild is then reversible by pointing the alias back, so a bad mapping costs a wasted rebuild instead of an outage. The price is peak storage for two full copies and double apply load during catch-up, and both numbers should be stated up front rather than discovered when disk fills.
2. Track position per log partition, not globally. Order is guaranteed only within an aggregate's partition, so progress is a vector, and the only number safe to publish is taken from the least advanced partition - the slowest one is what bounds completeness. Publishing the most advanced partition's position declares the projection current while another partition sits twenty minutes behind.
3. Make apply idempotent so the backfill and the live tail can overlap without a freeze. Each document records the aggregate_version it reflects, and any event at or below that version is discarded. This is why the event carries the full fact and not a delta: a delta cannot be discarded safely, and a consumer that calls back to read current state applies a state newer than the event it is processing, which is how a projection ends up with changes applied out of order.
4. Turn lag into a contract instead of a surprise. Return the watermark with every result set, and return the version produced by a write so the client can compare the two. A client that wrote version 7 and receives results at a watermark older than its own commit can show that its change is still landing, rather than rendering the previous value as current. Blocking the read until the projection catches up would convert a staleness problem into an availability problem at 9k queries/second, and choosing not to do that is the trade.
5. Define the cutover numerically. Writes are untouched by the rebuild - they commit to the primary and land in the outbox - so the only coupling is apply throughput. If catch-up applies slower than the 4k events/second arriving, it never converges. The cutover criterion is that lag is measurably decreasing and below a stated threshold, not that the backfill loop reached the end of its range.

Worked solution (30 min)

1. Write the rebuild sequence: snapshot source, populate the second index, attach the live tail, verify, swap alias, retire the old index - naming what is reversible at each step.
2. Define the per-partition position record and the single watermark derived from it, and show what each would report when one partition stalls.
3. Write the idempotent apply rule using the document's recorded aggregate_version, and replay one event twice against it.
4. State the cutover threshold in lag terms and the measurement that shows convergence rather than completion.

Expected result: A parallel index populated from resource_revision snapshots plus the live tail, cut over by an alias swap that is reversible. Position is stored per log partition and the published watermark comes from the least advanced one. Apply is idempotent on the document's recorded aggregate_version, so backfill and tail can overlap without a freeze. Results carry the watermark and writes return their version so clients can detect staleness. Cutover happens when lag is decreasing and under the stated threshold.

Verify it yourself

• Applying the same event twice leaves the document unchanged.
• The published watermark never advances past the least advanced partition.
• Pointing the alias back restores previous behaviour with no reindex.
• Catch-up rate exceeds the 4k events/second arrival rate before cutover, shown by the lag trend rather than by the loop finishing.

Follow-up

• The rebuilt index disagrees with the primary tables for 300 documents. Which is authoritative, and how do you decide without freezing writes?
• The rebuild doubles load on the log and pushes the projection p99 from 120 ms to 400 ms. What do you throttle, and which signal sets how much?
• Clients start polling until the watermark passes their write. What does that do at 9k queries/second, and what do you offer instead?

Exports duplicate a row range about once a week

Hard · leases · clock skew · idempotency · fencing

Roughly once a week an export writes a file containing a duplicated range of rows. The affected job_run rows show attempt = 1, status = succeeded, one started_at, and a lease_owner naming a different host from the one whose logs show the job starting. Leases last 30 seconds and are heartbeated every 10 from inside the handler; lease_expires_at is computed on the worker and compared against the database's now(). Find the mechanism, and give a fix that holds even if you cannot fix the clocks.

Approach

1. Start from the fact that eliminates the obvious answer. attempt = 1 means no retry was recorded, so this is not a re-run after failure; two workers ran the same row concurrently and the takeover path never touched the counter. lease_owner naming a host other than the one that started the job is the same statement from the other side.
2. Enumerate the mechanisms that cause a premature takeover, then find the signal that separates them. Either the lease genuinely expired because the heartbeat did not fire, which is what happens when the heartbeat runs on the handler's own thread and the handler makes a long blocking call, or it only appeared expired because two clocks disagree, since lease_expires_at is written from the worker's clock and evaluated against the database's. The discriminator is the distribution: incidents clustered on the longest exports indict the heartbeat, incidents clustered on one host indict skew. Measure both, and measure each host's offset against the database directly.
3. Read the reclaim query precisely. In PostgreSQL now() is transaction start time, not statement time, so a reclaimer holding a long transaction compares against an older timestamp than expected; clock_timestamp() is the statement-time function. This is worth ruling in or out before you redesign anything, because it changes which rows look expired.
4. Remove the second clock rather than trying to synchronise it. Issue and extend the lease in the database, with lease_expires_at = now() + interval '30 seconds' in both the claim and the heartbeat, so exactly one clock is ever compared and worker skew stops mattering to this predicate.
5. Accept that a lease can still expire under a slow worker, because a lease cannot distinguish slow from dead, and fence the work. Carry a monotonically increasing lease generation and make every write the handler performs conditional on still holding it, as UPDATE ... WHERE job_run_id = $1 AND lease_owner = $2 AND lease_generation = $3, so a displaced worker's writes affect zero rows and it aborts instead of duplicating.
6. Make the handler's writes idempotent independently of all that: give each exported chunk a natural key of (job_run_id, batch_start) with a unique constraint so a second copy conflicts rather than appends, move the heartbeat off the handler's thread, and increment attempt on takeover so the event is visible in a metric.

Follow-up

• The displaced worker has already streamed half the file to object storage. What makes that side effect safe to repeat?
• You now count takeovers. What alert fires on that counter, and at what threshold?
• What breaks if you simply raise the lease to five minutes?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Numbers before diagrams

• Build your own reference card of the figures you will re-derive all week: bytes for a realistic record, requests per second implied by a given daily active count, and the storage that a year at a given write rate produces. Derive each one rather than copying it, because the derivation is what survives a follow-up.
• Turn one product statement into capacity requirements. From ten million daily users at four writes and forty reads each, state the peak-to-average factor you are assuming and why, then produce peak write QPS, peak read QPS and a year of storage.
• Write the two numbers whose order of magnitude changes the design, the read-to-write ratio and the working-set size against memory per node, and state the threshold at which each one flips your answer.

Deliverable: A one-page numbers card and one worked capacity estimate with every assumption written down.

Day 2 — One system, from requirements to schema

• Spend the first ten minutes producing only functional requirements, non-functional targets with numbers attached, a p99 latency, a durability expectation, a consistency requirement, and an explicit out-of-scope list.
• Define the interface before the boxes: the three or four endpoints, their parameters, what each returns, and which of them are idempotent.
• Write the data model, then write the single access pattern that justifies it, and state what the schema would have to become if the dominant access pattern were the other one.

Deliverable: One design carried to endpoint-and-schema depth, with non-functional targets expressed as numbers and a written out-of-scope list.

Day 3 — The consistency you are actually buying

• Write out what a client sees under asynchronous replication when its write commits on the leader and its next read is served by a lagging follower, then write the two fixes, pinning that session's reads to the leader for a bounded window or carrying a version token the replica must reach, and the cost of each.
• Work the quorum arithmetic on paper for N of three with W and R of two, and separate what R + W > N does guarantee, that any read set intersects any write set, from what it does not: on its own it is not linearizability, and a sloppy quorum that accepts writes on nodes outside the preference list breaks even the intersection.
• Take two storage choices with different defaults, a single-leader relational store committing synchronously and a quorum-replicated store that converges eventually, and write the specific product behaviour that would be wrong under each, rather than a general statement about which is stronger.

Deliverable: A page separating what quorum overlap guarantees from what it does not, with one concrete product misbehaviour attached to each gap.

Day 4 — Failure is the design

• For one write path, work through the case where the client times out after the server has already committed, then design the idempotency key: who generates it, how long it is retained, and what the duplicate request returns.
• Express the retry policy as parameters rather than as a word: maximum attempts, base delay, backoff factor, jitter, and which error classes are retried at all. Then state why retrying a non-idempotent write without a key is a correctness bug and not merely waste.
• Compute the fan-out effect on tail latency. If a request waits on ten backends and each independently exceeds its p99 one percent of the time, the chance at least one is slow is 1 - 0.99^10, about ten percent. Then write why independence is the optimistic assumption and what correlates them in practice.
• Name the backpressure mechanism for one queue or one dependency in the design, a bounded queue with shedding or a concurrency limit, and write what the caller is told when it engages.

Deliverable: One write path with an idempotency design, a parameterised retry policy, and a written tail-latency calculation with its assumption named.

Day 5 — Scaling the hot path

• Choose cache-aside or write-through for one read path and write the staleness window each produces, then name the invalidation event and what the system does when that event is lost.
• Design against the stampede: either coalesce requests so only one recomputes a missing key, or refresh early with jittered expiry, and write why identical TTLs on keys populated in the same moment produce a synchronised expiry and a thundering herd.
• Shard one table by a key you choose, then answer the two questions that break the choice: which queries now require a scatter-gather, and what happens to the distribution when one tenant is a hundred times larger than the median.
• Write the cost of adding a node under plain modulo placement, where nearly every key moves, against consistent hashing, where roughly one key in n+1 moves, and state what virtual nodes are for.

Deliverable: A caching and sharding decision for one path, each with its failure mode and its rebalancing cost written beside it.

Day 6 — Keep the coding hand in, at the bar that applies to you

• Solve one medium problem in thirty minutes, then spend twenty more making it production-shaped: named invariants, validation at the boundary, and errors that distinguish a caller mistake from an internal fault.
• Write the tests you would require of a colleague's version of that function: one for empty input, one for the boundary, and one for the case the implementation is most likely to get wrong.
• Read a piece of your own code from six months ago and write the change you would ask for, phrased as you would actually phrase it in review.

Deliverable: One problem hardened to review standard, with its test list and one written review comment.

Day 7 — Defend it while being interrupted

• Run a forty-five-minute design mock with an interviewer briefed to change a requirement halfway, a tenfold traffic increase or a new strict consistency requirement, and to push on one number you estimated.
• Rehearse the two sentences a senior loop is listening for: naming the tradeoff you are choosing against and why, and saying what you would measure to learn that the choice was wrong.
• Prepare the design you regret: a real decision, the constraint that produced it, what it cost, and what you changed afterwards.

Deliverable: Mock notes recording how the design changed under the new requirement, plus a written account of one regretted decision.

Disagreement needs an outcome you did not choose

Conflict answers where you were right and everyone came round are the weakest ones. Stronger: the evidence you went and collected, what would have changed your mind, and what you did in the weeks after the call went against you. Implementing a design you argued against, properly, is a specific and checkable behaviour.

How do you prioritize tasks when working under tight deadlines with multiple stakeholders?

Medium · behavioural and collaboration

Approach

1. Pick a story where you made the decision, not one where you watched it.
2. Give the blast radius: what could have broken, and what you measured.
3. Close with what you would do differently, concretely.

Follow-up

• How did you know your change caused the improvement?
• What would you do differently if you ran that again?

Unblock an engineer without taking the keyboard

Easy · mentoring · leases · at-least-once

A teammate has spent two days on a job handler that occasionally writes duplicate rows. They are certain the queue is delivering twice by mistake. You suspect a lease expiring under a slow handler, so the job is running concurrently with itself. Describe how you have unblocked someone in this position: what you asked before offering a hypothesis, what you showed them rather than told them, and what you left them owning. Then say what you would do if their theory turned out to be the right one.

Approach

1. Ask before diagnosing, and ask for things answerable from data they already have: the attempt count on the job rows that produced duplicates, the handler's observed duration against its lease expiry, and whether the duplicate rows share a natural key that a unique constraint could have caught.
2. Teach the shape rather than the answer. A lease cannot distinguish a dead worker from a slow one, so a handler that outruns its lease is running twice by design, and deploys deliver the other half by killing handlers mid-run on every rollout. Both of their candidate theories produce identical duplicate rows, which is why the evidence has to come from timings rather than from argument.
3. Hand over a checklist they execute: a natural key on every write the handler performs so the second copy collides rather than appends, the record of intent written before any external effect, a lease heartbeat while running, and the metric that shows it working.
4. Keep ownership with them deliberately. Pair on the first write, then step back; if you finish it yourself you have closed one ticket and left the same person stuck on the next redelivery.
5. Close on the systemic gap that let two days pass, which is usually a missing dashboard for attempt counts or an undocumented at-least-once contract, and fix that rather than only the bug.

Follow-up

• How would you distinguish a genuine double-delivery from a lease expiry using only the data already stored?
• Their handler calls an external endpoint before recording that it did. What do you tell them to change first?
• What do you do the third time the same person brings you the same class of bug?

Narrate an outage you owned from page to postmortem

Hard · incident response · blast radius · postmortems

Pick an incident you personally drove, ideally one where writes were affected rather than reads. In six to eight minutes: state the symptom as it first appeared on a dashboard, the blast radius you established before you knew the cause, the mitigation you applied and when, the mechanism you eventually proved, and the follow-up that would prevent a repeat. Bring numbers: error rate, tenants affected, minutes to mitigate, minutes to resolve. If you cannot name what you measured, choose a different incident.

Approach

1. Open on the signal rather than the cause: which metric at which percentile moved, on which service, at what time, so the listener follows the same evidence you had rather than a conclusion you already reached.
2. Separate mitigation from diagnosis out loud. State what you did to stop the bleeding (flag off, shed traffic, drain a lease, roll back a deploy) and say plainly that you did it before the mechanism was known, because those are two jobs with different deadlines.
3. Establish blast radius in countable terms: how many tenants, how many writes, and crucially whether the effect was loss or only delay. An append-only revision table or a pending outbox row means the change survived and the projection was merely behind, which is a repair rather than a data-loss incident.
4. Prove the mechanism instead of asserting it. Name the trace span that grew, the plan that flipped to a sequential scan, the lease that expired, plus one alternative you ruled out and the signal that stayed flat while you ruled it out.
5. Close on the durable fix and its cost, distinguishing what landed that week from what needed an expand-and-contract migration across several deploys, and say which of the two you actually finished.

Follow-up

• What would you do differently in the first five minutes, given the same dashboard and no more information?
• Which follow-up action did you deliberately not take, and why was dropping it the right call?
• How did you convince yourself the mitigation was safe to apply while the cause was still unknown?