AI research lab Software Engineer Interview Guide

The 60-second brief

Candidates describe AI research lab as a company that builds infrastructure for AI model development: data-generation pipelines, evaluation tooling, agentic systems and human-in-the-loop workflows that researchers use to train and test frontier models. Candidates also report that its customers include foundation-model developers. The Software Engineer role sits on the shared platform layer. Reported duties include Kafka or RabbitMQ data pipelines, shared APIs and developer platforms, architecture standards for storage, compute and reliability, and collaboration with researchers on experiments. The stack candidates list is Python or JavaScript on GCP or AWS, and they report that the position is on-site in San Francisco.

This guide is for Software Engineer candidates preparing for the four reported rounds: Initial Screening, Technical Assessments, Deep-Dive Discussions and Final Decision. It covers the reported coding fundamentals (string reversal, binary tree views and traversals, stack and linked list operations, threads), the reported design and architecture questions (a memory-lean board game, Kafka leader election and consumer rebalancing, chat-to-agent matching, an MLOps pipeline), and the behavioral questions. Candidates describe requirements of 5+ years in distributed systems or platform infrastructure, so the plan spends as much time on explaining production work as on writing code.

Candidates report four rounds over roughly 3-5 weeks: Initial Screening, Technical Assessments, Deep-Dive Discussions and Final Decision. The reported questions mix coding fundamentals, system design and messaging internals, and behavioral questions about past projects.

Focus areas

• Data Structures & Algorithms (DSA)
• System Design
• Problem Solving

Overview

The work candidates describe is infrastructure for AI research: pipelines that move large volumes of events through Kafka or RabbitMQ, APIs and internal platforms that product engineers and researchers build on, and storage, orchestration and reliability standards for the whole organisation. Nice-to-have experience that candidates mention includes LLM evaluation systems, human-in-the-loop workflow orchestration and internal developer platforms. For preparation this means two things: you need to write correct, clean code on request, and you need to talk about queues, partitions, consumers, failure and observability from experience rather than from a diagram.

The reported questions fall into three groups. The fundamentals group is small and classic: reversing a string into a new variable, the left view of a binary tree, level order traversal, stack and linked list operations, and printing odd and even numbers with threads. The design and architecture group is varied and should not be treated as one topic. It includes a snake and ladder game with minimal memory, Kafka partition leader election, consumer rebalancing when nodes join or leave, matching customer chat requests to available agents with load balancing, and an MLOps framework with a pipeline for it. The behavioral group asks you to describe yourself and your past projects, say what you want from a new role, and explain how you scope requirements and plan iterative delivery for complex components.

Candidates also quote example scenarios: a pipeline handling millions of events per second reliably, a bottleneck you found and removed in a distributed system, a thread-safe producer-consumer implementation, and reducing an algorithm's space use without losing performance. Candidates are told to be ready to discuss every project on the resume in depth, including why each technical decision was made. Plan for breadth across the fundamentals, depth on Kafka and consumer groups, and a prepared set of project stories with real numbers and trade-offs.

The AI research lab interview process

These are the stages candidates describe, not a published process. The sources do not assign reported questions to specific rounds, so the questions are grouped by type below instead of by stage.

Initial Screening

Candidates describe this first stage as a review of whether your experience and skills fit the role. No format is reported beyond that, so treat it as a conversation about your background against the stack candidates list: distributed systems or platform infrastructure, Python or JavaScript, GCP or AWS, and Kafka or RabbitMQ pipelines. The questions about describing yourself and what you want from a new role are natural preparation material here. Have a short, specific account of one or two systems you owned and be ready to say why AI infrastructure appeals to you.

What to demonstrate

• Candidates report a review of fit between your experience and skills and the role, which they describe as asking for 5+ years in distributed systems or platform infrastructure
• Candidates report the self-introduction and what-are-you-looking-for questions, so expect to summarise your background and past projects clearly
• Candidates list Python or JavaScript, GCP or AWS, and Kafka or RabbitMQ as the stack, so be ready to show your match to it

How to prepare

• Write a spoken summary of your career built around two systems you owned, each with its scale, your decision and the result
• Make a one-line inventory of your experience with queues, cloud platforms and your main language, with an example for each
• Write a specific answer to what you are looking for in a new role that points at pipelines, platforms or researcher-facing tooling
• Be ready to discuss working on-site in San Francisco, since candidates report that arrangement

Technical Assessments

Candidates describe this stage as the hands-on technical part of the loop, where coding, system design and applying both to practical problems come up. The sources do not say which reported question belongs to which round, so the coding and design questions below are best used as preparation material for this stage. The coding questions reported are fundamentals on strings, trees, stacks, linked lists and threads. The design questions range from a memory-lean snake and ladder game to Kafka leader election, consumer rebalancing, chat-to-agent matching and an MLOps pipeline. Practise writing working code in your chosen language, then switch to explaining trade-offs aloud.

What to demonstrate

• Candidates report coding questions on trees, linked lists, stacks and strings, plus a threads exercise
• Candidates report concurrency topics such as thread-safe code and shared-resource handling, and a producer-consumer example
• Candidates report design questions about scalability, reliability and trade-offs in distributed systems
• Candidates are told to expect discussion of how messaging systems such as Kafka or RabbitMQ work internally and stay available in production

How to prepare

• Implement the string, left view, level order, stack and linked list questions from a blank file in Python or JavaScript and state the complexity of each
• Write the odd and even thread printer with a condition variable or semaphores, then a bounded producer-consumer queue
• Write a one-page explanation of Kafka partition leaders, in-sync replicas and consumer group rebalancing that you could say from memory
• Outline designs for chat-to-agent matching and an MLOps pipeline, naming the queue, the state store and the failure behaviour of each

Deep-Dive Discussions

Candidates describe this stage as conversations with engineering leaders, who go through your earlier projects and how you solved problems. They are also told to expect detailed questions about the reasoning behind every project on the resume. For preparation, the scoping question about planning iterative delivery for complex components fits this territory, and so does a bottleneck story from a distributed system, a scenario candidates quote without naming a round. Walk in with a dossier for each project: the problem, your role, the alternatives you rejected and what changed after launch.

What to demonstrate

• Candidates are told to be ready to discuss every resume project in depth, including the decisions you personally made
• Candidates are told to be ready to talk about owning systems end to end, including how you set up observability and monitoring
• Candidates describe being asked to justify trade-offs and to discuss edge cases in distributed environments

How to prepare

• For each resume project, write the decision, the alternative you rejected, the metric you watched and what you would change now
• Prepare a bottleneck story: how you detected it, the evidence that located it, the fix and the measured result
• Prepare an example of scoping a complex component into increments, including what you cut and how you changed the plan
• Rehearse stating a technology choice and defending it on reliability and performance grounds

Final Decision

Candidates describe the last stage as the decision, which follows the earlier assessments. No separate format or question set is reported for it, so the useful preparation is to make the earlier stages count and close any gaps while you still can. Keep a list of answers you were unsure about during the loop and review the topics behind them, especially Kafka internals and concurrency, in case they come back.

What to demonstrate

• Candidates describe the decision as following your performance in the earlier stages
• Because no separate questions are reported, consistency between your coding, design and project discussions is the safest thing to prepare

How to prepare

• After each stage, write down which question went worst and rework it the same day
• Keep one fully worked answer ready for each reported design topic: Kafka leader election, consumer rebalancing, chat matching and the MLOps pipeline
• Prepare your own questions about the platform, the researcher workflows it supports and how engineering standards are set
• Confirm logistics for an on-site San Francisco role so no open question remains

Where candidates lose points

Answering string reversal or the stack and linked list operations with a one-liner and no discussion

Write both the new-string version and a two-pointer in-place version, and note that strings are immutable in Python and JavaScript, so in place means working on a list or array of characters. Mention how multi-byte characters and surrogate pairs affect reversal, and state time and space for each version.

Implementing the left view of a binary tree by following only left children

Visit every level and keep the first node seen at each depth, using a breadth-first queue or a depth-first walk that visits left before right and records a node when depth equals the result length. Test it on a tree where the left subtree is shorter than the right.

Printing odd and even numbers from two threads with busy-waiting, sleeps or unsynchronised flags

Use a condition variable or a pair of semaphores so each thread waits for its turn and wakes the other. Handle termination so neither thread waits forever after the last number, and be able to say why the order is not guaranteed without synchronisation.

Saying a Kafka partition leader is simply elected, without how or what is at risk

Explain that each partition has a leader chosen from the in-sync replicas, that the controller drives the change, and that electing an out-of-sync replica trades availability for possible data loss. Tie it to producer acknowledgement settings, and connect it to consumer rebalancing.

Describing past projects as team results with no personal decisions, numbers or trade-offs

For each resume project, write the decision you made, the alternative you rejected, the scale and the measured outcome. Rehearse the failure modes and the monitoring you set up, since candidates are told to expect detailed questions on every project.

Questions & practice

Implement a Left View of a binary tree.

Medium · Technical Fundamentals and DSA

Approach

1. Define the output: the first node visible at each depth when you look from the left. It is not just the left spine. If the left subtree is shallower, nodes from the right subtree appear at the deeper levels.
2. BFS version: put the root in a deque. At each level, record len(queue), dequeue that many nodes, append the first one to the result and enqueue children left then right. O(n) time, O(w) space, where w is the widest level.
3. DFS version: preorder that visits left before right and carries the depth. When depth == len(result), this is the first node seen at that depth, so append it. O(n) time, O(h) stack, and h reaches n on a skewed tree.
4. Edge cases: an empty tree returns an empty list, and a node with only a right child must still show up. On deep skewed input, recursive DFS can hit Python's default recursion limit of 1000, so switch to an explicit stack or to BFS.

Follow-up

• Return the right view instead. Which line of your code changes?
• Compute the top view, where horizontal distance matters rather than depth.
• Which version would you ship for a tree with a million nodes that is close to a linked list, and why?

Implement stack and linked list operations.

Medium · Technical Fundamentals and DSA

Approach

1. Linked list: a Node with value and next, and a list holding head (and tail, for O(1) append). Insert at the head is O(1). Search and delete by value are O(n). A dummy head node removes the special case for deleting the first node.
2. Reverse iteratively with three pointers (prev, curr, next): O(n) time, O(1) space. Mention that the recursive version uses O(n) stack.
3. Stack: push, pop and peek in O(1). Build it on a linked list by inserting and removing at the head, or on a dynamic array where push is amortized O(1). The array version has better cache locality. The list version never pays for a resize copy.
4. Edge cases: pop or peek on an empty stack (raise an exception rather than return a magic value), deleting the only node, deleting the tail (update the tail pointer), and a value that is not in the list. Run each one on paper before saying the code is done.

Follow-up

• Add getMin in O(1). What extra state do you keep, and what happens on pop?
• Detect a cycle in the list without extra memory (Floyd's tortoise and hare), then find where the cycle starts.
• Implement a queue with two stacks and give the amortized cost of each operation.

Write a program to print odd and even numbers using threads.

Medium · Technical Fundamentals and DSA

Approach

1. State the invariant: one shared counter, the odd thread prints only when counter % 2 == 1 and the even thread only when it is 0, so the output is 1, 2, 3 ... N in strict order. Reading the counter, printing and incrementing must all happen under one lock.
2. Condition-variable version: each thread takes the lock, waits in a while loop until it is its turn or counter > N, prints, increments, then calls notify_all. In Python, Condition.wait_for(predicate) does the loop for you. Use while, not if, because of spurious wakeups.
3. Semaphore version: odd_sem starts at 1, even_sem at 0. The odd thread acquires odd_sem, prints, increments and releases even_sem, and the even thread mirrors it. The handoff order is enforced without a shared predicate.
4. For JavaScript candidates: Node runs your code on one thread, so real threads mean worker_threads sharing a SharedArrayBuffer, with Atomics.wait and Atomics.notify handing the turn back and forth. If you are free to choose, pick Python for this question.
5. Termination is the usual bug. When counter passes N, the finishing thread must still notify or release so the other thread is not left waiting, and both threads must check the limit after waking. Join both threads in main. A busy-wait on a flag works, but it burns CPU.

Follow-up

• Extend it to three threads printing 1, 2, 3 in round robin. Why can a single condition with notify() instead of notify_all stall?
• Turn this into a bounded producer-consumer queue. What do you wait on in put and take?
• Does Python's GIL make the lock unnecessary here? Explain why not.

Stop tag and share joins from fanning out a page

Medium · join fan-out · semi-joins · many-to-many

resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.

Approach

1. Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
2. Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
3. Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
4. Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.

Follow-up

• The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
• A resource can be shared with the same user twice under different permissions. Does your count change, and should it?

Hold a per-tenant active cap against concurrent creates

Hard · write skew · isolation levels · constraints

A tenant on the standard plan may hold at most 50 resources with status='active'. The create handler runs SELECT count(*) FROM resource WHERE tenant_id = $1 AND status = 'active', compares to 50, then inserts. Two creates arrive 3 ms apart on different instances and the tenant lands at 51. Name the anomaly, say whether PostgreSQL 16 READ COMMITTED or REPEATABLE READ prevents it and why, then give an implementation that holds the cap at READ COMMITTED with the exact statements. Finally, say what changes when the cap is 'at most one running export per tenant' on job_run.

Approach

1. Name it: write skew. The two transactions read an overlapping set and write disjoint rows, so there is no row-level conflict for the engine to detect and each commit is individually legal.
2. Rule out the levels precisely. READ COMMITTED takes a fresh snapshot per statement and takes no lock on the counted rows, so both see 49. PostgreSQL's REPEATABLE READ is snapshot isolation: it removes non-repeatable reads and phantoms within the snapshot but still admits write skew, because the anomaly is not a re-read of a changed row, it is a read of a set that a concurrent transaction invalidates. Only SERIALIZABLE closes it, by tracking the read dependency and aborting one transaction with SQLSTATE 40001 — a guarantee that exists only if the application re-runs the whole transaction from the read.
3. Convert the set predicate into a single-row conflict: keep tenant.active_resource_count and run UPDATE tenant SET active_resource_count = active_resource_count + 1 WHERE tenant_id = $1 AND active_resource_count < 50 in the same transaction as the INSERT. Zero affected rows is the cap, returned as 409. The row lock serialises the decision at any isolation level, and contention is bounded to one tenant's row — which is also the fair-scheduling unit, unlike a global counter that would convoy every tenant behind one row.
4. State the cost you just took on: a counter is a second source of truth that can drift, so every path that changes status must adjust it inside the same transaction, and a periodic reconciliation has to exist, with resource_revision as the authority for what the count should have been.

Follow-up

• A resource moves from archived back to active. Which statements change, and what breaks if the counter update and the status change land in different transactions?
• The cap becomes plan-dependent and a plan can change mid-month. Where does the number 50 live, and who reads it?

Reverse a string and store the result in a new variable.

Medium · Technical Fundamentals and DSA

Approach

1. Start by asking about the language. In Python, Java and JavaScript strings are immutable, so any reversal builds a new string, and storing it in a new variable leaves the original unchanged. In Python that is reversed = s[::-1] or ''.join(reversed(s)): O(n) time and O(n) extra space.
2. Write the manual version too: convert to a character list, swap from both ends with two pointers until they meet, then join. Say why prepending in a loop (res = ch + res) is O(n^2): each step copies the whole partial string.
3. Cover Unicode. In JavaScript, s.split('').reverse().join('') splits surrogate pairs, so emoji come out garbled. [...s].reverse().join('') iterates by code point. Java's StringBuilder.reverse() keeps surrogate pairs together. Code-point reversal still breaks combining marks and multi-code-point emoji.
4. Name the edge cases and test them out loud: empty string, single character, a palindrome, and a null or None input (raise an error or return it as-is, but decide which).

Follow-up

• Reverse a character array in place with O(1) extra space. What changes?
• Reverse the order of the words in a sentence but keep each word's letters in order.
• How would you reverse by user-perceived characters (grapheme clusters) instead of code points?

Provide a program for level order traversal.

Medium · Technical Fundamentals and DSA

Approach

1. Use a FIFO queue. In Python that is collections.deque (popleft is O(1); list.pop(0) is O(n)). In JavaScript, Array.prototype.shift() is O(n) on large arrays, so keep a head index into the array or build a small linked queue. Push the root, then repeatedly dequeue a node, visit it and enqueue its non-null children left to right.
2. To group nodes by level, record the queue size at the start of each level and process exactly that many nodes into one inner list. That is cleaner than pushing level markers or (node, depth) pairs.
3. Complexity: O(n) time, and O(w) space for the widest level. In a perfect tree the last level holds (n+1)/2 nodes, so the worst case is O(n).
4. Handle the empty root (return []), and know the DFS alternative: pass depth down and append to result[depth]. It uses O(h) stack instead of a queue, but the node order within a level depends on visiting left first.

Follow-up

• Print the levels in zigzag order: left to right, then right to left.
• Return the levels bottom-up, or the average value of each level.
• Connect every node to its right neighbour on the same level using O(1) extra space.

Design a snake and ladder game with minimum memory usage.

Medium · System Design and Architecture

Approach

1. Pin down the rules first. Is the board 10x10 (cells 1-100)? Does a roll past 100 leave the player in place, or is an exact finish not required? Does a ladder landing on a snake head chain? These choices change both the code and the validation.
2. Memory: store only the jumps. A sparse map from start cell to end cell costs memory proportional to the number of snakes and ladders. A flat byte array jump[101] costs about 101 bytes. For a 100-cell board the byte array is often smaller in practice, because hash-map entries carry per-object overhead. Say that trade-off out loud.
3. Players need one small integer position each (a byte fits 0-100) plus a turn index into a player list. Do not store a Cell object per square. The move is pos = pos + roll if it stays on the board, then pos = jump[pos] if a jump exists.
4. Validate at setup: snakes go down and ladders go up, nothing starts on cell 100 or cell 1, no two jumps share a start cell, and following jumps never loops. Keep Dice as its own class (seedable, for tests) apart from Board and Game.

Follow-up

• What is the minimum number of dice rolls to reach 100? (BFS over cells 1-100 with edges to i+1..i+6 after applying jumps: O(6 x cells).)
• How would you save and restore a game in as few bytes as possible?
• How does the design change for an N x N board, several dice or many concurrent games on one server?

Explain how a Kafka broker partition leader election happens.

Medium · System Design and Architecture

Approach

1. Set up the model. Each partition has one leader that takes all produce requests, and followers that fetch from it. The ISR is the set of replicas caught up within replica.lag.time.max.ms. A record is committed, and the high watermark (the offset consumers can read up to) advances, once every ISR member has it, whatever acks the producer chose. acks=all only makes the producer wait for that point before it gets an acknowledgement; acks=1 acknowledges after the leader alone has the write.
2. One broker or node acts as controller. In ZooKeeper mode it is whoever creates the ephemeral /controller znode. In KRaft mode it is the Raft leader of the controller quorum, and metadata lives in the __cluster_metadata log. The controller detects a dead broker through an expired ZooKeeper session or missed KRaft heartbeats.
3. For each partition the dead broker led, the controller picks the first live replica in the assigned replica list that is also in the ISR and bumps the leader epoch. The change goes out as LeaderAndIsr requests (ZooKeeper) or metadata records (KRaft). Clients get NOT_LEADER_OR_FOLLOWER, refresh metadata and retry.
4. If no ISR replica is alive, the partition stays offline, because unclean.leader.election.enable defaults to false. Turning it on restores availability by promoting an out-of-sync replica, and that loses acknowledged writes. Followers use the leader epoch to truncate any divergent log tail.
5. Durability settings: replication factor 3, min.insync.replicas=2 and acks=all survive one broker loss without losing acked data. With acks=1, writes the old leader acknowledged but never replicated can be lost. auto.leader.rebalance.enable later moves leadership back to the preferred replica.

Follow-up

• The ISR shrinks to one replica while min.insync.replicas=2. What do acks=all producers see?
• What concretely changes in controller failover between ZooKeeper mode and KRaft mode?
• How do idempotent producers keep retries after a leader change from creating duplicates?

Design a system to match incoming chat requests from customers to available agents using load balancing.

Medium · System Design and Architecture

Approach

1. Components: a connection gateway (WebSockets) for customers and agents, a waiting queue per skill or language (FIFO, with optional priority tiers), a presence service fed by agent heartbeats (status, active chats, max concurrent chats), a matcher, and a store for sessions and transcripts.
2. Matching policy: route to the least-loaded eligible agent (active/max ratio), break ties by longest idle time, and filter by skill. A sorted set of available agents per skill, keyed by load, gives O(log n) updates and O(log n) picks.
3. Make assignment atomic so two matchers cannot both claim an agent's last slot. Use a conditional update (active_chats < max_chats) in the database or a Lua script in Redis, or give each skill queue a single matcher. Give every assignment an id so a retried event is idempotent.
4. Failures: on a missed agent heartbeat, return that agent's chats to the front of the queue. Drop customers who abandon. When no agent is free, show queue position, overflow to a related skill group or fall back to an offline ticket. Route messages across gateway nodes with pub/sub keyed by chat id.
5. Metrics that show whether the balancing works: wait-time percentiles, abandonment rate, agent utilisation and how evenly chats are spread across agents.

Follow-up

• An agent who just logged in gets flooded with new chats. How do you stop that?
• How do you prioritise VIP customers without starving everyone else?
• The matcher crashes midway through an assignment. What state can be left behind, and how do you recover?

Explain the MLOps framework and how you would architect a pipeline for it.

Medium · System Design and Architecture

Approach

1. Lay out the stages: ingest data with schema and distribution checks; build features as versioned datasets, with one shared feature definition for training and serving to avoid training-serving skew; train as an orchestrated DAG (Airflow, Kubeflow Pipelines, or Vertex AI or SageMaker Pipelines on GCP or AWS); evaluate; register; deploy; monitor.
2. Reproducibility: every run records the code commit, the data snapshot, hyperparameters, the environment image and its metrics in an experiment tracker. The model registry stores artifacts with lineage, so any production model can be traced and rebuilt.
3. Promotion gate: a candidate is promoted only if it beats the current production model on a fixed held-out set and slice metrics. Roll out with shadow traffic or a canary, and keep a one-step rollback to the previous registered version.
4. Monitoring closes the loop: input drift (PSI or KS tests per feature), prediction drift, latency and error rate right away, and true quality once delayed labels arrive. Any of these can trigger retraining (continuous training) or an alert.
5. Trade-offs to name: batch versus online inference, managed services versus self-hosted orchestration, and how much evaluation runs per commit versus per release. For LLM work, the eval stage becomes versioned eval sets plus human review queues.

Follow-up

• Offline metrics look good, but production quality dropped after deploy. Walk through the debugging.
• How would you make a training run from six months ago bit-for-bit reproducible?
• How does the pipeline change when the model is an LLM evaluated with human raters instead of fixed labels?

Exports duplicate a row range about once a week

Hard · leases · clock skew · idempotency

Roughly once a week an export writes a file containing a duplicated range of rows. The affected job_run rows show attempt = 1, status = succeeded, one started_at, and a lease_owner naming a different host from the one whose logs show the job starting. Leases last 30 seconds and are heartbeated every 10 from inside the handler; lease_expires_at is computed on the worker and compared against the database's now(). Find the mechanism, and give a fix that holds even if you cannot fix the clocks.

Approach

1. Start from the fact that eliminates the obvious answer. attempt = 1 means no retry was recorded, so this is not a re-run after failure; two workers ran the same row concurrently and the takeover path never touched the counter. lease_owner naming a host other than the one that started the job is the same statement from the other side.
2. Enumerate the mechanisms that cause a premature takeover, then find the signal that separates them. Either the lease genuinely expired because the heartbeat did not fire, which is what happens when the heartbeat runs on the handler's own thread and the handler makes a long blocking call, or it only appeared expired because two clocks disagree, since lease_expires_at is written from the worker's clock and evaluated against the database's. The discriminator is the distribution: incidents clustered on the longest exports indict the heartbeat, incidents clustered on one host indict skew. Measure both, and measure each host's offset against the database directly.
3. Read the reclaim query precisely. In PostgreSQL now() is transaction start time, not statement time, so a reclaimer holding a long transaction compares against an older timestamp than expected; clock_timestamp() is the statement-time function. This is worth ruling in or out before you redesign anything, because it changes which rows look expired.
4. Remove the second clock rather than trying to synchronise it. Issue and extend the lease in the database, with lease_expires_at = now() + interval '30 seconds' in both the claim and the heartbeat, so exactly one clock is ever compared and worker skew stops mattering to this predicate.

Follow-up

• The displaced worker has already streamed half the file to object storage. What makes that side effect safe to repeat?
• You now count takeovers. What alert fires on that counter, and at what threshold?

Suggested practice checklist

The plan follows the reported loop: fundamentals first, then concurrency, messaging internals, design questions, an MLOps and cloud-mapping day, and a final day of behavioral answers and a full mock run.

Day 1 — Map the loop and inventory your projects

• Write the four reported rounds and, beside each, the question types you expect: background, coding and design, project depth, final decision
• List every resume project and write the decision, rejected alternative, scale and outcome for each
• Write a spoken answer to describing yourself and your projects built around your two strongest systems

Deliverable: A one-page loop map plus a project dossier with a decision, alternative, scale and outcome for every resume project.

Day 2 — Coding fundamentals in your main language

• Reverse a string into a new variable, then in place, and note the Unicode and immutability caveats
• Write left view and level order traversal for a binary tree, with both breadth-first and depth-first versions of the left view
• Implement a stack and a singly linked list with push, pop, insert, delete and reverse, and state the complexity of each

Deliverable: Working, tested solutions for the string, left view, level order and stack/linked list questions in Python or JavaScript, with complexities noted.

Day 3 — Concurrency

• Write the odd and even thread printer with a condition variable, then again with semaphores, and handle clean termination
• Write a bounded, thread-safe producer-consumer queue and say what happens when the queue is full or empty

Deliverable: Two correct thread-synchronisation programs and a written note on the race condition in each.

Day 4 — Kafka internals and consumer groups

• Write a one-page walkthrough of partition leaders, in-sync replicas, controller role and what happens when a leader broker fails
• Write how a consumer group rebalances, the triggers, eager versus cooperative, static membership, the KIP-848 protocol and the settings that cause storms
• Add a paragraph contrasting RabbitMQ competing consumers with Kafka consumer groups

Deliverable: A one-page Kafka reference you can explain from memory, covering leader election and consumer rebalancing.

Day 5 — Design: memory-lean game and chat matching

• Design snake and ladder with minimal memory: store only the jump map and player positions, and discuss a compact encoding
• Design chat-to-agent matching with load balancing: queueing, agent availability state, assignment strategy, timeouts and what happens when an agent drops
• Say aloud what breaks first at ten times the traffic for the matching system, and where you accept stale data

Deliverable: Two written design outlines with components, data model, failure behaviour and one named trade-off each.

Day 6 — MLOps pipeline, cloud mapping and debugging

• Outline an MLOps pipeline: data ingestion on a queue, validation, training, evaluation, registry, deployment, monitoring and rollback
• Sketch a pipeline that handles millions of events per second, naming partitioning, back-pressure, retries and dead-letter handling
• Map that pipeline onto your cloud, GCP or AWS: managed Kafka (MSK or Confluent), Pub/Sub or SQS/SNS, and note where each differs from self-run Kafka on ordering and replay

Deliverable: An MLOps pipeline diagram with failure handling, a cloud mapping of the event pipeline, plus a written root-cause sequence for the duplicate export problem.

Day 7 — Behavioral answers and a full mock

• Write and rehearse your answers to what you are looking for in a new role and to scoping and iterative delivery of complex components
• Prepare your bottleneck story with the detection evidence, the fix and the measured result
• Run a mixed mock: one coding question, one Kafka question and one project deep dive, answered aloud
• Re-rehearse your day-1 self-introduction aloud and tighten it using what the mock run showed

Deliverable: Three rehearsed behavioral answers, a bottleneck story and notes from a mixed mock run showing what to fix.

Project ownership and role-fit answers

The behavioral questions candidates report cover your background, what you want from the next role, and how you break complex components into deliverable pieces. Candidates also describe in-depth conversations about past work and decisions. Prepare stories where you owned a production system, made a decision between alternatives and can show what happened afterwards.

1. Describe a time you encountered a bottleneck in a distributed system and how you resolved it.

How do you handle consumer rebalancing upon the addition or removal of nodes?

Medium · System Design and Architecture

Approach

1. This one is technical, not behavioral. Start from the protocol: consumers in a group join through the group coordinator, one member computes the partition assignment, and every add, remove or missed heartbeat (session.timeout.ms, max.poll.interval.ms) triggers a rebalance.
2. Compare the assignors. Range and round-robin with the eager protocol revoke every partition from every member on each rebalance, which stops the whole group. CooperativeStickyAssignor (incremental cooperative rebalancing) moves only the partitions that have to change, so the other members keep consuming.
3. Static membership (group.instance.id) lets a restarted consumer rejoin under the same identity within the session timeout without any rebalance. That matters for rolling deploys.
4. Mention the newer consumer group protocol from KIP-848, generally available in Kafka 4.0. The broker-side group coordinator computes assignments incrementally, so there is no group-wide synchronisation barrier and members that keep their partitions keep consuming.
5. Correctness during handoff: commit offsets in onPartitionsRevoked (or before giving partitions up), keep processing idempotent or transactional because delivery is at-least-once, and remember that a group with more consumers than partitions leaves the extra consumers idle.

Follow-up

• A consumer takes too long between poll() calls and keeps getting kicked out of the group. How do you diagnose it and fix it?
• How does the KIP-848 consumer group protocol change rebalancing compared with the classic client-side assignor, and what would you check before migrating a group?
• How do you get exactly-once results when a partition moves mid-batch?

Describe yourself and your past projects.

Medium · Behavioral and Role Fit

Approach

1. Build a short narrative arc instead of a job-by-job resume tour: your current scope, then two projects chosen to show owning a production system end to end, then where you are heading. Pick projects close to the work candidates describe: queue-based pipelines, platforms other engineers use, reliability or observability.
2. For each project give the problem, the scale in numbers you can defend, the decision you made and the alternative you rejected. Use I for what you did and name the team's part separately, so the listener can tell your contribution from the group's.
3. Prepare to go deep on anything you mention, because candidates are told every resume project can be examined. Do not include a project you cannot discuss at the level of data model, failure modes and metrics.
4. Close by tying your direction to the infrastructure for AI research that candidates describe, with one concrete reason such as wanting to build tooling that researchers use daily. The mistake that sinks this answer is a list of technologies with no decisions, outcomes or personal ownership.

Follow-up

• Which part of that project did you personally design and build?
• Why did you choose that queue or storage technology over the alternatives?
• What would you change if you rebuilt it today?

What are you looking for in a new role?

Medium · Behavioral and Role Fit

Approach

1. Give two or three specific criteria, each tied to something you have already done. For example, owning shared infrastructure that other engineers and researchers build on, working on high-volume pipelines, and having a say in engineering standards, because you set up monitoring and standards in your last role.
2. Link each criterion to the role as candidates describe it: platforms and APIs for researchers, pipelines on Kafka or RabbitMQ, and reliability for customers who build foundation models. Show you know what the work is, not that you want a generic step up.
3. Be honest about trade-offs you accept. Candidates report the position is on-site in San Francisco, so say plainly whether that suits you and why.
4. Frame your current employer neutrally; your reasons for leaving should be about what you want to build, not complaints. The mistake that sinks this answer is a generic reply about growth and learning that would fit any company, or one that signals you want a product role when the work is infrastructure.

Follow-up

• What would make you leave this role after a year?
• What kind of work do you not want to do?
• How would working closely with researchers change how you build platforms?

How do you scope requirements and plan deliveries iteratively for complex components?

Medium · Behavioral and Role Fit

Approach

1. Choose a real component that was large and ambiguous, such as a pipeline, scheduler or internal API, and open with the situation in two sentences: who the users were, what was unclear and what the deadline pressure was.
2. Describe how you scoped it: asked consumers what they needed first, wrote down success criteria and non-goals, and listed the riskiest assumptions. Then describe slicing into thin end-to-end increments, starting with the smallest version that ran through every layer, with the riskiest piece spiked early.
3. Explain how the plan changed. Give a concrete increment, what you learned from shipping it behind a flag or to one consumer, what you cut or deferred and who agreed to that. Include the metrics or checks that told you each increment was safe to extend.
4. Say how you kept others informed: interface contracts written down, a short status note at each milestone, and how you handled a requirement that moved. The mistake that sinks this answer is describing a process framework by name without a project, or a plan that never changed.

Follow-up

• What did you cut, and who had to agree to that?
• How did you handle a requirement that changed halfway through?
• How did you estimate the first increment, and how close were you?