Interview guides · Capstan Medical

Capstan Medical Software Engineer Interview Guide

Capstan Medical Software Engineer interview guide: 3 reported rounds, IPC, real-time and OS questions, behavioral prompts and a 7-day prep plan.

The 60-second brief

Capstan Medical builds a robotic platform for minimally invasive heart valve treatment. Candidates describe the software role as work on the logic that turns surgeon input into catheter movement, on the real-time interface to the controls, and on the system services that keep a multi-process platform stable and safe during a procedure. The reported stack is C++ and Python on Linux, with Rust named as an advantage. The day-to-day also involves controls, mechanical, clinical, GUI and systems engineers, plus design reviews and the documentation that device validation needs.

This guide is for experienced software and systems engineers preparing for the Software Engineer loop at Capstan Medical. The requirements candidates describe include 5+ years of shipping complex systems, solid Linux knowledge (processes, IPC, kernel concepts) and the ability to debug across layers. The guide covers the three reported rounds and the twelve reported questions, which split into OS and real-time fundamentals, system and safety design, and behavioral prompts, and it ends with a seven-day plan.

Candidates report three stages over roughly 3-5 weeks: a Phone Screen with the hiring manager, an Onsite Visit with the manager and engineers, and a Site Tour of the hardware.

Focus areas

Python
Real-time systems
Multi-process robotic architecture

Overview

The reported work sits where software meets a physical device. Candidates mention real-time control loops, multi-process system state, hardening against security vulnerabilities, and low-level drivers or hardware abstraction layers. The listed requirements are C++ and Python, Rust as an advantage, deep Linux knowledge and the ability to debug across layers. ROS or ROS2, medical device or other safety-critical experience, distributed systems and system hardening are listed as nice-to-have.

That shapes what to practise. The twelve reported questions are conceptual and experience-driven: IPC in a high-reliability system, a race condition across process boundaries, scheduling policies under real-time constraints, memory safety in C++ or Rust, profiling, a state-management service, safety evaluation of a feature, maintainability, plus three behavioral prompts. Candidates describe the onsite as system design, debugging and device evaluation conversations rather than whiteboard syntax, so rehearse explaining a design out loud with concrete mechanisms (named primitives, failure modes, numbers you would measure). Keep Python and C++ fluent anyway, because both are on the stack.

Two themes recur across the reported topics. The first is determinism: latency treated as a safety property, priority inversion, buffers between fast control loops and slower services. The second is failure behaviour: sensor noise and data validation, fail-safe states when hardware communication is interrupted, and the documentation and verification a medical device needs. For every answer you prepare, say what happens when the component fails or the data is bad, not only what happens when it works.

Use your own projects as evidence. Candidates are told to be exact about what their role was and why they made specific trade-offs, and to follow the data end to end from sensor input to the display. Before day one of the plan, pick three projects and note for each the layers it touched, one cross-layer bug, one decision you defended and one place where verification or documentation shaped your schedule.

The Capstan Medical interview process

Candidates describe three stages, from a conversation with the hiring manager to an onsite and a tour of the hardware. The format below is what candidates report, not a published process.

Phone Screen

Candidates describe the Phone Screen as a conversation with the hiring manager about your background and your interest in the company's mission. No technical question is reported for this stage, so treat it as the place to make your experience legible: what you built, which layers you worked across, and why a heart valve robotics platform appeals to you. Be ready to link your history to the stack candidates list: C++, Python, Linux internals and cross-layer debugging.

What to demonstrate

How clearly you describe your background and your own role in past systems
Whether your experience maps to the listed stack: C++, Python, Linux, IPC and cross-layer debugging
Your stated interest in the mission of the product

How to prepare

Write a two-minute walk through your career that picks out one systems project with processes, real-time constraints or hardware in it
Match each listed must-have (C++, Python, Linux process management and IPC, multi-layer debugging) to a concrete project of yours, and note the gaps honestly
Read enough about transcatheter valve treatment to say why the problem interests you, without claiming clinical expertise
Prepare questions about the team, the hybrid onsite arrangement candidates mention, and what the next stage covers

Onsite Visit

Candidates report the Onsite Visit as professional but casual, with the manager and other engineers, and centred on system design, debugging and device evaluation. The reported question set suggests the kinds of conversation to rehearse: OS and real-time fundamentals such as IPC, scheduling and memory safety, architecture of a state-management service, safety evaluation of a feature, and cross-layer debugging. Candidates are advised to explain their reasoning aloud, so answer with a structure: requirements, mechanism, failure modes, how you would verify it.

What to demonstrate

Working knowledge of Linux processes, IPC, threading and memory management
Design reasoning for multi-component systems that interact with hardware
Debugging method across process and hardware boundaries
How you reason about the safety of a change in a medical device

How to prepare

Rehearse the IPC, scheduling, memory-safety and profiling questions aloud, naming specific mechanisms and trade-offs rather than general principles
Sketch a system-state service for a multi-component robot on paper: owners of state, update paths, what happens on a crashed component
Write a repeatable safety-evaluation checklist for a new feature: hazards, failure modes, mitigations, verification evidence, documentation
Prepare two debugging stories, one cross-process and one hardware-software, each with how you proved the cause

Site Tour

The Site Tour is described by candidates as a chance to see the hardware you would help control. No questions are reported for it. Treat it as a conversation about the physical system, and connect what you see to the hardware-software topics candidates mention: sensor noise and data validation, fail-safe states when communication is interrupted, and drivers or hardware abstraction layers.

What to demonstrate

How you relate software concerns to the physical system in front of you
Curiosity about sensors, actuators and the interfaces your code would depend on

How to prepare

Write a short list of questions about sensing, actuation, communication links and how software is tested against real hardware
Rehearse your hardware-software integration story so you can tell it with concrete detail if the tour leads there
Revisit how you would validate sensor data and what safe state the system should enter when a link drops

Where candidates lose points

Describing IPC as a list of mechanisms with no failure semantics

For the IPC question, pick one design and state message ownership, ordering, back-pressure, behaviour when the peer crashes or stalls, and how a receiver detects stale data. Compare shared memory, sockets and message queues on latency, copy cost and isolation.

Treating real-time as meaning fast

Define it as bounded, predictable response. Be ready to explain priority inversion and how priority inheritance or ceiling protocols address it, what a real-time scheduling class or PREEMPT_RT changes, and why unbounded work (allocation, locking, page faults) is kept out of control loops.

Debugging a cross-process race by adding prints or sleeps

Describe a method: capture event ordering with tracing or timestamped logs, run ThreadSanitizer or stress tests where they apply, shrink the window that makes the race reproducible, then fix the ownership or synchronisation and write a test that forces the interleaving.

Answering safety and maintainability questions like a web feature rollout

Frame feature evaluation around hazards, failure modes, fail-safe states, verification evidence and the documentation a medical device needs. Say what you would test before shipping, not what you would monitor afterwards.

Telling project stories in the plural, with the role unclear

Candidates are advised to be exact about their own role. Say what you decided, what you built, what you measured and what you would change, and be ready to explain the trade-offs you made.

Questions & practice

How do you ensure memory safety when working with C++ or Rust in a safety-critical context?

Medium · Technical Foundations & OS Concepts

Approach

In C++, make ownership explicit with RAII: unique_ptr by default, shared_ptr only where ownership really is shared, no raw new/delete. Treat span and string_view as non-owning, and check every lifetime. Use bounds-checked access or a hardened standard library mode (for example libstdc++ _GLIBCXX_ASSERTIONS) and a coding standard such as MISRA C++ or AUTOSAR C++14.
Back the rules with tooling in CI: -Wall -Wextra -Werror, AddressSanitizer and UBSan builds, ThreadSanitizer in a separate build (it cannot be combined with ASan), static analysis (clang-tidy, cppcheck or a commercial analyser), and fuzzing of parsers and message decoders with libFuzzer.
In Rust, safe code rules out use-after-free and data races at compile time. Keep unsafe blocks small, wrap each one in a safe API with its invariant written in a SAFETY comment, and run them under Miri. The FFI boundary to C or C++ is where the guarantees stop, so validate everything that crosses it.
Know the gaps. Integer overflow panics in debug builds but wraps in release unless overflow-checks is enabled. Leaks (Rc cycles), deadlocks and panics are still possible, so choose panic=abort or a defined recovery path for the device.
Memory safety alone does not make a system deterministic. In real-time paths, allocate everything at initialisation, use fixed-capacity containers, analyse stack depth, and enforce no-allocation with a hooked allocator or test that fails on any malloc after startup.

Follow-up

What can still go wrong in a program written entirely in safe Rust?
How would you introduce Rust into an existing C++ codebase incrementally?
Why should the control loop never call malloc, and how would you enforce that rule?

Diff a projection against the primary without per-row point reads

Hard · reconciliation · range hashing · throttling

The listing projection has drifted and some rows show a stale version. The primary holds 40,000,000 resource rows across 12,000 tenants while serving 1,200 writes and 14,000 reads per second. The obvious repair, reading each resource row and comparing its version against the projection, is correct and would eventually finish. Explain precisely why it is unacceptable here, then give a diff that finds the differing rows, state its complexity, and make it safe to run against a live primary. Replication lag is usually under 100 ms and is not bounded.

Approach

Quantify the naive cost rather than calling it slow: 40,000,000 point reads at even 0.5 ms each is over five hours serialised, and the only lever is concurrency, which is exactly what you cannot spend. The primary's pool is sized for the write path, and 40,000,000 random reads evict the buffer cache that sustains the 85 percent cache hit rate, so the audit degrades the system it is auditing.
Replace random access with one ordered pass per side. Both sides can be read in (tenant_id, resource_id) order, which is a sequential scan on each and a merge join in O(n) time and O(1) memory. For a dense diff that is the whole answer, and it reads the primary once instead of 40,000,000 times.
For the expected sparse case, compare range hashes instead of rows: partition the key space, compute per range an order-independent aggregate over hash(resource_id, version), compare aggregates, and descend only into ranges that differ. With d differing rows and branching factor B, at most d ranges mismatch per level, so the drill-down examines O(d log_B(n/d)) ranges and reads full rows only in mismatching leaves.
Aggregate with a sum modulo 2^64 or a multiset hash, never XOR. XOR is order-independent but self-cancelling, so two rows wrong in the same way, or a row duplicated on one side, leave the range aggregate matching and the range is declared clean.

Follow-up

The diff reports 900 stale rows. How do you decide between patching those rows and rebuilding the projection from resource_revision?
Same job, but the projection lives in a search index that cannot be scanned in key order. What changes?

Collapse a redelivered event batch into per-aggregate high-water marks

Easy · hashing · at-least-once · aggregation

You drain a batch of up to 5,000,000 events, each (aggregate_id BIGINT, aggregate_version INT, event_type, payload). The log guarantees order within one aggregate only; the batch merges 64 partitions, and a relay failover has redelivered a range, so an older version for an aggregate can appear after a newer one. Given a map of last_applied_version per aggregate, produce the events worth applying, at most one per (aggregate_id, version), plus the count discarded. Target O(n) time. State the memory for 2,000,000 distinct aggregates and what you do when it does not fit.

Approach

One pass, one hash map from aggregate_id to the highest version kept, and a discard counter. An event whose version is at or below last_applied_version for its aggregate is dropped without further work, which is the whole reason the event carries its version rather than a delta. O(n) expected time, O(d) space in distinct aggregates.
Keep the maximum, never the last occurrence. The redelivered range means the final appearance of an aggregate in the batch can be an older version than one seen earlier in the same batch, so last-wins applies stale state over newer state and the projection regresses with no error anywhere.
Cost the memory instead of calling it large: an 8-byte key plus a 4-byte version is 12 bytes of payload, and an open-addressed table held at a 0.7 load factor costs roughly 17 bytes per entry before per-slot metadata, so 2,000,000 aggregates is tens of megabytes in a native layout and several times that in a runtime that boxes both key and value.
If the distinct set exceeds memory, partition on hash(aggregate_id) mod P and reduce each partition independently. Every event for one aggregate hashes to the same partition, so the per-partition result is exact and the merge is concatenation rather than a second reduction.

Follow-up

The payload is a patch rather than a snapshot, so applying only the highest version loses the intermediate changes. What changes in your reduction?
How do you detect that version 7 arrived while version 6 was never delivered, and what should the consumer do about the gap?

Keep soft-deleted accounts from blocking re-registration

Easy · soft delete · unique constraints · engine differences

app_user holds user_id, tenant_id, email CITEXT, password_hash (NULL for SSO principals), email_verified_at, auth_version, status ('invited','active','suspended','deactivated'), created_at, updated_at, deleted_at. Two live accounts for one address inside a tenant must be impossible, but an address freed by a soft delete must be reusable, and the same tenant may delete and re-register it repeatedly. Write the uniqueness DDL for PostgreSQL 16, then the equivalent for MySQL 8 where partial indexes do not exist, and say what each permits once three deleted rows already hold that address.

Approach

Start from what is actually unique: not (tenant_id, email), but (tenant_id, email) among live rows. PostgreSQL says that directly — CREATE UNIQUE INDEX app_user_live_email ON app_user (tenant_id, email) WHERE deleted_at IS NULL. A full constraint over the same two columns burns the address permanently the first time someone deletes an account.
Keep case-insensitivity in the type or the index, never in the application: CITEXT as given, or UNIQUE (tenant_id, lower(email)) as an expression index where the extension is unavailable. A case-sensitive unique column is exactly how two accounts for one human appear.
For MySQL 8 the predicate has to move inside the key: add a discriminator column that is a constant 0 while the row is live and is set to user_id on delete, with UNIQUE (tenant_id, email, deleted_marker). Live rows share the constant and still collide; deleted rows differ from each other and stop colliding.
State the NULL variant and its dependency: leaving the marker NULL for deleted rows also works, because a unique index treats NULLs as distinct — true in MySQL, and true in PostgreSQL only under the default NULLS DISTINCT, which PostgreSQL 15 lets you reverse. Check the polarity against the three existing deleted rows: constant-on-live is what preserves the collision you want, and reversing it silently admits duplicate live accounts.

Follow-up

A deleted account re-registers with the same address the next day. Do the old resource rows follow the new user_id, and how does the API keep the two principals apart?
How do you honour an erasure request while resource_revision.actor_user_id still references this table?

Denormalise tenant onto revisions and backfill it live

Hard · denormalisation · online migration · backfill

resource_revision (revision_id, resource_id, version, actor_user_id, change_kind, patch, request_id, created_at) has 400M rows and no tenant column; tenant_id lives only on resource. Two reads need it: a tenant-scoped audit feed ordered by created_at DESC, and an offboarding purge. Both join back to resource today. Justify adding tenant_id to resource_revision against those two reads, name the anomaly the copy introduces and the constraint that prevents it, then give the ordered migration for a live table taking 1.2k writes/second — the lock each step takes, how the backfill is batched, and where each step stops being reversible. PostgreSQL 16.

Approach

Justify from the access path rather than from taste. Without the column, the audit feed either scans resource_revision by created_at and discards other tenants' rows, or resolves the tenant's resource_ids first and probes with them — both proportional to the tenant's whole history rather than to one page. With (tenant_id, created_at DESC, revision_id DESC) it is a seek that stops at 50 rows, and the purge becomes a ranged delete instead of a join.
Name the cost exactly: a second copy of a fact can disagree with the first. Make the disagreement unwritable rather than documented — add UNIQUE (resource_id, tenant_id) on resource so it can serve as a foreign-key target, then FOREIGN KEY (resource_id, tenant_id) REFERENCES resource (resource_id, tenant_id) on the revision table. A revision can then only ever carry its parent's tenant.
Step one, expand: ALTER TABLE resource_revision ADD COLUMN tenant_id BIGINT NULL, with no default, so it is a catalogue change and no rewrite. It still needs ACCESS EXCLUSIVE for an instant, and that instant queues behind the longest open transaction on the table while every later query queues behind it — set lock_timeout to 2s and retry rather than wait.
Step two, dual-write: deploy the writer that populates tenant_id on every new revision while reads still use the join. Reversible by redeploying the previous build, because nothing reads the column yet.

Follow-up

The backfill is half finished and a rollback is required. What state is the table in, and what does the previous build do with a half-populated column?
How do you verify the backfill actually finished, given rows are still being inserted while it runs?

How would you manage inter-process communication (IPC) in a high-reliability robotic system?

Medium · Technical Foundations & OS Concepts

Approach

Split the traffic by what it needs before picking a mechanism: high-rate sensor and setpoint data where only the latest value matters, versus commands and state changes that must arrive once, in order, and be acknowledged. One mechanism for both usually serves neither well.
For the high-rate path I use preallocated shared memory (shm_open plus mmap, mlockall so it never pages) with a lock-free single-producer/single-consumer ring or a seqlock: the writer bumps a sequence counter before and after each write, and the reader retries if the count is odd or changed, so it never acts on a torn sample. No allocation and no blocking syscalls in the control loop.
For commands I use Unix domain sockets, POSIX message queues or the middleware's reliable channel (ROS 2/DDS with reliable QoS). Copying costs more, but ownership is clear and a dead peer shows up as EOF or EPIPE. If a mutex has to live in shared memory, it needs PTHREAD_PROCESS_SHARED, priority inheritance, and the robust attribute, so a crashed owner yields EOWNERDEAD rather than a deadlock.
Every message carries a schema version, a sequence number, a CLOCK_MONOTONIC timestamp and a CRC. Each consumer checks for freshness, and heartbeats with deadlines feed a watchdog: when data goes stale, the consumer moves to a defined safe state rather than reusing the last setpoint.
Trade-off to state out loud: shared memory gives the lowest latency but no fault isolation, since one process can corrupt another's view. Back it with fault-injection tests: kill a publisher mid-write, delay messages, and send malformed or out-of-order frames.

Follow-up

If the process publishing setpoints crashes halfway through a write, what does the motion controller see and do?
How would you bound and measure end-to-end latency from sensor to actuator command?
How do you change a message format without breaking a peer that is already deployed?

What are the trade-offs between different scheduling policies when managing real-time constraints?

Medium · Technical Foundations & OS Concepts

Approach

Name the Linux options and how each behaves. SCHED_FIFO is fixed priority and runs until it blocks, yields or is preempted by a higher priority. SCHED_RR is the same but round-robins among equal priorities. SCHED_DEADLINE is EDF with runtime/deadline/period parameters and admission control. SCHED_OTHER is the fair scheduler and gives no latency guarantee.
Give the theory and its trade-off. Rate-monotonic fixed priority is guaranteed schedulable below the Liu and Layland bound n(2^(1/n) - 1), which tends to about 69% utilisation (sufficient, not necessary). EDF can reach 100% on one CPU with implicit deadlines, but under overload it can miss deadlines unpredictably, while fixed priority sheds the lowest-priority work first.
Cover the failure modes. Priority inversion is fixed with priority-inheritance mutexes (PTHREAD_PRIO_INHERIT; PREEMPT_RT turns most kernel spinlocks into rt_mutexes). A runaway FIFO thread can starve a core. RT throttling (sched_rt_runtime_us, 950000 of 1000000 by default) can stall a real-time task that runs too long.
Describe the deployment practice: isolate cores for the control loop (isolcpus/nohz_full or cpusets), pin threads, mlockall and preallocate, and keep non-real-time services under the fair scheduler on other cores.
Judge by worst-case latency, not the average. Use cyclictest under realistic load and timestamp every loop iteration to get a histogram and a count of missed deadlines.

Follow-up

Walk through how priority inversion happens and how priority inheritance resolves it.
When would you pick SCHED_DEADLINE over SCHED_FIFO for a periodic control task?
How would you show that a fixed-rate control loop never misses its deadline under load?

Explain your approach to profiling a system to identify performance bottlenecks.

Medium · Technical Foundations & OS Concepts

Approach

Define the target first: throughput, tail latency or jitter. Reproduce it with a representative workload and record a baseline, because without a number there is no way to know whether a change helped.
Work top down and find which resource is the limit: CPU, memory, I/O, lock contention or scheduling. Use vmstat, pidstat, iostat and top, and apply the USE method (utilisation, saturation, errors) to each resource.
For CPU time, use perf record -g with flame graphs, and perf stat for IPC and cache misses. For time spent blocked, use off-CPU analysis (perf sched, bcc offcputime). In Python, py-spy samples a running process without restarting it, while cProfile is deterministic but adds overhead. Valgrind callgrind is precise but slow.
For latency spikes, averages hide the problem. Timestamp each loop iteration, keep a histogram and the worst cases, and correlate spikes with ftrace or trace-cmd events (preemption, interrupts, page faults).
Change one thing at a time, measure again against the baseline, and account for the observer effect of the profiler itself. Then add a benchmark to CI so a regression is caught early.

Follow-up

Average latency looks fine but there are rare spikes. How do you find their cause?
What are the trade-offs between sampling and instrumenting profilers?
How do you profile a timing-sensitive loop without disturbing its timing?

How would you architect a service to manage system state in a multi-component robotic platform?

Medium · System Design & Robotic Architecture

Approach

Model the platform as an explicit state machine with an enumerated transition table, for example Off, Initialising, Homing, Ready, Active, Fault and SafeStop. One state manager is the sole authority: it accepts transition requests, checks their preconditions, rejects illegal ones with a reason, and broadcasts the result.
Each component reports its own sub-state and health through heartbeats with deadlines. The system state is derived from those reports, and any fault or missed heartbeat drives a transition to a safe state. Recovering from a fault requires an explicit request; nothing resumes by itself.
Interfaces: transition requests go over request/response with acknowledgement, and the state is published with a sequence number and monotonic timestamp. A component that starts late gets the current state immediately (in ROS 2, transient-local durability does this). Conflicting requests are serialised in the manager in a defined priority order.
Plan for the manager itself failing. A watchdog restarts it, and after a restart it enters a safe state rather than resuming motion from persisted state. Hardware interlocks that do not depend on this service stay as the last line of defence.
Log every transition with its cause and source for traceability. Unit-test the transition table exhaustively, and consider model-checking it, because the table is small and fully enumerable.

Follow-up

Two components request conflicting transitions at the same moment. What happens?
Which safety functions belong in this service, and which must sit below it in firmware or hardware?
How does the system recover if the state manager crashes while the robot is moving?

Describe your approach to evaluating the safety of a new feature in a medical device.

Medium · System Design & Robotic Architecture

Approach

Start with hazard analysis, not code. In ISO 14971 terms, list the hazards the feature adds or changes, then trace each one: hazard, sequence of events, hazardous situation, harm. Estimate severity and probability, then use FMEA (bottom-up) and fault tree analysis (top-down) to find the failure paths.
Check the software safety class under IEC 62304 (A, B or C). It sets how much rigour the change needs: architecture documentation, unit verification and the handling of third-party software (SOUP).
Choose risk controls in the ISO 14971 order: inherent safety by design first, then protective measures (limits, interlocks, watchdogs, plausibility checks), and information for safety last. Then confirm that no control introduces a new hazard of its own.
Show traceability from requirement to risk control to verification test. Verify each control with unit and integration tests, fault injection, simulation and hardware-in-the-loop runs, and add use-error analysis (IEC 62366) for anything the user interacts with.
Run a change impact analysis on existing features and their verification, then make a residual-risk judgement. Treat the documentation as part of the feature, not an afterthought.

Follow-up

How does the software safety class change what you have to produce for this feature?
How would you justify using a third-party library in a safety-relevant path?
What evidence would convince you that a specific risk control actually works?

What strategies do you use to ensure code maintainability in a long-lived, complex robotics project?

Medium · System Design & Robotic Architecture

Approach

Set the architecture boundaries first. A hardware abstraction layer lets drivers and hardware revisions change without touching control logic. Modules communicate through narrow, versioned interfaces, and dependencies point one way, from application logic down to platform code.
Build a test pyramid suited to robotics: unit tests on pure logic (kinematics, state machines, filters), simulation tests for system behaviour, and hardware-in-the-loop tests for the real interfaces. Run them in CI with formatting, static analysis and sanitizer builds, and require code review before merge.
Version parameters and calibration data separately from code, with schema validation at load time. A wrong config is as dangerous as a code bug and is harder to spot in review.
Make the environment reproducible so the project can outlive its tooling: pin toolchains and dependencies, build in containers, and record architecture decision records explaining why choices were made, so later maintainers do not undo them by accident.
Manage technical debt deliberately. Refactor behind tests in small steps, deprecate interfaces with a migration path, and in a regulated product weigh each refactor against the verification it forces you to repeat.

Follow-up

How do you keep the simulator and the real hardware from drifting apart over the years?
When do you refactor a module that works but that nobody fully understands?
How do you handle a core dependency, such as a middleware version, reaching end of life?

Can you explain how you would debug a race condition occurring across process boundaries?

Medium · Technical Foundations & OS Concepts

Approach

Pin down the invariant that breaks and the shared resource behind it: a shared-memory segment, a file, a device, or message ordering on a socket. Then measure how often the failure reproduces, because every later step needs a reproduction rate to compare against.
Instrument with timestamps from CLOCK_MONOTONIC, which every process on the host shares. Each process logs sequence numbers into its own in-memory ring buffer that gets dumped on failure, and I merge the buffers into one timeline. Kernel tracing (ftrace sched_switch, LTTng, strace -f -tt) shows the real interleaving. ThreadSanitizer only sees threads inside one process, so it will not catch this.
Widen the race window to make it reproducible: add sleeps or yields at the suspected points, load the CPUs with stress-ng, and change affinity with taskset (same core versus different cores changes the interleavings). Then run thousands of iterations in a loop.
Fix the protocol rather than the timing. That means acquire/release atomics or a process-shared mutex in shared memory, an explicit ownership handoff by message, or atomic file operations (O_EXCL, write-then-rename, flock). Never fix it by adding a sleep.
Verify the fix under the same stress harness that reproduced the bug, run it far more times than the original failure interval, and keep that harness in CI as a regression test.

Follow-up

Adding logging made the bug disappear. Why, and what do you do next?
How do you show the fix works rather than just not seeing the failure again?
What if one of the processes is a vendor binary you cannot modify?

Describe a situation where you had to debug an issue that turned out to be a hardware-software interface probl

Medium · Behavioral & Collaboration

Describe a situation where you had to debug an issue that turned out to be a hardware-software interface problem.

Approach

Pick a story with a concrete, measurable symptom and your own role in it, such as intermittent sensor timeouts, corrupted encoder readings or bus errors under load, and say what was at stake on the device.
Show how you separated the layers. Capture what was actually on the wire (logic analyser, oscilloscope, CAN or serial sniffer) and compare it with what the driver read. That single comparison tells you whether the fault is in hardware or software.
List the hypotheses you ruled out, and the evidence for each: protocol details (byte order, register map, SPI mode CPOL/CPHA, clock rate), timing (interrupt latency, bus contention), electrical causes (noise, grounding), and a firmware version mismatch.
Give the root cause, the fix on each side, and the defensive check you added in software (CRC or plausibility checks, timeouts, a fail-safe state on lost communication) so a recurrence is detected rather than silently used.
What sinks this answer: blaming the hardware without evidence, or telling a story where someone else found the cause.

Follow-up

Which measurement told you the fault was in the interface and not in your code?
How did you work with the electrical or firmware engineers to confirm the cause?
What did you add so the next occurrence would be caught automatically?

Suggested practice checklist

The plan follows the reported loop: orientation for the Phone Screen, four days on the OS, real-time, design and safety questions that fit the Onsite Visit, a day for behavioral stories, and a final run-through that includes hardware questions for the Site Tour.

Day 1 — Map the loop and audit your own experience

List the three reported stages and the listed requirements: C++, Python, Rust, Linux processes and IPC, kernel concepts, cross-layer debugging; rate yourself on each
Pick three past projects and write for each the layers it touched, the hardest cross-layer bug, one decision you defended and one place documentation or verification shaped the plan
Write a two-minute career walk-through for the hiring manager and a short answer to why this product interests you

Deliverable: A requirements self-rating, three project fact sheets and a written two-minute walk-through with a mission answer.

Day 2 — IPC and cross-process races

Compare shared memory, Unix sockets, pipes and message queues on latency, copying, ordering, isolation and crash behaviour, then choose one for a high-reliability robot and defend it aloud
Write the failure matrix for that design: peer crash, stalled reader, full buffer, stale message, restart
Walk through debugging a cross-process race: capture ordering, shrink the reproduction, apply a sanitizer or stress test, fix ownership, write a test that forces the interleaving

Deliverable: A one-page IPC design with its failure matrix and a written debugging sequence for a cross-process race, both spoken aloud once.

Day 3 — Real-time scheduling and memory safety

Compare fixed-priority, round-robin, earliest-deadline-first and non-real-time scheduling on predictability, utilisation and starvation; explain priority inversion and priority inheritance
Write what PREEMPT_RT or an RTOS changes, and list the operations you would keep out of a control loop (allocation, blocking locks, page faults)
Answer the C++ or Rust memory-safety question: ownership and RAII, avoiding raw owning pointers, bounds checking, sanitizers, static analysis, and what unsafe Rust requires you to prove

Deliverable: A scheduling comparison table and a memory-safety checklist for safety-critical C++ or Rust, each explained aloud.

Day 4 — Profiling method and Python fluency

Write your profiling sequence: define the symptom and metric, reproduce it, measure with a sampling profiler or tracing, find CPU, memory, I/O or lock contention, change one thing, re-measure
Practise explaining how profiling differs for a real-time control loop (jitter and worst-case latency) from a throughput service

Deliverable: A written profiling procedure plus two finished and explained Python exercises.

Day 5 — System state, safety evaluation and maintainability

Design a system-state service for a multi-component robotic platform: single owner of state, how components publish and read it, versioning, behaviour when a component dies or goes silent, and what is safe to read stale
Write a safety evaluation for a new feature: hazards, failure modes, mitigations, verification evidence and the documents to update
List maintainability practices for a long-lived robotics codebase: stable interfaces, hardware abstraction layers, simulation-based tests, review discipline, recorded design decisions

Deliverable: One architecture sketch, one safety-evaluation template and one maintainability list, each spoken aloud.

Day 6 — Behavioral stories with technical detail

Write and rehearse your hardware-software bring-up story, including the debugging sequence and the change that prevented a repeat
Write the story of defending an architectural decision to a team with conflicting priorities, including what you conceded
Write the speed versus documentation story, with how you tiered changes by risk
Practise the hardware-software interface debugging prompt with a different story or a different angle on the same one

Deliverable: Four story outlines in first person, each with situation, decision, evidence, outcome and what you would change.

Day 7 — Full run-through and Site Tour questions

Run a timed mock: the system-state design, the cross-process race and the defended architectural decision, aloud and recorded
Review the recording for vague mechanisms, missing failure modes and stories where your role is unclear
Write five questions for the hardware tour about sensing, actuation, links and how software is tested against real devices

Deliverable: A recorded mock with notes on fixes, and a list of five tour questions.

Bring-up, trade-off and documentation stories

Candidates report the behavioral prompts as centred on cross-functional work: hardware-software integration failures, defending architecture to teams with different priorities, and keeping development fast while documentation requirements are met. For each, prepare a first-person story with a decision you owned, the evidence you used, the cost it carried and what you would change.

Describe a situation where you had to debug an issue that turned out to be a hardware-software interface problem.
Tell me about a time you explained a software trade-off to a mechanical or controls engineer.

How do you handle hardware-software integration failures during system bring-up?

Medium · System Design & Robotic Architecture

Approach

Answer with a method first, then anchor it in one real bring-up. The method is layered isolation: confirm power, clocks and the physical link, then the driver or firmware, then the application, changing one variable at a time against a known-good baseline.
Show how you separated software from device behaviour: a simulated or fake hardware layer to test logic alone, then instruments below the software (bus traces, logic analyzer or oscilloscope, timestamped driver logs) to compare what you commanded with what the device actually did.
Keep the hardware safe while you test: bring subsystems up one at a time, validate sensor values (range, rate, staleness) before trusting them, and command actuators only with conservative limits and a defined fail-safe state if communication drops.
Tell the story in the first person. State the symptom, your first hypothesis and why it was wrong, the evidence that located the fault (sensor, actuator, wiring, timing or code), and what you changed afterwards, such as a validation check, watchdog, regression test or bring-up checklist.
The mistake that sinks it: blaming the hardware or firmware team without evidence, or describing a fix that was a retry or a sleep. Name how you proved the cause.

Follow-up

How did you decide whether the fault was in the sensor, the actuator or your code?
How did you keep the hardware and people safe while you were testing?
What did you change afterwards so the same failure would be caught earlier?

Tell me about a time you had to defend a technical architectural decision to a team with conflicting prioritie

Medium · Behavioral & Collaboration

Tell me about a time you had to defend a technical architectural decision to a team with conflicting priorities.

Approach

Pick a decision you owned where the priorities truly conflicted, for example latency against isolation, schedule against verification effort, or a rewrite against incremental change. A decision someone else made and you merely supported does not work.
Set up in two sentences, then spend the time on the reasoning: at least two real options, the criteria you used, and each group's priority in its own terms. Translate trade-offs for non-software colleagues into things they care about, such as risk, schedule, test effort or response time.
Show evidence rather than seniority: a prototype, a measurement, a failure analysis or a design review document. Say what the data showed and how you shared it.
Show what you conceded. Describe the change you made after pushback, the cost the final decision still carried, and how you recorded the rationale so it outlived the meeting.
The mistake that sinks it: a story with no real alternative, one where you won by persistence, or one that claims the decision had no downside.

Follow-up

What did the opposing team get right, and what changed because of it?
What would you have done if the measurements had gone against your recommendation?
How did you record the decision so a new engineer could understand it later?

How do you balance the need for speed in development with the rigorous documentation requirements of a medical

Medium · Behavioral & Collaboration

How do you balance the need for speed in development with the rigorous documentation requirements of a medical device?

Approach

Do not present speed and documentation as a trade you simply balance. Describe a process that makes documentation a by-product of the work: requirement to design note to test, written alongside the change, with review checklists, templates and automated test reports.
Explain how you tier by risk. A change touching control logic, system state or safety behaviour gets full review, traceable tests and updated documents. Tooling, logging or cosmetic changes take a lighter path. State the criterion you use to classify a change.
Use one concrete story: a deadline, what you produced and when (before merge, not after release), what you refused to skip, and what it cost in schedule. Include how you kept documents in step with the code afterwards.
Tie it to the reported setting: candidates describe design reviews and documentation for medical device validation as part of the role. Say that you treat verification evidence as part of the deliverable, and do not cite regulatory standards you have not worked with.
The mistake that sinks it: calling documentation bureaucracy, or claiming a lack of process made you fast.

Follow-up

Which kind of change would you never fast-track, and why?
How do you keep documentation accurate after the code changes?
What do you do when the schedule will not allow the full verification you wanted?