1X Software Engineer Interview Guide

The 60-second brief

1X builds robots, and candidates describe the company's work as embodied AI: software that lets a robot sense its surroundings, move through them and act on them. For Software Engineers, the reported problem spaces are Operating Systems, Core Systems, Cloud Infrastructure and NEO Reliability. That spans code running on the robot, where a bug can show up as physical motion or a safety problem, through to cloud infrastructure for managing a fleet. Candidates also describe day-to-day work with mechanical and electrical engineers on getting software into physical builds, iteration on core systems, faster CI/CD, and making the software stack more reliable.

This guide is for engineers interviewing for a Software Engineer role at 1X, whether the target team is closer to the operating system and core systems, cloud infrastructure, or reliability. Candidates list strong C++, Python or Rust and solid Linux/Unix experience as requirements, along with designing and debugging distributed or real-time systems. ROS, Docker/Kubernetes, hardware-software integration and kernel or driver work are listed as nice-to-haves. The guide covers the four reported rounds, the ten reported questions, the coding and design scenarios candidates mention, and the behavioral stories this loop asks for.

Candidates report four rounds over roughly 3-5 weeks: Technical Screening, Systems Design, Coding Proficiency and Team-Based Problem Solving.

Focus areas

• Cloud & Infrastructure
• Operating Systems
• Core Systems Engineering

Overview

The reported questions at 1X stay close to systems that touch hardware. They cover a communication protocol for a distributed robotic system, the trade-offs between concurrency models on a real-time OS, high availability and data integrity for cloud infrastructure that serves robots, performance on constrained hardware, and validating software before it runs on a physical machine. The rest are experience questions about debugging a race condition or memory leak, handling a failure in a deployed system, managing technical debt, changing course mid-project, and an architectural decision you made. Design answers for this loop should cover what happens on the robot when the network, a sensor or a deadline fails, not only how the cloud side scales.

Candidates also mention scenarios outside the main question list. The design ones are a system that ingests sensor data from many robotic units and a CI/CD pipeline that keeps firmware stable for hardware deployment. The coding ones are a thread-safe task queue with priority requirements and a refactor that improves performance and readability without changing behavior. All four can be practised directly. For the queue, practise correct condition-variable code in your primary language; for the refactor, write tests before you change anything.

The reported topic areas are Cloud & Infrastructure, Operating Systems, Core Systems Engineering, Reliability Engineering, and Identity & Access Management (IAM). In practice that means reviewing Linux scheduling, threads, memory and I/O. It also means real-time concepts such as priority inversion and bounded latency, replicated and idempotent data pipelines, observability and incident handling, and per-device identity and credential rotation for a robot fleet. Prepare in your strongest language among C++, Python and Rust, and know its memory model and concurrency primitives well enough to explain how your code interacts with the OS.

Candidates point to past projects that bridge software and hardware as an advantage. Before the loop, pick two or three projects where you can explain the physical or operational consequences of your software choices. For each one, know the measurements, the failure modes and the trade-offs you rejected. Those stories support the debugging, architecture and pivot questions.

The 1X interview process

Candidates describe four rounds, starting with a technical screen and moving through systems design and coding to a collaborative problem-solving session. Below is what each stage is reported to cover and how to prepare for it.

Technical Screening

Candidates describe the Technical Screening as the first check of technical skills and knowledge. No more detail on its format is reported, so prepare for breadth. You should be able to give concise, correct answers on your primary language and on Linux and systems fundamentals. Lead with the core idea and a concrete example, then offer depth instead of reciting everything you know.

What to demonstrate

• Baseline technical skills, as candidates describe this first stage
• Breadth of technical knowledge relevant to the role

How to prepare

• Review your primary language (C++, Python or Rust) in depth: ownership or RAII, the memory model, standard concurrency primitives, and the common pitfalls such as iterator invalidation, the GIL, or Send/Sync.
• Refresh Linux fundamentals out loud: processes vs threads, scheduling policies (SCHED_FIFO and SCHED_RR in the real-time class vs SCHED_OTHER, which CFS served until EEVDF replaced it in Linux 6.6), virtual memory and page faults, file descriptors and blocking vs non-blocking I/O.
• Practise concise spoken answers on your primary language and Linux fundamentals, each with one example from your own work.

Systems Design

Candidates describe Systems Design as a discussion of how you would architect a system and the principles behind your choices. In the design area they report distributed state and communication across nodes, CPU, memory and power constraints, monitoring and error recovery, and more advanced material such as real-time scheduling and hardware-in-the-loop testing. The design scenarios they mention include sensor data ingestion from multiple robotic units and a CI/CD pipeline that keeps firmware stable. Structure each design as requirements, data and message classes, components, failure modes, then trade-offs, and say explicitly what the robot does when the cloud or the link is unavailable.

What to demonstrate

• Architecture reasoning for distributed systems that include robots and cloud services
• Design trade-offs you can state and defend
• Failure handling and recovery in the design

How to prepare

• Design sensor data ingestion from many robots end to end: on-robot buffering, idempotent upload keyed by robot id and sequence number, a replicated log, an object store for bulk data, and backpressure.
• Design a firmware CI/CD pipeline with build reproducibility, simulation and hardware-in-the-loop gates, canary units, A/B partitions and automatic rollback.
• Prepare design answers you can sketch from memory, with the failure modes labelled.

Coding Proficiency

Candidates describe Coding Proficiency as practical coding exercises. In the coding area they mention a thread-safe task queue with priority requirements and a refactor for performance and readability that must keep behavior unchanged, alongside data structures and algorithms, standard-library fluency and unit testing. Write compiling, tested code in your primary language. State complexity, cover shutdown and edge cases, and add tests that would catch a regression.

What to demonstrate

• Working code produced in practical exercises
• Correctness, efficiency and readability of the implementation
• Testing of your own code

How to prepare

• Implement a bounded priority task queue: a heap keyed on (priority, insertion counter) so equal priorities stay FIFO, one mutex, not_empty and not_full condition variables waited on in while loops, and a close() that wakes every waiter.
• Practise a refactor kata: write characterization tests first, then remove repeated allocation or copies, then rename and extract, and run the tests after each step.
• Do the overlapping-job-attempts and heaviest-tenants practice problems in your primary language under time, writing tests for empty input and boundary timestamps.

Team-Based Problem Solving

Candidates describe Team-Based Problem Solving as a collaborative session focused on teamwork and communication. No specific problem type is reported, so practise working through an open problem with another person. Restate the goal, share your plan before acting on it, invite input at decision points, and explain your reasoning as you go. Stories from past team projects are useful preparation for this stage.

What to demonstrate

• Teamwork while solving a shared problem
• Clear communication of technical reasoning

How to prepare

• Run a paired mock with a friend on an open design problem, such as adding remote diagnostics to a robot fleet, and practise checking in at each decision point.
• Prepare a story from a team project that names who disagreed and how the decision was resolved with evidence.
• Prepare a short explanation of one hardware-adjacent constraint, such as a control-loop deadline or a power budget, that a mechanical or electrical engineer would follow.

Where candidates lose points

Designing the robot communication or fleet cloud system as if it were an ordinary web service, with no answer for a dropped link, a late command or a cloud outage.

For every component in a design, say what the robot does when that component is slow or gone. Use command expiry, heartbeats with a safe-state fallback, durable on-robot buffering and idempotent uploads. Rehearse the communication-protocol and cloud high-availability questions until those failure paths come out without prompting.

Answering the real-time concurrency question with a list of primitives and no mention of priority inversion, bounded latency or schedulability.

Learn priority inheritance and priority ceiling, the rate-monotonic utilization bound, why allocation and page faults are kept off the real-time path, and what PREEMPT_RT and SCHED_FIFO change on Linux. Tie each concurrency model to the worst-case behavior it gives you.

Writing the thread-safe priority task queue with subtle concurrency bugs: an if instead of a while around the condition wait, no shutdown path, or equal-priority tasks coming out in arbitrary order.

Practise the queue until you can write it correctly from memory in your primary language. Wait in a loop on the predicate, use an insertion counter as a tie-breaker, bound the capacity, add close() semantics that wake all waiters, and write a multithreaded test that checks no task is lost or duplicated.

Telling the race-condition, memory-leak or deployed-failure story with no named mechanism and no proof the fix worked.

Prepare each debugging story with the exact symptom, the tools used (sanitizers, heap snapshots, core dumps, log replay), the root cause stated as a mechanism, the regression test that failed before the fix, and the monitoring added afterwards.

Claiming deep C++, Python or Rust proficiency but being unable to explain how the code interacts with the OS, memory or network.

Pick one primary language and review it below the surface: RAII and the C++ memory model, the GIL and multiprocessing in Python, or ownership and Send/Sync in Rust. Also know what system calls your typical code makes and where it blocks.

Questions & practice

Explain the trade-offs between different concurrency models in a real-time OS environment.

Medium · Technical & Systems Expertise

Approach

1. Name the models and the property that matters on a real-time OS, which is bounded worst-case latency rather than throughput. The models are preemptive priority threads sharing state behind locks, message passing between tasks over queues, a cooperative event loop, a time-triggered cyclic executive, and interrupt handlers that defer work to tasks.
2. Shared state with mutexes is familiar and cheap, but it risks priority inversion: a low-priority task holds a lock, a medium-priority task preempts it, and the high-priority task waits without bound. Mitigations are priority inheritance or priority ceiling mutexes, short critical sections, and no blocking calls while holding a lock.
3. Message passing removes most shared state and gives natural backpressure through bounded queues. The cost is copying and queueing latency, so use preallocated, fixed-size buffers. An event loop avoids locking entirely but one long handler blocks everything, which breaks deadlines. A cyclic executive is the most predictable model and the least flexible.
4. Lock-free structures avoid blocking. An SPSC ring buffer with monotonic head and tail indices has no ABA problem and needs only acquire/release ordering on those indices; CAS-based multi-producer structures such as Treiber stacks and MPMC queues do need ABA handling. Lock-free does not mean wait-free, so worst-case bounds still need analysis.
5. Tie the answer to schedulability. Rate-monotonic scheduling has the utilization bound n(2^(1/n) - 1), about 69 percent as n grows, while EDF can reach 100 percent under ideal assumptions. On Linux, mention PREEMPT_RT, SCHED_FIFO, mlockall, CPU isolation, and avoiding allocation and page faults on the real-time path.

Follow-up

• How would you detect priority inversion in a running system?
• When would you choose an event loop over threads for a component on the robot?
• What does PREEMPT_RT change compared with a stock Linux kernel?

Find overlapping job attempts and peak concurrency from lease records

Medium · sweep line · intervals · leases

A day of job_run history yields about 50,000,000 attempt records: (job_run_id, job_type, attempt, started_at, finished_at which is NULL when the worker died, lease_expires_at). Leases expire on a clock, so a job that outran its lease ran twice. Produce (a) every job_run_id whose attempts overlapped in wall-clock time and (b) the peak number of simultaneously running attempts per job_type with the minute it occurred. Target O(n log n). State how you treat a NULL finished_at and what clock skew does to your answer.

Approach

1. Define the interval before sorting anything: an attempt occupies [started_at, COALESCE(finished_at, lease_expires_at)). finished_at is observed and lease_expires_at is only a promise, so every attempt without a finish contributes an estimate and the whole result is a lower bound on overlap rather than an exact count.
2. For peak concurrency, sweep: emit 2n endpoints, sort by (timestamp, kind) with ends ordered before starts at equal timestamps, then walk the sequence maintaining a counter per job_type and record each type's maximum with its timestamp. O(n log n) dominated by the sort, O(n) space, or O(1) extra if the sort is external and the walk streams.
3. For overlap detection, do not compare attempts pairwise. A single global sort by (job_run_id, started_at) gives both the grouping and the order; within a group, keep the maximum end seen so far and report an overlap exactly when the next start is less than that running maximum, which is one linear pass after the sort.
4. Half-open intervals matter and are easy to get wrong: with closed intervals an attempt ending at the same millisecond another begins reads as concurrency two, and across 50,000,000 records that artefact swamps the real signal.

Follow-up

• A handler is not idempotent and you have found 400 overlapping jobs. Which of them actually caused damage, and what would you query to find out?
• Peak concurrency for one job_type is 4 against a configured cap of 4. Is the cap working, or is the data hiding attempts that never started?

Identify the heaviest tenants in a five-minute window under memory pressure

Medium · top-k · heavy hitters · streaming

The edge service handles about 3,000 requests per second across roughly 50,000 tenants, peaking near 9,000. Expose the 50 heaviest tenants by request count over the trailing five minutes so limits can be tightened before one tenant's backfill starves the fleet. You may not retain five minutes of raw records. Give the exact solution and its memory, then the bounded-memory approximation with its error stated as a formula, and say which you would ship and at what tenant cardinality that choice changes.

Approach

1. Do the exact version first, because it is affordable at this cardinality: a ring of 300 one-second counters per tenant, advanced lazily, is 1,200 bytes of counters per tenant and roughly 60 to 90 MB for 50,000 tenants with overhead. Carry a running total and subtract the bucket you overwrite so a window read is O(1) rather than 300 adds.
2. Extract the top 50 with a size-k min-heap over the tenant sums: O(d log k) for d tenants, against O(d log d) to sort them all. Maintaining the heap continuously instead of on query requires a tenant-to-heap-index map, because incrementing a count already inside the heap means sifting from a known position, and without that map you rebuild the heap on every request.
3. State the approximation precisely rather than gesturing at sketches. Misra-Gries with m counters retains every item whose true count exceeds N/(m+1), and each retained count underestimates the truth by at most N/(m+1). With m = 1,000 and N = 900,000 requests in the window the error is roughly 900 requests, which is fine for spotting a tenant sending 50,000 and useless for ranking two tenants 200 apart.
4. Say what breaks when the window slides: Misra-Gries and Space-Saving are insert-only and cannot be decremented as records age out. The workable construction is one summary per sub-window, say ten seconds, with 30 summaries merged at query time, and the merged error is the sum of the per-summary errors, so the bound degrades linearly in the number of sub-windows.

Follow-up

• The heaviest tenant is heavy because of one export job rather than user traffic. Should the limiter treat those as the same tenant?
• Two tenants sit tied at the boundary of the top 50. Does your answer flap, and does the flapping matter?

Stop tag and share joins from fanning out a page

Medium · join fan-out · semi-joins · many-to-many

resource_tag is (resource_id, tag_id) with PK (resource_id, tag_id); resource_share is (resource_id, shared_with_user_id, permission). The tagged-and-shared listing inner-joins resource to both, filters tenant_id, tag_id = ANY($2) and shared_with_user_id = $3, orders by updated_at DESC and takes 50. Pages come back with fewer than 50 distinct resources and the total in the header is far too high. Explain the row multiplication, rewrite both the page query and the count query so each is correct, and name the index each one needs. PostgreSQL 16.

Approach

1. Do the arithmetic against the predicates that are actually there. An inner join emits one row per matching child row, and both joins are filtered: tag_id = ANY($2) admits only the requested tags, shared_with_user_id = $3 admits one user's share rows. So a resource holding three of the requested tags and shared with $3 once yields three rows, not one — the multiplier is its count of matching tags times its share rows for that single user, and that second factor is 1 unless the table admits duplicate (resource_id, shared_with_user_id) pairs. LIMIT 50 then limits rows rather than resources, and COUNT(*) counts pairs — the header is the product, not the population.
2. Reject DISTINCT as the fix. It deduplicates after the product has been built, so the planner must materialise and sort the fanned-out set before the LIMIT can apply, and it leaves any SUM or AVG in the same select list wrong.
3. Rewrite both filters as semi-joins, keeping resource as the only row source: AND EXISTS (SELECT 1 FROM resource_tag rt WHERE rt.resource_id = r.resource_id AND rt.tag_id = ANY($2)) and the same shape against resource_share. A semi-join stops at the first match per resource and preserves the driving index order, so ORDER BY updated_at DESC, resource_id DESC LIMIT 50 still stops after 50 rows.
4. Count with the same predicates and no join at all: SELECT count(*) FROM resource r WHERE r.tenant_id = $1 AND r.status = 'active' AND EXISTS (...) AND EXISTS (...). Nothing multiplies a resource, so the number is the population.

Follow-up

• The filter changes from 'any of these tags' to 'all of these tags'. Rewrite it and state what it costs relative to the ANY form.
• A resource can be shared with the same user twice under different permissions. Does your count change, and should it?

Replace offset paging on the resource feed with keyset

Easy · keyset pagination · composite index · sort order

resource holds resource_id, tenant_id, owner_user_id, title, body_ref, version, status ('draft','active','archived','deleted'), created_at, updated_at, deleted_at, with an index on (tenant_id, status, updated_at DESC, resource_id DESC). The listing endpoint returns active resources for one tenant, newest update first, 50 per page, today with LIMIT 50 OFFSET n. Tenants reach page 400 and rows are created while they read. Write the keyset query, define what the cursor carries and how it is encoded, and say which part of the index each predicate uses. Assume PostgreSQL 16.

Approach

1. Name the two failures separately. OFFSET 20000 makes the server produce and discard 20,000 rows, so page cost grows with depth rather than with page size. Independently, any write that changes how many rows sort above the offset moves the window between two fetches, and the direction decides which anomaly you get: an insert lands at the head of updated_at DESC and pushes already-returned rows down past the boundary, so they are returned a second time; a delete above the offset, or a row whose updated_at is bumped above the cursor, pulls rows up and one is never returned at all. Nothing in the response reveals either.
2. Write the seek: WHERE tenant_id = $1 AND status = 'active' AND (updated_at, resource_id) < ($2, $3) ORDER BY updated_at DESC, resource_id DESC LIMIT 50. The row-value comparison is one index range rather than a disjunction, and both columns are NOT NULL, which is what makes that comparison well defined.
3. Map each predicate onto the index: tenant_id and status are equality on the leading columns, (updated_at, resource_id) is the range, and the ORDER BY matches the index order so no Sort node appears and the scan stops after 50 rows. The DESC in the definition only matters for mixed directions — a plain ascending btree on the same columns is read backwards for this query.
4. Put both sort columns in the cursor and nothing the client can tamper with into another tenant: base64 of (updated_at, resource_id), validated server-side, with tenant_id taken from the principal.

Follow-up

• The client asks for 'jump to page 400'. What do you offer instead, and what does the honest version cost?
• Sort order becomes user-selectable across four columns. How many indexes is that, and which would you refuse to add?

How would you design a robust communication protocol for a distributed robotic system?

Medium · Technical & Systems Expertise

Approach

1. Separate traffic into classes before choosing a transport. High-rate sensor streams can tolerate loss and only need the latest value. Control commands need ordering, bounded latency and acknowledgement. Configuration and state changes need reliable, idempotent delivery. Map each class to a transport or QoS policy, for example best-effort pub/sub for sensors, reliable delivery for commands, and ROS 2 QoS settings (reliability, history, deadline) if you use ROS.
2. Define the envelope: a versioned schema (protobuf, FlatBuffers or an IDL), source id, per-topic sequence number, monotonic timestamp and checksum. Sequence numbers detect gaps and reordering. Schema rules (additive fields only, never reuse a tag) let units on mixed software versions keep talking.
3. Design for link failure explicitly. Use heartbeats with a deadline. Give each command an expiry time so a delayed command is dropped rather than executed late. When a heartbeat is missed, the actuator side moves to a defined safe state such as hold or stop. Commands carry ids so at-least-once delivery plus deduplication is safe.
4. Keep the control path non-blocking. Use bounded queues with drop-oldest for sensor topics so a slow consumer never stalls a control loop, and use priority or separate channels so bulk telemetry cannot starve commands. Use synchronized clocks (PTP or NTP) for cross-node timestamps and monotonic clocks for timeouts.
5. Name the trade-offs. Retransmission improves reliability but adds jitter. Zero-copy formats such as FlatBuffers and Cap'n Proto still allow additive schema changes; what gets harder is evolving raw packed structs or shared-memory layouts that have no schema. Any link that leaves the robot needs authentication and encryption (mutual TLS, per-device credentials).

Follow-up

• What happens if the network partitions halfway through a multi-step command?
• How would you roll out a message schema change across a fleet running mixed software versions?
• How would you measure end-to-end latency and jitter for the control channel?

How do you ensure high availability and data integrity in cloud-based infrastructure for robotics?

Medium · Technical & Systems Expertise

Approach

1. Split the data by its integrity needs. High-volume telemetry and logs are append-only. Fleet state and configuration need strong consistency. Software releases and commands must never be duplicated or corrupted. Each class gets a different store and delivery guarantee.
2. Plan for intermittent connectivity. Robots buffer durably on local disk and upload with an idempotency key (robot id plus sequence number), so a retry after a timeout cannot double-write. On the server side, deduplicate on that key and keep consumers idempotent to get an exactly-once effect from at-least-once delivery.
3. Run stateless ingestion gateways across availability zones in front of a replicated log, then write bulk sensor data to an object store with content hashes checked on upload and on read. Use replicated databases for fleet state, with point-in-time recovery and backups that are proven by regular restore tests.
4. Keep the cloud out of the robot's safety-critical loop. Define SLOs per service and make sure a cloud outage only delays uploads and dashboards. Robots should keep operating safely and catch up later, with rate-limited reconnection so the whole fleet does not return at once.
5. Cover identity, which is a reported topic. Give each robot its own certificate, use mutual TLS, scope IAM roles to least privilege, automate credential rotation, and keep a revocation path for a lost or compromised unit.

Follow-up

• A robot comes back online with a week of buffered data. What happens to ingestion and to downstream consumers?
• What are your RPO and RTO for fleet state, and how would you prove you can meet them?
• How would you revoke one robot's credentials without disrupting the rest of the fleet?

How do you optimize software performance when working with constrained hardware resources?

Medium · Technical & Systems Expertise

Approach

1. Start from the budget: CPU share, memory ceiling, latency deadline or control-loop period, and power. Profile on the target hardware, not a development machine, using perf and flame graphs for CPU and heaptrack or Massif for memory, and measure worst-case and p99 latency, not the mean.
2. Fix the algorithm before the constants. Process sensor data at the rate the consumer needs, downsample or crop early, and avoid recomputing results that can be cached or done incrementally.
3. Then the constant factors: remove allocation from hot loops (preallocated buffers, object pools, arenas), lay data out contiguously for cache locality (structure of arrays where it helps), avoid copies between pipeline stages (zero-copy or shared memory), use SIMD or fixed-point arithmetic where precision allows, and offload to a GPU or accelerator when one is available.
4. Account for concurrency and power: pin real-time threads, reduce lock contention, batch work to cut wakeups when latency allows, and note that batching trades latency for throughput.
5. Lock in the gains with a benchmark under realistic load, a regression check in CI, and a stated before/after number. An optimization without a measurement is only a claim.

Follow-up

• Give a case where you traded memory for CPU or the reverse. How did you decide?
• The profiler shows no single hotspot. What do you do next?
• How do you stop a performance fix from regressing six months later?

How do you approach technical debt while maintaining rapid delivery cycles?

Medium · Problem-Solving & Engineering Mindset

Approach

1. Treat debt as a carrying cost and rank it by interest. Debt that slows every change in a hot area, or that sits on a safety or data-integrity path, is paid first. Cosmetic debt in code that rarely changes can wait.
2. Make it visible: record each item with its observed cost (minutes lost per build, incidents caused, features blocked) so trade-offs against features are argued with numbers.
3. Pay it down alongside delivery. Refactor the area a feature already touches, reserve a steady share of capacity, and use the strangler pattern or feature flags to replace components incrementally instead of in a big rewrite.
4. Name where you would not take shortcuts: interfaces and persisted formats that are expensive to change later (fleet message schemas, stored data), and anything on a safety path.
5. Give one real example with numbers, such as a slow CI pipeline you sped up and what that unblocked.

Follow-up

• Describe a time you deliberately took on debt. Did it pay off?
• How did you convince a manager or team to spend time on paydown?
• How did you measure whether the paydown was worth it?

How do you validate your software before deploying it to physical hardware?

Medium · Problem-Solving & Engineering Mindset

Approach

1. Describe layers, each catching a different class of bug. Unit tests run against a hardware abstraction layer with mocked drivers. Property-based tests check the maths (kinematics, transforms, filters). Static analysis and sanitizers run in CI.
2. Simulation and log replay come next: physics simulation for behaviors, and deterministic replay of recorded sensor logs so a regression shows up as a changed output. Fix seeds and control time so replays are repeatable.
3. Hardware-in-the-loop: run real controllers or compute boards against a simulated plant, then bench tests with actuators unloaded or limited before full motion.
4. Staged deployment with guardrails: dedicated test units first, with velocity or torque limits, watchdogs and an emergency stop active, then canary robots, then the fleet. For firmware, use A/B partitions and automatic fallback when a health check fails.
5. Be honest about limits. Simulation misses contact dynamics, sensor noise and timing on real hardware, which is why the hardware stages exist.

Follow-up

• What classes of bugs does simulation fail to catch?
• How do you make log-replay tests deterministic?
• An update breaks the network stack on a robot. How do you recover it?

Explain a complex architectural decision you made and the long-term impact it had on the project.

Medium · Problem-Solving & Engineering Mindset

Approach

1. Choose one decision you owned with lasting consequences. State the constraint that forced it (latency, scale, team size, hardware limits) in one or two sentences.
2. Lay out at least two real alternatives and the criteria you used to compare them: performance, operability, cost, reversibility, and the team's ability to maintain each.
3. State the trade-off you accepted explicitly, for example 'I chose a single-writer service and accepted a throughput ceiling in exchange for simple ordering guarantees.'
4. Give long-term evidence: metrics from months after launch, what the decision made easy, what it made harder, and any migration it eventually required.
5. Close with hindsight: what you would keep, what you would change, and whether it was a reversible decision. Answers fall flat when they describe the system without the decision, or claim the decision had no downside.

Follow-up

• Which alternative came closest, and what would have had to be true for it to win?
• What got harder a year later because of this decision?
• How would the decision change at ten times the scale?

Describe a situation where you had to debug a race condition or memory leak in a production system.

Medium · Technical & Systems Expertise

Approach

1. Use STAR, but spend most of the time on the investigation. Give the symptom precisely (an intermittent crash, a corrupted value, or RSS growing by a fixed amount per hour), the impact, and why it was hard to reproduce.
2. Show a real toolchain. For a race: ThreadSanitizer, stress runs with many threads or randomized delays to widen the window, logs carrying thread ids and monotonic timestamps, and core dumps inspected in gdb. For a leak: heap growth over time, Valgrind/Massif, LeakSanitizer, heaptrack or tracemalloc, and a diff of two heap snapshots to find what accumulates.
3. Name the root cause as a mechanism, for example a check-then-act on a shared map without a lock, a callback capturing a shared_ptr that forms a reference cycle, or an unbounded cache keyed by request id. 'It was a threading issue' is not a root cause.
4. Explain the fix and how you proved it: narrowed lock scope, an atomic, a weak_ptr, or an ownership change. Add a regression test that failed before the fix, plus a metric or alert. Note that a race 'fixed' by changing timing has not been fixed.
5. Close with what changed afterwards: sanitizers in CI, a review checklist item, or a design change that removed the shared state.

Follow-up

• How did you know the fix worked rather than just shifting the timing?
• Why didn't existing tests catch it, and what did you add?
• What would you do differently if it happened on a robot in the field instead of a server?

Walk us through your process for troubleshooting a critical failure in a deployed system.

Medium · Problem-Solving & Engineering Mindset

Approach

1. Contain first. For a robot fleet that means putting affected units in a safe state, pausing any rollout in progress, and rolling back if a recent change is the likely trigger. Then size the blast radius: one unit, one hardware revision, or the whole fleet.
2. Build a timeline from facts: what changed and when (software release, configuration, firmware, hardware swap, environment), plus logs, metrics, core dumps and recorded sensor data from the moment of failure. Check the instrumentation is trustworthy before trusting the symptom.
3. Narrow it down by comparison and bisection. Diff affected units against healthy ones by software version, hardware revision and configuration, reproduce on a bench or in simulation with replayed logs, and bisect releases. Rule out sensor or actuator faults separately from software faults.
4. Separate the trigger from the root cause, ship the fix with a test that reproduces the failure, and verify on canary units before a wider rollout.
5. Finish with a blameless postmortem whose action items close the detection gap (an alert, a test or a rollout gate), not just the bug.

Follow-up

• How do you decide between rolling back and fixing forward?
• What do you do when you cannot reproduce the failure?
• What would you add so this class of failure is caught before it reaches a robot?

Suggested practice checklist

Each day targets one reported round or cluster of reported questions, moving from language and OS fundamentals through coding, design and debugging to the collaborative and behavioral round. Every task produces something you can review on the final day.

Day 1 — Map the loop and lock in your language and Linux basics

• Write the four reported rounds (Technical Screening, Systems Design, Coding Proficiency, Team-Based Problem Solving) on one page and mark the one you are least ready for.
• Choose your primary language among C++, Python and Rust, and write a one-page cheat sheet of its concurrency primitives, memory model and common pitfalls.
• Explain out loud, and then write down, Linux processes vs threads, scheduling policies, virtual memory and page faults, and blocking vs non-blocking I/O.

Deliverable: A round map with the weakest round marked, a language cheat sheet, and written Linux fundamentals notes.

Day 2 — Concurrency on real-time systems and the priority task queue

• Write a full answer to the real-time concurrency-models question, covering priority inversion, its mitigations, message passing, lock-free trade-offs and rate-monotonic scheduling.
• Implement a bounded, thread-safe priority task queue with FIFO order for equal priorities and a close() method, plus a multithreaded test that checks no task is lost or duplicated.
• Solve the overlapping-job-attempts and peak-concurrency practice problem with a sweep over sorted start and end events.

Deliverable: A written concurrency-models answer, a tested priority task queue, and a timed solution to the peak-concurrency problem.

Day 3 — Performance on constrained hardware and the refactor exercise

• Write your answer to the constrained-hardware performance question around one real optimization from your work, including before and after numbers.
• Take a slow, messy function from your own code, write characterization tests, then refactor it for allocation, copies and readability while keeping the tests green.
• Solve the heaviest-tenants-under-memory-pressure practice problem, and state its memory bound and what you would do if the key space does not fit in memory.

Deliverable: A performance answer with measurements, a refactored function with its tests, and a timed solution to the heaviest-tenants problem.

Day 4 — Robot communication and sensor ingestion design

• Design the distributed robotic communication protocol: message classes, envelope fields, QoS per class, heartbeat and safe-state behavior, and schema evolution.
• Design sensor data ingestion from many robotic units, from on-robot buffering to idempotent upload, replicated log and object storage, and sketch it from memory.
• Run both designs aloud against a timer and list the three failure modes you covered least well.

Deliverable: Two sketched designs (protocol and ingestion) and a list of failure modes to tighten.

Day 5 — Cloud reliability, identity, containers and pre-hardware validation

• Write your answer to the cloud high-availability and data-integrity question, including idempotency keys, multi-zone replication, restore testing and per-robot identity with credential rotation.
• Write your answer to the pre-hardware validation question as layers: unit tests behind a hardware abstraction, log replay, simulation, hardware-in-the-loop, then staged rollout.
• Design a firmware CI/CD pipeline with simulation and hardware-in-the-loop gates, canary units, A/B partitions and automatic rollback.
• Containerize the ingestion gateway from day 4 (Docker/Kubernetes are listed as nice-to-haves): write its liveness and readiness probes, resource limits and a rolling-update strategy with the signal that triggers rollback.

Deliverable: Written answers on cloud reliability and validation, a sketched firmware CI/CD pipeline, and a container spec for the ingestion gateway with probes and rollout strategy.

Day 6 — Debugging and incident stories

• Write your race-condition or memory-leak story with the symptom, tools, mechanism-level root cause, the fix and the regression test that proves it.
• Write your process for a failure in a deployed system: contain, blast radius, timeline, comparison and bisection, root cause vs trigger, postmortem actions.
• Say both answers aloud and cut any sentence that does not name a tool, a measurement or a decision.

Deliverable: Two rehearsed debugging answers, each with a named root cause and a verification step.

Day 7 — Team-based problem solving and behavioral stories

• Write STAR stories for the pivot question, the architectural-decision question and the technical-debt question, each with the cost and a measured outcome.
• Write the argued-and-lost and reversed-decision practice stories, naming the mechanism you predicted and what the reversal cost.
• Run a paired mock with a friend on an open robotics design problem, practising check-ins at each decision point, then write two questions to ask your interviewers about their team's technical problems.

Deliverable: Five written behavioral stories, notes from a paired mock, and two questions for your interviewers.

Behavioural preparation

Candidates describe the final round as collaborative problem solving that looks at teamwork and communication, and several reported questions ask about past work: changing course, debugging, technical debt and architectural decisions. Prepare stories from projects where software met hardware, and make each one specific about what you decided, what it cost and how you measured the result.

1. Describe a situation where you had to debug a race condition or memory leak in a production system.
2. Explain a complex architectural decision you made and the long-term impact it had on the project.
3. How do you approach technical debt while maintaining rapid delivery cycles?
4. Tell me about a project where your software choices had physical or hardware consequences, and how you worked with mechanical or electrical engineers to resolve them.
5. Describe a design you argued against and lost: the failure you predicted, the evidence you brought, and what you did after the decision was made.

Describe a time you had to pivot your technical approach based on new data or changing requirements.

Medium · Problem-Solving & Engineering Mindset

Approach

1. Pick a story where you made the call to change course, not one where the change was imposed on you and you complied.
2. Show that the original plan was reasonable: what you knew, what you chose, and why.
3. Name the new data precisely (a benchmark result, a field failure, a hardware limitation discovered in testing, or a changed requirement), and say how quickly you recognised it and what you checked before acting on it.
4. Cover the switch itself: what you kept from the first approach, what the change cost in time or rework, and how you told stakeholders and adjusted the plan.
5. End with the outcome and a lesson tied to an earlier signal you could have caught. The answer falls apart if it blames the requirements or if you cannot say what the pivot cost.

Follow-up

• What signal could have told you earlier that the first approach would not work?
• How did you communicate the change to people depending on your original plan?
• What did you salvage from the first approach?

Argue against a design, lose, and commit anyway

Medium · disagreement · service boundaries · decision records

Describe a design you argued against and lost. State the failure you predicted as a named mechanism, not a feeling about complexity: two services that would need one transaction, a projection with no rebuild path, a write path with no idempotency key. Say what evidence you brought, what the decision maker weighed instead, and what you did after the decision was made: what you instrumented, what you wrote down, and whether the prediction came true. Five minutes.

Approach

1. State the prediction in falsifiable form up front: the mechanism, the condition that triggers it, and the observable outcome. A prediction that cannot be checked also cannot be credited to you later.
2. Show the evidence you had at the time and label each piece honestly as measured, analogous, or intuition. Keeping the intuition is fine; disguising it as data is the thing that erodes your standing in the next argument.
3. Represent the opposing case at full strength, including the constraint you did not control: a fixed date, a team boundary, or the fact that the decision was cheap to reverse and yours was not.
4. Make disagree-and-commit concrete. Name the artefact you left behind so the prediction could be settled without you: the alert and its threshold, the counter on the dashboard, the decision note that recorded the trade-off and the condition that would revisit it.

Follow-up

• What threshold on that alert would have proved you right, and did anyone ever look at it?
• If the same proposal arrived tomorrow with the same deadline, would you argue it the same way?

Reverse your own decision and price the reversal

Medium · reversibility · measurement · migrations

Describe a technical decision you made and later reversed. Pick one that cost something: a service you split and merged back, a cache you added and removed, an index you created that pushed the planner onto a worse plan, a projection you rebuilt from scratch. State what you believed when you decided, the measurement that changed your mind, how long the wrong version ran in production, and what the reversal cost in migrations, dual writes, and a deprecation window for callers you did not own.

Approach

1. State the original rationale without irony, in the version you would still defend given what was known then. If it is not defensible, the story is about carelessness rather than judgement, and a different example serves you better.
2. Give the measurement that moved with a before and after: the p99 that did not improve, the cache hit rate that sat at 40%, the plan that flipped to a sequential scan once the table passed a size you can name.
3. Cost the reversal in steps, not adjectives: expand-and-contract deploys, the dual-write window, the callers who had to be notified, the rows already written in the wrong shape that had to be backfilled or abandoned.
4. Distinguish reversal from rewrite by naming what you kept. Most good reversals preserve the schema or the interface and undo one decision inside it, which is also why they were affordable.

Follow-up

• What in that decision was irreversible, and did you know it was irreversible when you made it?
• How did you tell the people who had already built on top of the original decision?