Together Ai Software Engineer Interview Guide

Interviewing at Together Ai

• Practice bank: 8+ questions
• Rounds: 1
• Typical prep: 1–2 weeks

Interview rounds

Round	Questions
Technical Screen	8

How hard is the Together Ai Software Engineer interview?

Difficulty	Share	Questions
Easy	63%	5
Medium	25%	2
Hard	13%	1

What Together Ai actually tests for

Topic	Share	Questions
Software Engineering Fundamentals	50%	4
Coding & Algorithms	38%	3
System Design	13%	1

The 60-second brief

Together AI's Software Engineer role works on the AI Acceleration Cloud, infrastructure that virtualizes machine learning hardware, including NVIDIA GB200/GB300 GPUs and BlueField DPUs, for LLM inference and training. Customers use it to provision on-demand compute, managed Kubernetes clusters and Slurm workloads, and the platform serves internal products as well as external enterprise customers. The engineering spans physical hardware, networking such as Infiniband, and highly available cloud services written mainly in Go.

Together Ai candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

What a Software Engineer does at Together Ai

The role is described as sitting on the Together Cloud Platform or Together Cloud Infrastructure team. The listed work covers backend services and API microservices behind customer-facing products, written primarily in Go; infrastructure as code for physical and virtual resources; design and code reviews, developer documentation and testing strategy; and an on-call rotation for infrastructure incidents. The systems involve Infiniband networking, exabyte-scale data pipelines and distributed scheduling.

The listed must-haves are 5+ years building fault-tolerant distributed systems and API microservices, strong proficiency in a backend language with Go preferred, systems knowledge across compute, networking and storage (concurrency, memory management, performant I/O), hands-on PostgreSQL, and Kubernetes, containers and CI/CD. Nice-to-haves include QEMU/KVM and KubeVirt, SR-IOV, GPU virtualization, Infiniband or RDMA, DPUs and SmartNICs, CUDA or NCCL, and Kafka, Airflow or Kinesis.

The reported questions track that list closely: distributed scheduling and storage design, a Kubernetes operator in Go, GPU virtualization trade-offs, data-center networking, Terraform and Ansible delivery, Prometheus observability, and PostgreSQL and Linux performance debugging. Be ready to explain how each tool works underneath, not only when you would choose it. The company's open-source work includes FlashAttention and RedPajama, so a quick read on what each one is makes useful background.

The Together Ai interview process

Together Ai candidates report 3 rounds · ≈ 3-5 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

Candidates describe the first stage as a recruiter conversation to align on background, career goals and compensation expectations. Use it to shape the rest of the process. Connect your experience to the areas the role lists: distributed systems and API microservices, Go, compute, networking and storage fundamentals, PostgreSQL and Kubernetes. Then find out which of the reported technical focuses your phone screen will use: systems programming, concurrent coding or architecture design. The role is described as sitting on either the Together Cloud Platform or the Together Cloud Infrastructure team, so ask which one you are being considered for. The answer tells you whether to weight service design or hardware-level infrastructure.

What it evaluates

• Whether your background lines up with the listed must-haves: fault-tolerant distributed systems, a backend language (Go preferred), systems knowledge across compute, networking and storage, PostgreSQL and Kubernetes
• Whether your career goals fit infrastructure work that sits between physical hardware and cloud software
• Whether your compensation expectations fit the role, a topic candidates report comes up at this stage

How to prepare

• Write two short project summaries that map to the listed requirements: one distributed system you built or ran in production, and one piece of infrastructure or low-level work such as virtualization, networking, storage or a Kubernetes controller
• Settle a compensation range before the call so you can answer with a number
• Ask which team the process is for, which languages the coding screens accept, and whether the phone screen is coding, systems programming or architecture design

Technical Phone Screens

Candidates report one or more technical phone screens focused on systems programming, concurrent coding or high-level architecture design. The format can be any of the three, so prepare all three to screen depth rather than betting on one. For code, reported guidance stresses clean, idiomatic Go with attention to concurrency and tests. A goroutine-based solution that shuts down cleanly and passes the race detector covers much of that. For systems programming, explain what the runtime and kernel actually do, not only which API you would call. For architecture, a compact design with a clear failure story is stronger than a broad one with no failure handling.

What it evaluates

• Whether you can write correct concurrent code: bounded goroutines, channels or mutexes chosen for a stated reason, cancellation through context, and no leaked goroutines
• Whether you can explain the systems behaviour underneath the code, such as garbage-collection cost, buffer reuse and the file I/O path
• Whether a design answer states its failure modes and delivery semantics, not only its components

How to prepare

• Implement a bounded worker pool in Go with context cancellation and error propagation, then run it under go test -race
• Practise the coding titles in the question bank: detecting and breaking cycles in pod dependencies, scheduling GPU pods and draining a node, splitting a chunked text stream into line-balanced parts, and the first unique character index
• Prepare spoken answers on GC versus non-GC memory management and on file I/O bottlenecks, including zero-copy and asynchronous system calls
• Rehearse one scheduling design in compact form that ends with what happens during a network partition

Virtual or On-site Interview Loop

Candidates report a virtual or on-site loop covering distributed systems, networking, infrastructure automation and behavioural alignment. Reported guidance says design discussion goes down to the hardware and network layers: how memory is managed, how packets are routed, how database transactions are isolated. The loop spans several areas, so prepare each one separately: a distributed scheduling or storage design, a networking and virtualization comparison, an infrastructure-as-code or observability pipeline, and behavioural stories about outages and disagreements. Keep the facts about any project you mention identical in every conversation.

What it evaluates

• Whether a distributed design handles node failure, network splits and hardware degradation, with its delivery and consistency guarantees stated
• Whether you understand networking and virtualization choices such as VLAN, VXLAN and VPC isolation, Infiniband and RDMA, and hardware passthrough
• Whether infrastructure automation is safe to run against production: reviewed plans, staged rollout, drift handling and guarded node lifecycle actions
• Whether behavioural stories show ownership of incidents and a clear, evidence-based way of resolving technical disagreement

How to prepare

• Work the reported design questions in this guide (at-least-once task scheduling under partitions, a global management plane, on-demand GPU scheduling, multi-exabyte object storage), taking each one layer below the box diagram
• Write one-paragraph comparisons of VLAN vs VXLAN vs VPC and of VFIO vs SR-IOV vs PCIe passthrough, and practise defending them aloud
• Sketch a Terraform and Ansible delivery pipeline and a Prometheus/Grafana node-health pipeline that cordons and drains nodes, naming the safety check in each
• Prepare outage, disagreement and shifting-requirements design-doc stories with a fact sheet per project so figures stay the same in every round

Questions Together Ai candidates report

System design and architecture

Turn the prompt into a workload before drawing anything: requests per second at peak, the read to write ratio, the size of one stored object, how long it is kept, and how each of those grows. Those numbers decide the architecture, and without them every component is a guess. Size the simplest thing that fits, then add a tier only where a number says the simple version breaks. What separates a strong answer is that every box has a quantity attached to it. A cache, a queue and a partition scheme that each answer a specific number read as judgement; the same diagram without numbers reads as recall.

• How would you design a distributed, highly available task scheduling system that guarantees at-least-once execution under heavy network partitions?
• How would you set up and automate Infiniband partitioning and parallel storage provisioning in a bare-metal data center?
• Explain how you would architect a global management plane to control compute, networking, and storage across dozens of independent data centers.
• Design an on-demand GPU scheduling system (similar to Instant Clusters) that dynamically allocates physical hardware to incoming containerized workloads while minimizing fragmentation.

Languages, concurrency and fundamentals

Many questions in this area arrive as a symptom: code that passes every test and fails weekly in production, memory that climbs until a restart, a handle that runs out. Treat the symptom as a reachability or a scheduling question. For a leak in a collected runtime, ask what is still reachable, because collectors free unreachable objects and nothing else, so a cache or a listener list that is never pruned is a leak by design. For intermittent wrongness, ask which schedule produces it. The separator is producing that schedule or that retained path concretely rather than restating the symptom as a suspicion.

• How does memory management differ when writing high-throughput network applications in a garbage-collected language like Golang versus a non-garbage-collected language?
• Describe the performance differences between VFIO, SR-IOV, and standard PCIe passthrough when virtualizing hardware accelerators like GPUs.

Behavioural and engineering judgement

Most engineering disagreements worth asking about are not settled by evidence, because both positions are defensible on what is known at the time: a schema shape, an abstraction one of you thinks is premature, fixing versus rewriting. The useful move is to say what would settle it, and what you do when nothing will. A common tiebreaker is which option is cheaper to undo, so you take the reversible one and set a date to look again. Weak answers argue who turned out to be right. Strong ones explain how the call got made while it was still undecidable.

• Tell me about a situation where you had a strong technical disagreement with a peer or lead. How did you present your arguments, and how was the conflict resolved?
• Describe a time you had to resolve a high-severity production outage under intense time constraints. How did you identify the root cause, and what steps did you take to prevent recurrence?

Practice exercises with worked solutions

The prompts above tell you what gets asked.

Decide which facts an invoice line copies instead of joining

invoice_line_item already denormalises tenant_id, which is reachable through invoice_id, and stores amount_minor even though quantity times unit_price_micros would recompute it. A reviewer asks you to normalise both away, and separately asks whether the tenant's legal name and billing address should be copied onto the invoice header. Decide each case. For every field you keep denormalised, name the read pattern or the invariant that justifies it, the anomaly the copy can develop, and the mechanism that prevents that anomaly here.

How a strong candidate works it

• Split the question into two kinds of copy, because they fail differently. A copy of a currently mutable fact is a cache: it drifts and needs invalidation. A copy of a fact frozen at write time is not a cache at all, it is the record of what happened, and normalising it away destroys information the source no longer holds.
• Keep tenant_id on the line. It costs 8 bytes, it leads every index on the table so no read is ever accidentally cross-tenant, and it turns a wrong join into an empty result rather than another tenant's money. Prevent the drift structurally: a unique constraint on invoice (invoice_id, tenant_id) plus a composite foreign key from the line on (invoice_id, tenant_id) makes a mismatched pair impossible, so the database enforces agreement instead of a code review.
• Keep amount_minor. Rounding must happen exactly once, at a named site, with a stated mode (half-even here). If readers recompute from quantity and unit_price_micros, every reader owns a rounding decision, and half-up and half-even diverge systematically across thousands of lines rather than cancelling out. A check constraint can bound the stored value but deliberately cannot re-derive it.
• Copy the legal name and billing address onto the invoice header, written once and never updated. The statement must show what was true when it was sealed, and the tenant record will change afterwards. This is a snapshot for the same reason source_rollup_watermark is stored per line: without it, nobody can reconstruct what the customer was told.
• Name the read pattern that pays for all of it. Rendering, dispute response and export are per-tenant, per-period reads over thousands of lines that would otherwise join back to slowly changing dimensions that no longer hold the historical value. The write side is a once-per-period batch, so the extra columns cost nothing that matters.
• Concede the case where the reviewer is right: a mutable operational attribute such as the tenant's current plan name has no business on a line. If a report wants it, join. If a statement needs the plan as of the period, that is another snapshot and it belongs on the header with the rest.

Expected result (25 min): tenant_id kept and enforced by a composite foreign key to (invoice_id, tenant_id); amount_minor kept because rounding happens once with a stated mode; legal name and billing address snapshotted onto the invoice header and never updated; mutable operational attributes left normalised. Each decision paired with its read pattern, its anomaly, and the mechanism that prevents the anomaly.

Check yourself

• A line cannot be attached to an invoice belonging to another tenant, and the rejection comes from the database rather than the application.
• Re-rendering a sealed invoice after editing the tenant record produces identical output.
• Rounding per line and rounding once at the end differ measurably over a thousand lines, which is the argument for storing the value.
• No column on the line is recomputed by any reader.

Follow-ups

• Write the composite foreign key and the unique constraint it requires on the parent. What does it cost on every line insert, and what does it do to a bulk load?
• A tenant is renamed after being invoiced. Which rows change, and what does the customer see on last quarter's PDF?
• Where does currency live, and what breaks if a tenant's billing currency changes between two periods?

Find the join that inflates every invoice total

invoice_line_item holds line_id, invoice_id, tenant_id, sku, rate_tier, quantity, unit_price_micros, amount_minor (bigint), currency, kind, voided_at. invoice_payment_attempt holds attempt_id, invoice_id, tenant_id, amount_minor, status (succeeded, failed, pending), created_at, and an invoice has many attempts. A finance report runs select i.invoice_id, sum(l.amount_minor), count(p.attempt_id) from invoice i join invoice_line_item l using (invoice_id) join invoice_payment_attempt p using (invoice_id) group by 1 and the totals are wrong. Say precisely what the sum now equals, and write a version that is also correct for invoices with zero attempts.

How a strong candidate works it

• Compute what the query actually returns before fixing it. The two joins form a Cartesian product per invoice, so each line row repeats once per attempt row: sum(l.amount_minor) is the true total multiplied by the attempt count, and count(p.attempt_id) is attempts times lines. Three lines and two attempts report double the money and six attempts.
• Reject the reflex repair. count(distinct p.attempt_id) does fix the count, because attempt_id is unique. sum(distinct l.amount_minor) does not fix the sum, because two legitimate lines with equal amounts collapse into one. DISTINCT inside an aggregate deduplicates values, not rows, and the difference stays invisible until two lines happen to match.
• Aggregate each branch to invoice grain before joining: one CTE summing lines by invoice_id, one counting attempts by invoice_id, then join the two results. A LATERAL subquery per invoice is equivalent and sometimes plans better when the outer set is small. Either way every aggregate stays at the grain it was defined at.
• Keep invoices with no attempts by making the attempt branch a LEFT JOIN with coalesce(attempt_count, 0). An inner join here silently drops every unpaid invoice, which is usually the exact population finance is asking about.
• Push each filter to its own grain: where l.voided_at is null belongs inside the line CTE, not the outer query, or it would also filter the attempt branch through the join. Put the tenant predicate on both branches, since the denormalised tenant_id is what stops a wrong join crossing tenants.
• Leave yourself a standing check: an invoice total is a function of its non-voided lines and of nothing about payments, so if changing the payment filter moves the money figure, the fan-out is back.

Follow-ups

• Add a third branch for credit notes applied to the invoice. Does the CTE shape still hold, and when would a single pass with filter (where ...) be better?
• Over 500k invoices this report takes minutes. Which grain would you materialise, and how do you keep it correct when a line is voided?
• The same report is needed per tenant per month. What index makes the line CTE cheap?

Seal an hour under late data with bounded memory

Metering ingest reads 256 partitions at 10,000 to 40,000 events/second. Events carry occurred_at and ingested_at, and during a producer replay the gap between them is hours. Seal each UTC hour once no more than 50 parts per million of that hour's eventual quantity can still arrive, using memory that does not grow with the size of the replay. Define the watermark, the lateness parameter and how you measure it, the structure holding open hours, and the write that performs the seal. State what an idle partition does to your watermark.

How a strong candidate works it

• Two clocks, two jobs. Bucket by occurred_at, because that is the hour the customer is billed for, and advance the watermark on ingested_at, because that is what the fold has consumed and what source_max_ingested_at records. Conflating them is what makes late data invisible.
• The global watermark is the min over partitions of each partition's committed ingested_at, not the max: the fold is trustworthy only as far as the slowest partition. The consequence is that one idle partition pins the watermark forever and nothing seals, so an idle partition must promote its watermark to wall clock after a stated idle timeout, and that timeout becomes a correctness parameter, because a partition that is slow rather than idle gets sealed past.
• Choose the lateness L from the measured distribution of ingested_at - occurred_at, weighted by quantity rather than by event count. The target is 50 ppm of the hour's quantity, and a replay is rare in events while carrying disproportionate mass, so an event-weighted quantile picks an L that is comfortably wrong at exactly the moment it matters.
• Measure that quantile in bounded memory. A Greenwald-Khanna summary gives epsilon-approximate quantiles in O((1/epsilon) log(epsilon n)) space; a t-digest costs more per merge but has relative error that tightens at the tails, which is the half of the distribution you are reading at p99.99. Keep a separate summary per tenant class, because one tenant's batch importer is not the population.
• Hold open hours in a min-heap keyed by hour_start. When the watermark advances, pop every hour with hour_end + L < W and seal it: O(log H_open) per advance and O(1) amortised per event to touch its bucket. Memory is open hours multiplied by distinct (tenant, workspace, sku) keys, so cap the number of simultaneously open hours and spill the oldest into usage_rollup_hourly as status='open' with a revision bump. While an hour is open the row is upsertable, so the store is your overflow.
• The seal itself is a conditional write: update ... set status='sealed', sealed_at=now() where status='open' returning .... Two sealers race on every restart, and the loser must see zero rows and stop rather than write a second value. After the seal, an event for that hour is not an upsert but an adjustment, and source_max_ingested_at is what proves it arrived afterwards.

Expected result (40 min): The quantity-weighted p99.99 is hours larger than the event-weighted one. Choosing L from the event-weighted number lets roughly the tail's 3% of quantity land after the seal, 600 times the 50 ppm target. With the min watermark and no idle timeout, the stalled partition blocks all sealing; with a 60-second timeout, the stall is sealed past and its events arrive late.

Check yourself

• Post-seal quantity as a fraction of the hour's total is at or below 50 ppm at the chosen L, measured rather than assumed.
• The second sealer's update reports zero rows affected and writes nothing.
• Heap size equals the number of open hours and does not grow with the replay's event count.

Follow-ups

• A replay starts during the sealing window for a period you are about to close. What do you do, and what is the customer-visible consequence of each option?
• Your measured quantity-weighted p99.99 lateness is six hours and the invoice must be issued at 02:00 UTC on the first. How do you reconcile those two numbers?
• How would you detect that L has drifted before it costs you an hour's quantity?

Write the delivery guarantee and replay API for outbound events

The webhook-delivery service fans events to customer endpoints, tracked in webhook_delivery (subscription_id, tenant_id, event_id, redelivery_seq, status, attempt_count, next_attempt_at, lease_token, last_response_code, payload_digest). Customers are asking for exactly-once and in-order delivery. Write the contract you can actually honour: the guarantee, the headers that make it usable, which customer response codes are retryable, the retry schedule and terminal condition, what happens to an endpoint that has been down for a day, and the shape of the redelivery endpoint. State plainly what you are not promising and what the customer must do instead.

How a strong candidate works it

• Refuse exactly-once with the mechanism rather than with policy: the customer's acknowledgement can be lost after they have already committed, so the sender cannot distinguish unprocessed from processed-but-unacknowledged and must retry. The deliverable is at-least-once with a stable event identifier and the customer documented as the deduplicating party.
• Price ordering instead of promising it. With parallel attempts per subscription, a retried event overtakes a newer one, so in-order delivery requires a single in-flight attempt per subscription, which turns any slow endpoint into head-of-line blocking for that subscription's whole queue. Offer it per subscription with that cost written down.
• Define the response contract from the customer's side: 2xx is accepted, 408, 429 and 5xx are retryable, 410 disables the subscription, and every other 4xx is permanent and terminal. Publish the read timeout and tell the customer to acknowledge first and process asynchronously, since their processing time otherwise consumes your worker occupancy.
• Publish the schedule and its end: capped exponential backoff with full jitter, sleeping uniformly in [0, min(cap, base * 2^attempt)] so a mass failure does not re-synchronise the herd, terminal after a stated attempt count or age, plus a per-endpoint circuit breaker that records further deliveries as dropped_circuit_open and notifies the owner rather than burning shared worker capacity.
• Shape redelivery as an insert, not a reset: POST to a redeliveries collection with event ids or a time window creates rows at redelivery_seq + 1 carrying the same bytes, which payload_digest lets you prove, leaving the original terminal rows intact as the record of what happened.
• Compare the event's tenant against the subscription's tenant at enqueue and again immediately before signing, because cross-tenant delivery originates in an enqueue path that took the subscription from one lookup and the payload from another, not in the worker.

Expected result (30 min): At-least-once with a stable event id, ordering offered only as an opt-in single-in-flight mode with head-of-line blocking stated, a four-class response table, a jittered capped schedule with a terminal state, and redelivery implemented as new rows at an incremented sequence carrying identical bytes.

Check yourself

• The customer can deduplicate using headers alone, without parsing the payload.
• Nothing in the contract requires the sender to know whether the customer committed.
• A day-long outage has a defined end state rather than unbounded retries.
• Redelivery leaves the original delivery rows readable as history.

Follow-ups

• A customer wants everything they missed during their six-hour outage. Do the dropped_circuit_open rows let you answer that, and what retention bound does the answer depend on?
• You offer the ordered mode and one customer's endpoint slows to two seconds per request. What do their delivery metrics look like, and what do you owe them in the docs?

One tenant's counter writes stall the whole connection pool

A change that made a per-tenant usage counter correct now produces site-wide latency whenever one large tenant writes: unrelated endpoints time out waiting for a connection while database CPU stays low and no statement is slow. The change wraps the counter update in a transaction that takes SELECT ... FOR UPDATE on one row, calls an external pricing service, then updates and commits. Give an ordered checklist, the arithmetic that bounds that tenant's write rate, and three repairs with the cost each one accepts.

How a strong candidate works it

• Separate waiting from working. Low database CPU alongside high application latency points at a queue, so instrument connection-acquisition wait separately from query execution time; that queue forms in the application and is invisible in database metrics, which is why the database looks healthy throughout.
• Confirm the lock rather than assuming it: sample waiting sessions and group by wait event, relation and tuple. Contention concentrated on one tuple belonging to one tenant is the signature; a deadlock would instead show the database aborting transactions after its detection timeout, which is not happening here.
• Do the arithmetic out loud. Throughput on a serialised row is one divided by the lock hold time, and the hold spans the external call, so a 20 ms pricing call caps that tenant near 50 writes per second no matter how many pods run. Every waiter also holds a pooled connection while it queues, so the shared pool drains and unrelated tenants fail at acquisition.
• Repair one: shrink the critical section to a single statement with the price resolved before the transaction opens. Cost is a stale price for the duration of one request and a second round trip; benefit is a hold time measured in the database's own execution time.
• Repairs two and three change where the contention lives rather than how long it is held. Sharding the counter into per-(tenant, bucket) rows and summing on read multiplies write throughput by the shard count, at the cost of an aggregate on every read and a shard count you must size against the largest tenant rather than the median. Accumulating in memory and flushing periodically removes the per-write round trip entirely, paid for with a bounded loss window on crash, which is acceptable for a rate limiter and not for a billing counter.
• Contain independently of which repair wins: a separate pool or per-tenant concurrency cap for this write class, a statement timeout low enough that a pathological query dies before it accumulates waiters, and an idle-in-transaction timeout so a stuck client cannot pin a connection and its locks.

Follow-ups

• What would a genuine deadlock look like here, which two code paths would produce one, and how does the database's response differ from what you observed?
• If a transaction-pooling proxy sits in front of the database, which of your three repairs changes behaviour, and what stops working that would have worked on a direct connection?
• The counter also enforces a quota. Why is SELECT the count and then INSERT still wrong after you have fixed the contention?

Reverse a webhook ordering decision after measuring its cost

You argued for strict per-subscription ordering in webhook-delivery, which means one in-flight attempt per subscription. It shipped. Three months later a single unresponsive endpoint holds one subscription's queue at a six-hour backlog, and two customers report events arriving out of order anyway once their own retries are counted. Describe a decision you reversed: what you originally optimised for, the measurement that changed your mind, what the reversal cost in engineering time and customer change, and how you told the people who had already built on the original guarantee.

How a strong candidate works it

• State the original decision as a trade you made knowingly. Ordering across a network requires a single in-flight attempt per subscription, and its price is head-of-line blocking whenever one endpoint is slow. 'We priced it wrong' is a much stronger opening than 'we did not realise', and it is usually the true one.
• Bring the measurement that flipped it, not the anecdote: backlog age at the ninety-ninth percentile per subscription, the share of subscriptions where one slow endpoint gated an otherwise healthy queue, and the delivery throughput lost to serialisation. A reversal justified by complaints is indistinguishable from a reversal justified by fatigue.
• Name what you learned about the guarantee itself, which is the engineering content of this story. At-least-once delivery means a retried event already arrives after newer ones and the consumer already must be idempotent, so a guarantee the customer has to defend against anyway was never worth what it cost to provide.
• Describe the migration, because reversing a published contract is the hard half and the part candidates skip. Parallel attempts behind a per-subscription flag, a monotonically increasing sequence number added to the envelope so order-sensitive consumers can sort or discard, documentation that states at-least-once and unordered in those words, and a deprecation measured in quarters because the client is a pinned SDK inside a build pipeline you cannot see or redeploy.
• Give the cost in the two currencies that matter: engineer-weeks, and how many customers had to change code. Then say who you told before it shipped rather than in a changelog afterwards, and which large customer you left on the old behaviour and for how long.
• Close with the signal you now weight differently, stated as something you would do earlier next time: measuring the blocking cost on the slowest decile of endpoints before committing to the guarantee, rather than after a customer noticed.

Follow-ups

• A customer insists they need ordering. What do you offer them that is not global serialisation?
• How did you choose the deprecation window given that you cannot see or redeploy the clients?
• What would have to be true for you to reverse back?

Where candidates lose points

Claiming exactly-once execution in the at-least-once task scheduler design

At-least-once means a task can run twice. A worker can finish and lose its acknowledgement, or it can be partitioned away while its lease expires and another worker picks the task up. Say that directly, then make it safe: leases with expiry, a fencing token checked on every commit so a returning stale worker is rejected, and task execution that is idempotent on the task id. Walk through a partition and its healing before the interviewer asks for it.

Treating on-demand GPU scheduling as generic bin-packing

Counting free GPUs per node misses what makes the problem hard. Multi-GPU jobs need all their devices at once (gang allocation), placement should respect interconnect and network topology, and scattering small jobs strands capacity that no large job can use. Define a fragmentation metric, say when you preempt or migrate work, and explain how you drain a node without killing long-running training jobs.

Naming Kubernetes, etcd or a hypervisor without explaining how it works

Reported guidance asks candidates to explain tools under the hood. For the custom operator question, describe the reconcile loop as level-triggered and cover the informer cache and the workqueue. Explain why the workqueue never hands the same key to two workers at once, what raising MaxConcurrentReconciles changes, and why reconcile must be idempotent, since it will run again against a stale cache. Bring the same depth to VFIO, SR-IOV and PCIe passthrough.

Writing Go concurrency that passes once but leaks goroutines or races

Unbounded goroutine spawning, channels nobody closes, sends that block forever after the receiver has returned, and shared maps without a lock can all pass a single happy-path run. Bound concurrency with a worker count or semaphore, pass a context for cancellation, make it explicit which side closes each channel, and say you would run the tests with -race. Explain these choices aloud, because Go-specific concurrency and performance questions are reported in systems-focused rounds.

Guessing at a cause in a performance-debugging question

For a throughput drop in PostgreSQL, CPU, disk or network, start with a method, not a hypothesis: check utilisation, saturation and errors for each resource, then narrow down. Name the tool at each step: top or pidstat and perf for CPU, iostat -x for disk queueing and latency, ss and interface counters for the network, and pg_stat_activity wait events with pg_blocking_pids() for database lock waits. For each step, also say what reading would rule that cause out.

Breadth then depth, for a generalist facing a mixed loop

Four days sample coding, design, fundamentals and the practical rounds at deliberately shallow depth, which is enough to surface the topics you did not know were in scope. That map, rather than a guess made on day one, decides where the last three days go.

Day 1 — Map the role and prepare the recruiter screen

• List the must-haves and nice-to-haves the role names (distributed systems, Go, compute/networking/storage fundamentals, PostgreSQL, Kubernetes; QEMU/KVM, KubeVirt, SR-IOV, Infiniband, RDMA, DPUs, CUDA/NCCL, Kafka) and mark each one strong, working or absent for you.
• Write two project summaries for the recruiter screen: one distributed system you ran in production and one infrastructure or low-level piece of work, each with the problem, your change and a measured result.
• Settle a compensation range and write the questions you will ask: which team, which languages are accepted, and which focus the phone screen uses.
• Read every reported question in this guide and rate it: can answer now, can answer after a day's work, or unfamiliar.

Deliverable: A skills map against the role's listed requirements, two project summaries, and the reported question list rated by readiness.

Day 2 — Go concurrency for the technical phone screens

• Build a bounded worker pool in Go with context cancellation, error propagation and clean shutdown; run it under go test -race and fix anything the race detector reports.
• Solve the bank coding titles: detect and break cycles in pod dependencies (three-colour DFS or Kahn's algorithm), schedule GPU pods and drain a node, and split a chunked text stream into line-balanced parts. State the complexity aloud for each.
• Warm up with the first unique character index problem: one pass to count characters, a second pass to find the first with count one, O(n) time.
• For each solution, list the unit tests you would add, including an empty input and, where relevant, a concurrent case.

Deliverable: A race-clean Go worker pool, three bank-style solutions with complexity stated, and a test list for each.

Day 3 — Systems programming and performance debugging

• Answer aloud: GC vs non-GC memory management for high-throughput network services (allocation rate, GC CPU and pause cost, buffer reuse with sync.Pool, manual lifetime bugs), and file I/O bottlenecks with zero-copy and asynchronous system calls.
• Write a comparison of VFIO, SR-IOV and standard PCIe passthrough: what each shares, what each isolates, and where overhead appears.
• Write an ordered diagnosis checklist for each bank troubleshooting title: a CPU-bound Linux server, storage I/O saturation, slow network throughput including containers, and a server saturated by logging. Name the tool and the metric at each step.
• Work the PostgreSQL throughput-drop question and this guide's connection-pool lock-contention drill: find blocked sessions through pg_stat_activity wait events and pg_blocking_pids(), and separate waiting time from execution time.

Deliverable: Three written systems explanations and four ordered troubleshooting checklists, each naming tool, metric and next step.

Day 4 — Distributed design: delivery semantics and consensus

• Design the at-least-once task scheduler under network partitions: a durable queue, leases with expiry, fencing tokens so a stale worker cannot commit, and idempotent execution keyed by task id.
• Work the worked exercise 'Write the delivery guarantee and replay API for outbound events' as practice in stating what at-least-once does and does not promise.
• Answer the state synchronisation and consensus question: which state needs consensus (for example etcd for leadership and cluster metadata), which can be eventually consistent, and where regional replicas may serve stale reads.
• End each design with a failure walkthrough: leader lost, partition healed, worker returning after its lease expired.

Deliverable: Two designs, each with its delivery guarantee, consistency choices and a written failure walkthrough.

Day 5 — GPU infrastructure design

• Design on-demand GPU scheduling for containerized workloads: gang allocation for multi-GPU jobs, topology-aware placement, a fragmentation metric, and preemption for high-priority work.
• Compare it with the bank's GPU-aware pod scheduler design and with the reported scenario of short, high-priority inference sharing a GPU pool with long, low-priority fine-tuning jobs.
• Design the global management plane across dozens of data centers: what is global, what stays regional, and how each data center keeps operating while cut off from the global plane.
• Sketch the multi-exabyte object storage design for parallel reads of training data, plus the reported checkpoint-coordination scenario for a 512-node training run.

Deliverable: Three infrastructure designs, each with a placement or data-layout decision, a named failure mode and its recovery path.

Day 6 — Networking, infrastructure as code and observability

• Write the VLAN vs VXLAN vs VPC comparison: network layer, identifier space (12-bit VLAN IDs vs 24-bit VNIs), overlay vs underlay, and where each fits in multi-tenant isolation.
• Design automated Infiniband partitioning and parallel storage provisioning for bare metal: partition keys configured through the subnet manager, provisioning driven from an inventory source of truth, and verification before a node goes to a tenant.
• Lay out a Terraform and Ansible delivery pipeline: remote state with locking, plan review before apply, staged rollout, drift detection and rollback.
• Design a Prometheus and Grafana node-health pipeline that cordons and drains unhealthy nodes, with rate limits so the automation can never drain a whole cluster.

Deliverable: Written networking comparisons and two pipeline designs, each with the safety check that stops the automation from causing an outage.

Day 7 — Behavioural stories and a mixed mock loop

• Prepare stories for the reported prompts: a high-severity outage (root cause and prevention), a strong technical disagreement, a design document written while requirements shifted, and replacing urgent manual operations with automation.
• Write a fact sheet per project so scale, team size and timeline stay identical wherever the project comes up.
• Run a mock back to back: one design question from days 4-5, one Go concurrency problem from day 2 and one behavioural story.
• Note where an answer lost structure and redo that segment once.

Deliverable: Four rehearsed behavioural stories with fact sheets, and notes from one mixed mock loop.