CoreWeave Software Engineer Interview Guide

The 60-second brief

CoreWeave operates GPU clusters designed for large-scale AI training, high-throughput inference and other cloud workloads, on a platform built on Kubernetes and Linux. Software Engineers build the services, controllers, APIs and automation that provision, secure and scale that compute, mostly in Go and Python, with C++ on firmware and hardware-adjacent teams.

This guide covers the five rounds candidates report for the CoreWeave Software Engineer loop, from the recruiter screen to the final leadership conversations. It also covers the question categories reported for the role: practical API and concurrency coding, infrastructure system design, Linux and Kubernetes operations, systems programming and platform security, and behavioral questions. Each round below says what to prepare, and the seven-day plan assigns those categories and the worked exercises to specific days.

Focus areas

• Scope every query and cache key by tenant
• Build at-least-once pipelines with explicit deduplication horizons
• Evolve APIs without breaking pinned SDK clients

Overview

CoreWeave runs GPU clusters built for large-scale AI training, high-throughput inference and other cloud workloads. Software Engineers work on the platform layers that turn that hardware into cloud resources. Teams named for the role include Kubernetes Core Interfaces, Inference Platform, Server Fleet Infrastructure, SaaS Infrastructure and Security Infrastructure. The work ranges from control planes and device-provisioning pipelines to data ingestion systems and identity and access management.

Most backend and platform work is described as Go and Python, with C++ for firmware and hardware-adjacent teams, all running on Kubernetes and Linux. Engineers are described as owning the operational health of what they deploy: defining SLIs and SLOs, setting up observability with Prometheus and Grafana, building deployment pipelines and taking part in on-call rotations.

This has two consequences for preparation. First, the coding questions candidates report are practical: API clients, JSON parsing, concurrent fetching and command-line tools. Second, the design and operations questions are about infrastructure: fleet-wide reboots, multi-tenant IAM, event pipelines fed by hardware, and diagnosing a pod stuck in Pending. Drilling algorithms alone covers little of that. Building small working tools and reasoning about failure in real systems covers most of it.

The CoreWeave interview process

CoreWeave candidates report 5 rounds · ≈ 4-6 weeks. The stages below are what candidates describe, not a published process.

Recruiter Screen

Half of this call is the part candidates treat as small talk: start date, notice period, work authorisation and its timing, location and time zone, on-call, and the number. Those are what kill offers late, after several engineers have each spent a day. Surfacing a hard constraint now costs you nothing and occasionally buys you something, since a loop compressed to fit a competing deadline can usually only be arranged if it is asked for early. The common failure is deflecting the compensation question twice, then discovering at offer stage that the band never reached your number.

What to demonstrate

• Whether your hard constraints are compatible with the role before a loop gets booked: earliest start, notice period, what authorisation you hold and when it needs action, days on site, willingness to carry a pager
• Whether you give a compensation range with something behind it, such as current total compensation or a competing timeline, rather than leaving the band untested
• Whether your stated timeline is real, since a competing deadline raised now is something scheduling can sometimes work around and the same deadline raised at offer stage usually is not

How to prepare

• Write each constraint down in one line before the call and state them as facts rather than negotiating them live under a question you were not expecting
• Set your range from two or three current data points for that level and location, and name the structure you are quoting in, so the number is comparable to the one they are holding
• If another process is running, say where it stands and by when, and ask directly whether this loop can be scheduled inside that window

Technical Assessments

Candidates describe this stage as one or more technical screens that check baseline fluency. Depending on team and level, it may include a timed take-home or an online pre-assessment on Linux and Kubernetes fundamentals. A live CoderPad pair-programming screen follows, built around practical API automation, concurrency or command-line tooling. The reported coding questions in this guide fit that shape: call a /reboot or /shutdown endpoint and parse the JSON it returns, fetch content with a bearer token, compute a checksum over data you can only reach through an API, or fan requests out concurrently in Go. Treat it as a small production task rather than a puzzle. Aim for working code with timeouts, status checks and clear error handling, and keep live solutions plain rather than wrapping them in abstract design patterns.

What to demonstrate

• Whether the code runs end to end against the endpoint as described: correct method, headers and bearer token, a request timeout, and a status check before the body is trusted
• Whether failure paths are handled explicitly: a non-2xx response, a body that is not valid JSON, a missing field, and a server that never answers
• Whether concurrency is bounded and safe: a fixed worker pool or semaphore rather than one goroutine or thread per item, no unsynchronised writes to shared state, and per-item errors collected instead of lost
• Whether your Linux and Kubernetes fundamentals hold up, if your team's version includes a pre-assessment on them

How to prepare

• From a blank file and without autocomplete, write a Python client (requests or the standard library) and a Go client (net/http). Each should POST a JSON body with a bearer token, set a timeout, check the status and decode the response into a typed structure
• Build a small CLI that reads server serial numbers from arguments or stdin and calls a reboot endpoint for each through a bounded pool (a WaitGroup and buffered channel in Go, ThreadPoolExecutor in Python). Have it print a success or error line per serial
• Stand up a local HTTP server that sometimes returns 500s, slow responses or malformed JSON. Run your client against it until every failure is reported instead of crashing the program
• Ask the recruiter which language the screen expects. Some teams reportedly require Go

Architectural Discussions

This round is described as in-depth discussion of system design and architecture. The reported design questions are about infrastructure: batch reboots across thousands of physical nodes while keeping the cluster available, and an IAM system for a multi-tenant cloud with fine-grained authorization over hardware and API resources. Others are a Kubernetes platform that runs user-supplied runtimes and scales pods with load, and an event system where hardware and external APIs publish events and new internal subscribers can attach without a redesign. Candidates also describe a deep dive into the architecture of past projects in the main loop, so have one of your own systems ready at the same depth. Strong answers here carry operational detail: data flow, API contracts, where state lives, what happens when a node or dependency fails, and how the system is rolled out and observed.

What to demonstrate

• Whether you pin down scope and scale before drawing anything: who calls the system, how many nodes or tenants it serves, and what availability must hold while it runs
• Whether failure handling is concrete: what happens to a reboot batch when nodes fail to come back, to an authorization check when the policy store is unreachable, or to an event when a subscriber is down
• Whether storage and consistency choices are justified against the access pattern rather than just named, including where you accept staleness
• Whether you can walk through a system of your own down to its contracts, failure modes and the decisions you would change

How to prepare

• Work the batch-reboot design end to end: select nodes by failure domain, limit concurrency per rack or cluster, cordon and drain before rebooting, and require a health check before the next batch. Add an abort threshold, and make requests idempotent so a restarted orchestrator cannot reboot a node twice
• For the IAM question, separate authentication from authorization and write the policy model: principals, resources, actions, conditions and the tenant boundary. Then decide where decisions are cached, for how long, and how a revocation reaches that cache
• For the event system, design the envelope and the schema-versioning rule first. Then cover partitioning, delivery semantics (at-least-once with consumer deduplication), retention and replay, and how a new subscriber is added without producer changes
• Pick one system you built and prepare its diagram, API, data model, one incident and one decision you regret, so it holds up whichever of these the interviewer probes first

Behavioral Interviews

Candidates describe interviews with peers and leadership that cover cultural alignment and operational mindset. The reported behavioral questions ask about a technical disagreement with a manager or senior engineer, your biggest engineering mistake and how you communicated it, why you want this company, and how you balance shipping quickly against long-term stability and technical debt. Prepare a small set of stories deep enough to survive repeated follow-up questions rather than one rehearsed answer per prompt, and decide in advance which project you would use for each prompt.

What to demonstrate

• Whether a disagreement story ends with something checkable, such as a benchmark, a prototype or a written comparison, and a clear account of who decided
• Whether the mistake story names your own action, its impact, how and when you told stakeholders, and the guardrail you added afterwards
• Whether a speed-versus-debt answer names what you shipped, what you knowingly deferred, how it was tracked and when it was paid down
• Whether your reason for wanting the job refers to the actual work, such as GPU infrastructure, Kubernetes control planes or fleet automation, rather than to the company in general

How to prepare

• Choose four projects and write each one out several levels deep: what you did, why that option, why not the alternative, and what you would change now. If you cannot reach the last level for a project, replace it with another
• Have someone ask why three times in a row on one thread of a story, and note where you start repeating yourself. That is where the story needs more detail
• Build a speed-versus-debt answer around one concrete trade you made: what you shipped, what you knowingly deferred, how you tracked it, and when it was paid down
• Answer why this company in terms of two or three problems from the role description that you want to work on

Final Leadership Conversations

Coding rounds mostly set a floor. They decide whether you clear the bar, not where you land on the ladder. Level tends to come out of the design discussion and the ownership stories, so the question worth auditing beforehand is whether the scope you describe matches the scope of the job. Work that stops at your own service, or a story whose hard part was writing the code rather than getting several people to agree on an interface, reads a level below where you think you are interviewing, and that gap is usually resolved downwards.

What to demonstrate

• Whether the largest thing you describe owning ran end to end — the decision, the migration path, the rollout, and what you did when it went wrong — or stopped at the change you merged
• Whether design answers include what you would not build, what you would defer, and what you would measure before committing, rather than only what the boxes are
• Whether a disagreement in a story was settled with something checkable — a benchmark, a prototype, a written proposal — instead of by seniority or by waiting it out
• Whether you can say which calls you made alone and which you escalated, and why the line sat where it did

How to prepare

• Write your largest piece of owned work as a timeline of decisions — who decided what, when, and what you did when the plan broke — then delete every sentence whose subject is "we" and see how much survives
• Take one system you know well and drill the migration answer: how old and new paths run side by side under live traffic, how you compare their outputs, what the rollback is once writes are going to both, and which step you would not automate
• Map each line of the ladder in the job posting to a specific thing you have done, find the line you cannot support, and prepare the closest evidence you have plus an honest account of the gap

Where candidates lose points

Writing an API client that only handles the happy path

The reported coding tasks are small: call /reboot or /shutdown, fetch with a bearer token, compute a checksum over data reachable only through an API. With tasks that small, failure handling is what separates answers. Set a timeout on every request, check the status code before decoding, and handle a body that is not valid JSON or is missing the field you need. Report a clear error per item instead of a stack trace. Say out loud which errors you would retry and which you would not: a reboot or shutdown call is not obviously safe to repeat after a timeout.

Spawning unbounded goroutines or threads for a fan-out task

For concurrent fetching or batch operations, cap concurrency with a worker pool, a semaphore or ThreadPoolExecutor's max_workers. Close channels from the sending side only and wait with a WaitGroup or errgroup. Collect per-item errors so that one failure neither hangs the program nor disappears. Protect any shared map or counter with a mutex or confine it to one goroutine, and name go test -race as the way you would check.

Designing a fleet-wide reboot as a loop over every node

The batch-reboot and Kubernetes platform questions turn on keeping the system available while it changes. Size batches by failure domain, cordon and drain before rebooting, and require a health check to pass before the next batch starts. Set an abort threshold for when too many nodes fail to return. Make the operation resumable and idempotent so a crashed orchestrator cannot reboot a node twice. Say what an operator sees on a dashboard while the rollout runs.

Remediating a Pending pod before reading the evidence

Start with kubectl describe pod and its events, which usually state the scheduler's reason: insufficient CPU, memory or GPU against allocatable, an untolerated taint, an affinity or node-selector rule no node satisfies, or an unbound PersistentVolumeClaim. Only then change something, and explain why the change is safe. For example, lower a request rather than add a toleration that lets the pod land on nodes reserved for other workloads. Keep quota problems separate: a ResourceQuota or LimitRange violation is rejected at admission, so the pod is never created. If the pod is missing rather than Pending, look for quota errors in the ReplicaSet or Job events.

Telling the incident or mistake story without the prevention step

The reported behavioral and operations questions ask how you triaged, recovered and prevented a recurrence. A story that ends at recovery leaves out the part that shows ownership. Close with the specific guardrail you added, such as a dry-run mode, a canary batch, rate limiting, an approval step, or an alert backed by an SLO, and with how you know it has worked since.

Questions & practice

Explain memory management principles in C++, detailing stack vs. heap allocation, smart pointers, and strategies to prevent memory leaks in long-running processes.

Medium · languages, concurrency and fundamentals

Approach

1. Say what the runtime actually does before reasoning about the code.
2. Name what is shared across threads and what owns each piece of state.
3. Reach for the cheapest primitive that closes the race, not the broadest lock.

Follow-up

• How would you prove the race exists rather than suspect it?
• Where could this allocate more than you expect?

How do you structure secure authentication flows (such as mTLS, OIDC, or API access keys) between control plane services and edge compute nodes?

Medium · languages, concurrency and fundamentals

Approach

1. Name what is shared across threads and what owns each piece of state.
2. Distinguish a value from a reference to it, and say which one you handed out.
3. Reach for the cheapest primitive that closes the race, not the broadest lock.

Follow-up

• How would you prove the race exists rather than suspect it?
• Where could this allocate more than you expect?

Parse and verify a timestamped multi-signature webhook header

Easy · parsing · hmac · constant-time-compare · replay-protection

An inbound webhook carries a signature header of at most 1 KiB shaped t=<unix seconds>,v1=<64 hex chars>, with up to five v1 values during secret rotation and possibly unknown scheme keys. You hold the raw request body bytes and the currently active signing secrets. Write the parser and the verifier: accept when any active secret reproduces a signature and the timestamp is within a five-minute tolerance in either direction, reject otherwise. Single left-to-right pass over the header, no regular expression. State what is inside the MAC and why.

Approach

1. Parse in one scan: split on ,, then on the first = only, since a value may itself contain = under a future scheme. Accept t exactly once and treat a second t as a reject rather than last-wins. Push every v1 onto a short list and ignore any other key, so a v2 can be introduced later without breaking this verifier.
2. Say what is signed: HMAC-SHA256 over the exact byte string <t>.<raw body bytes>, yielding 32 bytes or 64 hex characters. The timestamp sits inside the MAC because otherwise an attacker replays yesterday's body with its still-valid signature and only has to edit the header timestamp.
3. Hash the bytes as received. Verifying against a re-serialised JSON body is the usual defect: key order, whitespace and number formatting all change the bytes while the parsed objects compare equal, so signatures fail for honest senders and the popular 'fix' is to stop checking.
4. Compare in constant time over fixed-length digests. Decode the hex to 32 bytes, accumulate acc |= a[i] ^ b[i] across the whole length, and test acc == 0 at the end. Evaluate every candidate without an early exit; at five candidates that is five HMACs over the body, linear in body size and negligible beside the network.
5. Apply the tolerance as a two-sided bound, rejecting when |now - t| > 300 seconds. A sender whose clock runs ahead of yours is an ordinary case, and an unbounded future timestamp is a free replay window.
6. Complexity: O(L) over the header producing k candidates, plus k HMACs at O(|body|) each. Space is O(k) beyond the body itself. Do the cheap rejections, including the tolerance check, before any cryptography runs.

Worked solution (15 min)

1. Write the grammar on one line before coding: header := field (',' field)*, field := key '=' value, split on the first = only.
2. Implement the parser to return {t: int, v1: [hex, ...]}, rejecting a missing t, a duplicate t, any v1 that is not 64 hex characters, and a header over 1 KiB, all before any cryptography runs.
3. Implement the verifier: for each active secret compute HMAC-SHA256(secret, f'{t}.'.encode() + raw_body), compare it in constant time against each parsed v1, and OR the results with no early exit.
4. Test with a valid signature; the same body with t moved 400 seconds into the past; the same body with t 400 seconds into the future; a header carrying an unknown v2= alongside a valid v1; and a body re-serialised with different JSON key order.

Expected result: The valid case accepts. Both out-of-tolerance cases reject, including the future one. The unknown v2 field is ignored and the v1 still verifies. The re-serialised body fails, which is correct and is exactly why the raw bytes must be retained.

Verify it yourself

• Timing the comparison over 10,000 mismatching signatures that differ at the first character versus the last shows no separation you can measure.
• Removing t from the MAC input makes an old captured body with its original signature pass. Write that test, watch it pass, then put t back.
• Every rejection except the signature mismatch itself returns before an HMAC is computed.

Follow-up

• The body is 40 MB. What changes about where you verify, and what can you do before the whole body has arrived?
• A customer reports that signatures fail for exactly the requests whose body contains a non-ASCII character. What is your first hypothesis?
• How do you rotate the signing secret with no failed deliveries, and how long do both secrets stay live?

Migrate a live partitioned event table without blocking ingest

Hard · online migration · ddl locks · expand and contract · partitioning

usage_event is range-partitioned daily on ingested_at, holds roughly 250M rows per day across 400 live partitions, and is written at 10-40k rows/second. Two changes are required: quantity must move from double precision to numeric(20,6), and a new environment column must become NOT NULL with a default of 'production'. Ingest cannot stop. Give the ordered plan, naming for each step the lock it takes, what that lock blocks, and roughly how long it is held. Identify the one step that cannot be rolled back cleanly once traffic depends on it.

Approach

1. Classify the two changes before planning anything. Adding a column with a non-volatile default has been metadata-only since PostgreSQL 11, so it is cheap. Changing double precision to numeric is not binary-coercible, so alter column ... type rewrites every partition under ACCESS EXCLUSIVE and rebuilds its indexes; on this volume that is hours of blocked ingest and is simply not an option, which is why the plan is expand-and-contract rather than one statement.
2. Expand: add quantity_numeric numeric(20,6) and environment with its default on the parent. Both are catalogue-only but both take a brief ACCESS EXCLUSIVE that cascades to partitions, so run each with lock_timeout set to a second or two and retry on failure. A queued ACCESS EXCLUSIVE request blocks every reader behind it, which is how a metadata-only change turns into an outage.
3. Dual-write: deploy producer code that populates both columns on every insert, and leave it running before anything reads the new column. This is the step that cannot be reverted cleanly. Once readers depend on quantity_numeric, reverting the writer leaves rows with a null there, and the gap is only discoverable by re-reading the old column, which the readers have stopped doing.
4. Backfill older partitions in batches keyed on the primary key, oldest first, committing every few thousand rows with a pause between batches, and skipping the partition still receiving writes until it rotates. Each batch is an ordinary UPDATE taking row locks only. The cost is bloat and WAL rather than blocking, so watch dead tuples and let autovacuum keep pace instead of wrapping 400 partitions in one transaction.
5. Make NOT NULL cheap with the three-step form: add constraint ... check (environment is not null) not valid (brief ACCESS EXCLUSIVE, no scan), then validate constraint (SHARE UPDATE EXCLUSIVE, scans while reads and writes continue), then set not null, which from PostgreSQL 12 uses the validated check and skips its own full scan. Do this per partition, then on the parent.
6. Switch and contract: move reads to the new column behind a flag, verify over a full period that both columns agree on freshly written rows, drop the old column (metadata-only), and only then remove the dual-write. Any index on the new column goes on with CREATE INDEX CONCURRENTLY per partition, since CIC is not supported on a partitioned parent: create the parent index with ONLY, build each child concurrently, then ALTER INDEX ... ATTACH PARTITION until the parent index becomes valid.

Worked solution (45 min)

1. On a scratch cluster, build 10 partitions of 2M rows each and run a writer at a few thousand inserts/second.
2. Run the naive type change and measure how long writes stall and how far ingest lag grows before killing it.
3. Run the expand step with lock_timeout = '2s' while the writer runs, and observe a clean lock timeout and retry instead of a pile-up of blocked readers.
4. Backfill in 5k-row batches and chart dead tuples and WAL generated per batch.
5. Run the not-valid, validate, set-not-null sequence and confirm from pg_stat_activity and timings that nothing held an exclusive lock through a full scan.
6. Add an index with CIC per partition plus ATTACH PARTITION and confirm the parent index reports valid only after the last attach.

Expected result: An ordered expand, dual-write, backfill, switch, contract plan with the lock named per step: metadata-only ADD COLUMN under a brief ACCESS EXCLUSIVE taken with lock_timeout; batched UPDATE backfill under row locks; CHECK NOT VALID, then VALIDATE under SHARE UPDATE EXCLUSIVE, then SET NOT NULL; CREATE INDEX CONCURRENTLY per partition with ATTACH PARTITION; and the dual-write-to-read switch identified as the step that cannot be reverted cleanly.

Verify it yourself

• No step in the plan holds ACCESS EXCLUSIVE for longer than the configured lock_timeout.
• The writer's error rate stays at zero through the rehearsal apart from the deliberate lock timeout.
• After VALIDATE, SET NOT NULL completes in milliseconds on a table where the naive form scanned every row.
• Freshly written rows agree on both columns for a full verification period before the old column is dropped.

Follow-up

• A CREATE INDEX CONCURRENTLY fails halfway through the partition list. What state is the table in, how do you detect it, and what do you run?
• The producer computes quantity itself. What happens to a request already in flight when the dual-write deploy lands, and does it matter?
• Give two queries that prove the backfill is complete: one cheap enough to run every minute, one authoritative.

Enforce a concurrent-run quota that survives simultaneous requests

Medium · isolation levels · write skew · quotas · lock contention

A plan allows at most 20 concurrently running rows in job_run per tenant. The table holds run_id, tenant_id, workspace_id, status (queued, leased, running, succeeded, failed, timed_out, cancelled, lost), lease_token, leased_until, started_at and finished_at. Today the service runs select count(*) from job_run where tenant_id = $1 and status = 'running', compares the result to 20, then inserts. Under load a tenant exceeds the cap by exactly the number of concurrent requests. Name the anomaly, say which isolation levels do and do not prevent it, and give a version that holds, as SQL.

Approach

1. Name it: write skew. Each transaction reads a predicate (the count of running rows), neither modifies what the other read, and both then insert rows that jointly violate an invariant no single row expresses. Read committed permits it. So does repeatable read, because snapshot isolation's first-updater-wins check fires only on conflicting row updates, and these are inserts touching disjoint rows.
2. Enumerate the fixes with their real costs. SERIALIZABLE works: PostgreSQL's SSI tracks the predicate read and aborts one transaction with SQLSTATE 40001, which obliges the caller to retry and makes the abort rate rise with contention on a hot tenant. Folding the predicate into the write as insert ... select ... where (select count(*) ...) < 20 narrows the race to the statement's snapshot but does not close it under read committed.
3. Give the version that holds at read committed: serialise on a row both transactions must touch. update tenant_concurrency set running = running + 1 where tenant_id = $1 and running < 20 returning running updates zero rows when the cap is reached, and zero rows is the rejection. This works because at read committed a blocked UPDATE re-evaluates its WHERE clause against the newly committed row; at repeatable read the same statement raises a serialisation error instead, so the isolation level changes the calling contract.
4. State the cost you just bought. That row is now a per-tenant serialisation point, so admission throughput for the tenant is bounded by one divided by the lock hold time; at a 2 ms hold that is roughly 500 admissions/second. Keep the critical section to the single UPDATE, with no network call or scheduling decision inside the transaction, and decrement in the same transaction that writes the terminal status.
5. Close the leak the status enum implies: a run can end as lost, so a crashed worker otherwise consumes a slot forever. Reconcile on a schedule against status = 'running' and leased_until < now(), and treat the counter as a fast path over job_run, which stays the system of record.

Follow-up

• Write the retry loop for the SERIALIZABLE version. What does the caller see when it keeps aborting, and what bounds the retries?
• Two regions each keep a counter. What is the effective cap, and what does admission do when the counter store is unreachable?
• The cap changes mid-flight on a plan upgrade. Do running jobs get killed, and what does the counter row look like during the change?

Design a Kubernetes-based platform that enables users to upload custom application runtimes and automatically scale compute pods based on real-time load.

Medium · system design and architecture

Approach

1. Name the read and write paths separately; they rarely have the same bottleneck.
2. Name the failure you are designing for, then the recovery path.
3. Choose a partition key and say what query it makes expensive.

Follow-up

• What would you drop to keep the system up under load?
• What breaks first when traffic grows ten times?

Design an Identity and Access Management (IAM) system for a multi-tenant cloud platform handling fine-grained authorization for hardware and API resources.

Medium · system design and architecture

Approach

1. State the consistency you need, and where you are willing to be stale.
2. Name the failure you are designing for, then the recovery path.
3. Fix the scope first: who calls this, how often, and what they do when it fails.

Follow-up

• What would you drop to keep the system up under load?
• What breaks first when traffic grows ten times?

Design a system capable of orchestrating batch reboots across thousands of physical server nodes reliably while maintaining cluster availability.

Medium · system design and architecture

Approach

1. Choose a partition key and say what query it makes expensive.
2. Fix the scope first: who calls this, how often, and what they do when it fails.
3. Name the failure you are designing for, then the recovery path.

Follow-up

• What breaks first when traffic grows ten times?
• How does this behave when that dependency is down for an hour?

Architect an event-driven system where hardware components and external APIs produce events, allowing multiple internal services to subscribe dynamically without requiring architectural redesigns.

Medium · system design and architecture

Approach

1. Name the failure you are designing for, then the recovery path.
2. Fix the scope first: who calls this, how often, and what they do when it fails.
3. State the consistency you need, and where you are willing to be stale.

Follow-up

• What would you drop to keep the system up under load?
• What breaks first when traffic grows ten times?

Build a CLI tool or script that interacts with a backend API endpoint (e.g., POST /reboot) to pass server serial numbers and execute hardware state changes.

Medium · api and integration design

Approach

1. Separate accepted, pending, failed and confirmed; they are different facts.
2. Say who the caller is and what they do when the call fails halfway.
3. State how the contract changes without breaking existing clients.

Follow-up

• What happens if the caller retries after a timeout?
• How does a client discover it is on an old version of this contract?

Write a program given an API endpoint (e.g., /shutdown) that sends a request to shut down a server and correctly parses the returned JSON payload.

Medium · api and integration design

Approach

1. State how the contract changes without breaking existing clients.
2. Say who the caller is and what they do when the call fails halfway.
3. Design the error taxonomy before the success shape; callers branch on it.

Follow-up

• What does a partial failure look like to the caller?
• How does a client discover it is on an old version of this contract?

Per-tenant rate limiting that holds across one hundred pods

Medium · rate limiting · hot keys · atomicity · write skew

The same gateway enforces each tenant's plan rate limit in requests/second and its monthly quota. Traffic is heavily skewed: a handful of tenants exceed all others combined, while a small tenant's requests may all land on one of the 120 pods. Both limits must hold in aggregate rather than per pod, per credential or per region, and the check may add at most 2 ms at p99. Specify the algorithm, where the state lives, the exact operation performed per request, and the behaviour when the counter store is unreachable - separately for the rate limit and the quota.

Approach

1. Do the arithmetic on per-pod buckets before discarding them, and be precise about which half fails. Giving each pod rate/N keeps the sustained aggregate correct - 120 buckets of 5 rps sum to the 600 rps limit - but a tenant whose traffic all lands on one pod is throttled at 5 rps, a hundred and twentieth of what it pays for. The over-admission comes from burst depth, not sustained rate: if each pod carries a full bucket depth B, the aggregate burst is 120B against a limit that intended to allow B.
2. Move the rate limit to a shared counter accessed in exactly one atomic round trip - a token bucket or GCRA as a single server-side script returning allow/deny plus a retry-after - never a read followed by a write. Keep the key tenant-scoped and region-local; a globally shared counter costs a cross-region round trip that alone exceeds the 2 ms budget, so shard the global limit per region in proportion to observed regional share and accept that a tenant whose traffic shifts regions is briefly throttled until the shards rebalance.
3. Cut round trips for the largest tenants with leased tokens: a pod leases 50 at a time and spends them locally, reducing counter round trips by 50x at the cost of bounded over-admission of pods x lease_size, which is 6,000 requests in the worst case here. Shrink the lease for tenants with small limits, where that bound would swamp the limit itself.
4. Treat the quota as a different problem, not a longer window. It is a count that other in-flight requests are changing, so select-the-usage-then-insert is write skew: read-committed permits it and repeatable read permits it too, because snapshot isolation is exactly what allows two transactions to read a consistent count and both write. Enforce it inside one statement - an atomic increment whose returned value is compared, or UPDATE ... SET used = used + $1 WHERE used + $1 <= limit RETURNING - or with a constraint that makes the surplus write fail.
5. Write the unavailability policy per limit rather than one policy for both. The rate limit fails open to a conservative local bucket, because it protects capacity and failing closed turns a counter blip into a total outage. The quota fails open for tenants whose last known usage was well below the cap and closed for those at or above it, since that is the only decision with revenue attached, and reconciles from the usage ledger on recovery - which is why the monthly counter must be re-seedable from that ledger rather than being the only copy.

Worked solution (30 min)

1. For a 600 rps limit across 120 pods, compute the sustained rate and the aggregate burst under local buckets of depth B, then the throughput of a tenant whose traffic all lands on one pod.
2. Write the shared-counter operation as a single atomic script: its inputs, its return value, and the number of network round trips per request.
3. Size a lease of 50 tokens: compute the worst-case over-admission as pods x lease_size and the reduction in round trips, then decide the smallest limit at which the lease is still acceptable.
4. Write the two unavailability policies side by side and the reconciliation step that runs when the counter store returns.

Expected result: A per-region shared token bucket keyed (tenant, region, window) executed as one atomic script at roughly one sub-millisecond intra-region round trip; the monthly quota enforced by a single atomic increment-and-compare that is re-seedable from the usage ledger; leases only for high-limit tenants with over-admission bounded at 120 x 50 = 6,000 requests; the rate limit failing open to a conservative local bucket and the quota failing closed only above the last known cap.

Verify it yourself

• Separate the two halves of the local-bucket claim: sustained aggregate is correct at rate/N per pod, burst is not. If your answer says local buckets let a tenant exceed its sustained rate, the arithmetic does not support it.
• Re-read your quota check and confirm no branch reads the counter and then writes it. If the predicate is not inside the write or enforced by a constraint, the race is still there.
• Compare worst-case lease over-admission against the smallest tenant limit you apply leases to. If 6,000 exceeds that tenant's per-minute allowance, the lease size is wrong for that tier.

Follow-up

• One tenant is 40% of all traffic and its counter key lands on a single shard. What do you change, and what does that cost in accuracy?
• Clients receive 429s and retry. What stops the well-behaved ones from synchronising into a herd?
• A tenant reports being throttled below its limit. Which evidence do you produce, and does your design emit it today?

Describe an incident where a production automation script failed or misconfigured hardware. How did you triage the issue, recover the service, and prevent future recurrences?

Medium · debugging and production issues

Approach

1. Say what evidence would prove you wrong, then go and look for it.
2. Establish what changed and when, before forming any theory.
3. Check the instrumentation before believing the symptom.

Follow-up

• What would you look at first, and what would it rule out?
• How would you tell a cause from a coincidence here?

Suggested practice checklist

For a candidate senior enough that the loop turns on design and judgement rather than on whether the coding round gets finished. Five days build one system properly and then stress it; coding gets a single maintenance day, on the assumption that the risk at this level is an unexamined tradeoff rather than a missed algorithm.

Day 1 — Recruiter screen and API client fundamentals

• Before the recruiter screen, write down your constraints. Ask which team the loop is for, which language the coding screen expects, and whether it includes a Linux and Kubernetes pre-assessment.
• Write an HTTP client in your screen language that POSTs a JSON body with a bearer token and a timeout to a /reboot-style endpoint, checks the status and parses the JSON response (reported-api-5).
• Repeat for a /shutdown endpoint, giving a non-2xx status, invalid JSON and a missing field each their own error message (reported-api-6).

Deliverable: Two working API clients, a one-line-per-item constraints sheet, and your questions for the recruiter.

Day 2 — Concurrency and CLI tooling

• Build a Go program that takes a list of hostnames, queries a status endpoint for each through a bounded worker pool, and prints healthy and unhealthy nodes with per-host errors.
• Rebuild the reboot CLI in Python with ThreadPoolExecutor, argument parsing and a final summary. Run it against a local server that returns failures and slow responses.
• Write a program that reads data from an API endpoint and computes a deterministic checksum over it. State what the checksum covers, either the raw bytes as received or canonicalised records (sorted keys, stable ordering across pages), and justify why your choice gives the same result on repeated calls.
• Work the drill-coding-3 exercise (webhook signature parsing) to practise the same habits: parse strictly, reject early, and be explicit about exactly which bytes are verified.

Deliverable: A bounded-concurrency Go fetcher that passes the race detector and a Python CLI; both report per-item failures without hanging.

Day 3 — Linux and Kubernetes operations

• Write your diagnostic sequence for a pod stuck in Pending: events, the scheduler message, requests versus allocatable, taints and tolerations, affinity and PVC binding. Add a separate note for pods that never appear, where quota or LimitRange rejections show up in the controller's events.
• Practise tcpdump, strace, journalctl and sysctl on a local VM. For each, write which question it answers when you are isolating network or storage latency.
• Define SLIs and SLOs for one platform service, such as a node-reboot API, and write the condition each would page on.
• Prepare the automation-incident story (reported-debugging-9) as a timeline that ends with the guardrail you added.

Deliverable: A one-page Pending-pod runbook, an SLI and SLO sheet for one service, and a written incident timeline.

Day 4 — Fleet and event-system design

• Design batch reboots across thousands of physical nodes (reported-systemdesign-3): batching by failure domain, drain, health gates, an abort threshold, and resumable, idempotent orchestration.
• Design the event system where hardware and external APIs publish events and new subscribers attach without a redesign (reported-systemdesign-4): envelope, schema versioning, partitioning, delivery semantics and replay.
• Work the drill-design-4 exercise (per-tenant rate limiting across pods) to practise stating the exact per-request operation and the behaviour when a shared store is unreachable.

Deliverable: Two designs, each with its failure cases and its rollout and observability plan written next to the diagram.

Day 5 — Multi-tenant platform and security design

• Design the multi-tenant IAM system (reported-systemdesign-2): policy model, tenant boundary, where decisions are evaluated and cached, audit trail, and how revocation propagates.
• Design the Kubernetes platform for user-supplied runtimes with load-based scaling (reported-systemdesign-1): tenant isolation, admission checks, the metric that drives scaling, and scale-down behaviour.
• Explain authentication between control-plane services and edge nodes (reported-lowlevel-8): mTLS certificate issuance and rotation, OIDC for human users, scoped API keys, and how each is revoked.
• Work drill-sql-2 to practise the write-skew reasoning behind any per-tenant limit you enforce in these designs.

Deliverable: An IAM design with its policy model and revocation path, and a platform design with its tenant-isolation and scaling rules.

Day 6 — Systems fundamentals and behavioral stories

• Explain C++ memory management (reported-lowlevel-7): stack versus heap, RAII, unique_ptr and shared_ptr, weak_ptr for breaking cycles, and how you would find a leak in a long-running process. Go deeper if your team is firmware or hardware-adjacent.
• Write a short explanation of root of trust, secure boot and platform attestation, following the chain from hardware to operating system.
• Write four stories covering a technical disagreement, your biggest mistake, speed versus technical debt (reported-behavioral-10) and why this company, then practise follow-ups with drill-behavioral-5 and drill-behavioral-6.

Deliverable: A one-page explainer on C++ memory and secure boot, plus a story index that maps each behavioral prompt to a project.

Day 7 — Mock loop and leadership conversation

• Run a mock coding screen with a partner: an API and concurrency task, done under a timer in a plain editor without autocomplete.
• Run a mock design round on batch reboots or IAM in which the interviewer changes one requirement partway through.
• For the final leadership conversation, write your largest piece of owned work as a timeline of decisions. Use the drill-sql-1 exercise as a model for describing a live migration with its rollback points.
• Prepare questions for your interviewers about the team, its on-call rotation and what your first project would be.

Deliverable: Notes from both mocks with the fixes you made, and a written decision timeline for your largest piece of owned work.

Risky changes and the reversal path

The reported behavioral questions cover a technical disagreement, your biggest mistake, why this company, and balancing delivery speed against technical debt. One reported operations question asks about an incident caused by automation. Prepare four or five stories you can defend through several follow-up questions, and tell operational ones in order: detection, triage, mitigation, recovery, prevention.

1. Tell me about a significant technical disagreement with a manager or senior team member. How did you handle it, and how was it resolved?
2. Describe your biggest engineering mistake in a recent role: the impact, how you communicated it to stakeholders, and what you changed afterwards.
3. Why do you want to work at CoreWeave, and which part of its GPU cloud and infrastructure platform interests you most?
4. How do you balance shipping features quickly during rapid growth against long-term architectural stability and reducing technical debt?
5. Describe an incident where a production automation script failed or misconfigured hardware. How did you triage it, recover the service, and prevent a recurrence?
6. How would you define SLIs and SLOs for a platform service to reduce on-call burden?

How do you balance shipping features quickly during hyper-growth against building long-term architectural stability and reducing technical debt?

Medium · behavioural and engineering judgement

Approach

1. State the situation in two sentences and spend the rest on the reasoning.
2. Name the disagreement and how you resolved it with evidence.
3. Pick a story where you made the decision, not one where you watched it.

Follow-up

• How did you know your change caused the improvement?
• What did you decide not to do, and why?

Ship metered billing with a named deduplication horizon

Medium · technical debt · deduplication · deadline pressure · detectors

Metered billing must be on in three weeks. usage_event is partitioned daily, so its unique index must include the partition key and deduplicates only within a day: a producer retry that crosses midnight, or a replay run a week later, gets through. A cross-partition dedup store is two weeks you do not have. Describe shipping with debt you named in advance: what you shipped, what you wrote down, the detector you added, the trigger and date for paying it off, and what you would have refused to ship under the same pressure.

Approach

1. Show you can separate the two kinds of debt, because that distinction is what the question actually probes. Debt that costs engineering time later is shippable on a deadline. Debt that silently corrupts a number a customer gets charged for is not shippable unless the corruption is detectable, and detectability is the whole negotiation.
2. Make the exposure narrow and measured rather than gestural. The hole is duplicates whose occurrences straddle a UTC day boundary, plus any replay older than partition retention. Measure it before arguing about it: how often an idempotency_key recurs at all, and the distribution of the gap between first and last occurrence. If the ninety-ninth percentile of that gap is four minutes, the residual risk is a small band around midnight and you can say so numerically.
3. Add the detector before the feature, not after. A nightly job counting keys that appear in more than one partition is one grouped scan over recent partitions, and it converts a silent overcount into a page. State what it costs to run and what it fires on.
4. Buy the cheap half of the real fix immediately: extend partition retention so the dedup horizon exceeds the producer's maximum retry window plus the longest replay you intend to support. That reframes retention as a correctness parameter rather than a storage cost, which is the sentence you need on record before someone optimises the bill.
5. Make repayment mechanical instead of aspirational: a dated entry with a named owner, plus a threshold that pulls the date forward — first detector hit above N events, or first customer dispute. Debt with a trigger gets paid; debt with only a date does not.
6. Answer the second half honestly by naming what you would refuse under identical pressure: the sealing path, because a sealed row is frozen and a wrong number there stops being a bug and becomes an adjustment line, a dispute and an audit question.

Follow-up

• The detector fires on forty duplicate events for one tenant, and two of their invoices have already sealed. What happens next?
• Whom did you tell that the billing numbers had a known hole, and in what words?
• Finance asks you to cut storage by shortening partition retention. What do you say, and to whom?

Argue against failing open when the control plane is unreachable

Hard · revocation · cache staleness · fail-open vs fail-closed · influence

The gateway caches credential-to-context decisions with a sixty-second TTL. A design proposal says that when control-plane reads fail, pods should keep serving from expired entries indefinitely so a control-plane outage never becomes a product outage. You believe that converts every revocation into an unbounded one. Describe a design you argued against while it was still a live proposal: what you measured or modelled to make the case, what you conceded, who decided, and what happened afterwards. Say what would have changed your mind before the decision, not after it.

Approach

1. Reframe it from a values argument into a bounded-staleness argument. Both sides already accept the cache; the disagreement is only about the ceiling on how long a revoked credential keeps authorising. Put a number on the table — serve stale for up to fifteen minutes, then fail closed — and make the other side argue against a number rather than against a principle.
2. Bring arithmetic rather than adjectives: the rate of revocations with revoked_reason in ('suspected_leak','auth_version_bump'), the observed distribution of control-plane unavailability, and the product of the two, which is expected requests served by revoked credentials per outage-hour. At 30k requests/second the unbounded version is not a subtle exposure and the number says so.
3. Concede the strong half of the opposing case first, because that is what buys you the room: failing closed turns one service's outage into a total outage across three regions, and a control plane doing tens of writes per second is not engineered to the gateway's availability target. A proposal you have not steelmanned reads as reflex.
4. Propose the asymmetry that usually resolves this: stale entitlements cost bounded money (a quota fifteen minutes out of date over-serves by a computable amount), while a stale revocation costs unbounded access. Split the cached decision by what it authorises, give the two halves different staleness ceilings, and let the entitlement half fail open while the revocation half fails closed.
5. State the propagation dependency plainly, since it is the part that is missed: validity is also derived from the principal's auth_version, so password reset and sign-out-everywhere flow through this same cache. A design that bounds staleness for explicit revocation and not for auth_version bumps has only fixed half of it.
6. Say who decided, and what you did afterwards in either outcome: write the decision down with its number and a review date, and instrument the exposure you were worried about so the next round of the argument is settled by data instead of by seniority.

Follow-up

• Publish-subscribe invalidation is lossy under a partition, and a TTL is the only hard bound. What TTL do you pick, and what does it cost you at 30k requests/second?
• The key was revoked because it was found in a public repository. Does your answer change, and where does that urgency live in the design?
• You lost the argument and six weeks later the failure you predicted happens. What do you say in the review, and what do you not say?