Interview questions · OpenAI

Design a sandboxed cloud IDE

This question evaluates a candidate's competency in designing multi-tenant distributed systems with strong isolation and sandboxing, runtime lifecycle management, real-time output/log streaming, data persistence, autoscaling, and observability.

System Design: Sandboxed Cloud IDE (Colab-like)

Design a multi-tenant, browser-based cloud IDE/notebook that lets users run code inside an isolated sandbox (similar to a hosted notebook product such as Colab). The defining challenge is safely executing arbitrary, untrusted user code at scale while keeping the experience interactive.

Core User Experience

A user opens a workspace (project/notebook), edits code in the browser, and runs cells or shell commands.
Output appears in the UI: stdout/stderr plus rich output (plots, tables, images).
Users can watch streaming logs while their code is still running.

Requirements

Functional

Provision an isolated compute environment per workspace/session.
Execute arbitrary user code safely (sandboxing).
Stream execution output and logs to the browser in near real time.
Support basic file operations (upload/download, persisted workspace state).
Basic real-time collaboration is optional — call it out explicitly if you choose to include it.

Non-functional

Strong isolation between tenants — security is the primary, non-negotiable constraint.
Reasonable startup latency when launching a new session.
Support autoscaling and fair resource sharing across users/orgs.
Observability: metrics, tracing, and audit logs.

Focus Areas to Cover

Compute substrate — how you choose and manage it (VMs vs. containers vs. microVMs vs. userspace-kernel sandboxes), including the GPU case.
Isolation model — filesystem, network, process/kernel, and credentials.
Log/output streaming architecture — near-real-time, resumable, backpressure-aware.
Lifecycle management — create, run, idle, suspend/resume, terminate.
Data persistence strategy — workspace files, runtime overlay, checkpoints/snapshots.

Please state your assumptions, provide an API sketch, and describe a high-level architecture (a diagram described in text is fine).

Hint: Where to start

The single hardest requirement here is running untrusted code. Treat the user as an adversary trying to escape the sandbox, steal credentials, or reach internal services — then optimize latency, density, and cost subject to that isolation guarantee, not the other way around.

Hint: Frame the planes

A common clean structure for this problem is a multi-tenant control plane (stateless: auth, scheduling, lifecycle, quotas) versus a per-session data plane (the actual sandboxes). Ask yourself which components must hold per-session state — that tier won't be stateless.

Hint

The answers to "concurrent active vs. idle tail" and "cold-start SLA" are what justify two of the biggest design levers — warm pools (for fast start) and suspend-to-snapshot (so the idle tail costs near-zero). Anchor those numbers early.

Hint: Capacity estimate

Size the fleet from both vCPU and RAM and take the binding dimension. Then notice that the idle/suspended tail must cost ~0 in running CPU/RAM — that single observation drives the suspend/snapshot strategy more than any other number.

Hint: Isolation boundary

For untrusted code, the shared host kernel is the attack surface: one kernel CVE = full host compromise across tenants. Rank options by what kernel they share — plain containers share the host kernel; a userspace-kernel sandbox or a microVM does not.

Hint: Default vs. exception

Pick a default substrate from your ranking and justify it on the three-way tension between isolation strength, cold-start, and density. Then pressure-test it against the GPU case: does whatever you chose for the common workload actually let you attach an accelerator? If not, what has to be different for those sessions?

Hint: Most-attacked path

Outbound network is where exfiltration and credential theft happen. From inside the guest, what is the most dangerous thing reachable over the network in a typical cloud environment, and how do you stop the guest from getting to it while still letting legitimate traffic (e.g. package installs) through? Separately: what credentials, if any, should the guest itself hold?

Hint: Snapshot hazard

Restoring many sessions from one pre-booted/snapshotted runtime is great for cold-start — but what state inside that image is unsafe to share across sessions? Think about anything that was supposed to be unique or unpredictable per session. What has to happen at restore time to make each resumed session safe?

Hint: Resumability

A client's connection drops mid-run and reconnects while the code is still producing output. How does it recover exactly the output it missed during the gap — no holes, no scrambled ordering? What would the client and the gateway each need to track to make that work, and what delivery semantics are you committing to?

Hint: Backpressure & big blobs

What happens on a while True: print() flood, or a multi-megabyte plot? Consider an output cap with an explicit truncation marker, and sideloading large rich-output blobs to object store instead of inlining them on the low-latency text stream. Also: should execution keep running if the user disconnects?

Hint: Cost lever

The idle tail is the cost killer. Suspend-to-snapshot (persist VM memory + disk, free host CPU/RAM) makes idle sessions nearly free; warm pools + post-import snapshots make resume/cold-start fast. Consider why GPU sessions probably can't be suspend-resumed the same way.

Hint: Durability classes

Don't conflate three things: workspace files (durable source of truth → object store), the runtime overlay (installed packages/caches → rebuildable cache), and live VM memory (best-effort snapshot for fast resume). Autosave the first; treat the second as disposable.

Clarifying Questions to Ask

A strong candidate scopes the problem before designing. Good questions here include:

Scale: How many registered users vs. concurrent active sessions at peak? How large is the idle/suspended tail relative to active sessions?
Workload shape: Are sessions bursty and interactive (short cells, long idle gaps), or long-running batch jobs? Which languages/runtimes must we support first?
GPU: Do we need to support GPU/accelerator sessions, and roughly what fraction of sessions?
Trust & threat model: Is all user code untrusted by default? What's the blast radius we must contain (single tenant, single host, whole fleet)?
Startup SLA: What's the acceptable cold-start latency for a fresh session (sub-second? single-digit seconds?)?
Persistence expectations: Must workspace files survive host death? Is there a free vs. paid tier with different durability/quotas?

Constraints & Assumptions

Use these to anchor the design (adjust if you prefer, but state your numbers):

Untrusted by default — assume every session may be a hostile attacker.
A session is single-user; multiple sessions belong to a workspace.
Workloads are bursty and interactive: short cells, long idle gaps; mostly Python/JS, with a minority needing a GPU.
Order-of-magnitude scale to design against: roughly 100k registered users, on the order of a few thousand concurrent active sessions at peak, plus a much larger idle/suspended tail (≈10× active).
A typical session ≈ 1–2 vCPU, 2–4 GB RAM; GPU sessions are the exception, not the norm.
Cold-start target: a fresh session should become interactive in single-digit seconds at p95.
You control a cloud fleet (bare-metal / nested-virt-capable hosts) and can run a virtualization layer.

What a Strong Answer Covers

This lists the dimensions the interviewer is grading — not the answers themselves:

Capacity estimate: active vs. idle sizing, host count, and what the idle tail implies for cost.
Substrate choice with explicit tradeoffs: isolation strength vs. cold-start vs. density across container / hardened-container / userspace-kernel / microVM / full-VM — and a clear default with justification.
The GPU exception: recognizing the default substrate may not support accelerators, and routing GPU sessions appropriately.
Layered isolation: process/kernel, filesystem, network egress, and credential/identity handling — defense in depth, no single fatal bypass.
Streaming design: low latency, ordering, reconnect/replay, backpressure, and keeping runtimes off the public internet.
Lifecycle & cold-start strategy: warm pools, suspend/resume, idle reclamation, quotas.
Data durability classes: distinguishing source-of-truth files from rebuildable runtime state.
Scheduling, autoscaling, fairness, and observability (including security-relevant audit signals).
Failure modes: explicit handling of crashes, floods, abuse, and snapshot/identity hazards.
Communication: stated assumptions, a clean API sketch, and a coherent architecture narrative.

Follow-up Questions

Be ready for deeper probes after the main design:

Scale-out: What breaks first as you go from a few thousand to 100k concurrent active sessions? Which tier is the bottleneck — scheduler, gateway, or the host fleet?
GPU isolation: If your default substrate can't attach a GPU, how do you isolate GPU sessions while still exposing the device — and how do you bound the blast radius?
Streaming failover: A streaming gateway shard dies mid-run with the client mid-stream. Exactly how does the reconnecting client recover without losing output?
Abuse at scale: A coordinated set of free-tier users runs crypto miners and fork bombs. Which controls (cgroups, egress, abuse signals) trip, and what's the automated response?

Solution

1) Clarify scope, assumptions, and rough scale

I'll design a Colab-like, multi-tenant cloud IDE where the primary constraint is safely running untrusted code. Everything else (latency, density, cost) is optimized subject to that isolation guarantee.

In scope: per-session isolated compute, secure execution of arbitrary code, near-real-time output streaming, durable workspace files, full session lifecycle. Optional: real-time collaboration (called out at the end).

Assumptions:

Workloads are bursty and interactive — short cells, long idle gaps. Mostly Python/JS; a minority need a GPU (which, as §3 explains, forces a different substrate — GPUs can't ride the default microVM).
A session is single-user; multiple sessions belong to a workspace.
Untrusted by default: assume the user is an attacker trying to escape the sandbox, exfiltrate data, mine crypto, or reach internal services.
We control a cloud fleet (bare-metal or nested-virt-capable hosts) and can run a microVM layer; GPU hosts are provisioned separately.

Back-of-envelope (to ground density/lifecycle decisions):

Target ~100k registered users, ~5k concurrent active sessions at peak, plus a long tail of idle/suspended sessions (say 10×, ~50k).
A modest session ≈ 1–2 vCPU, 2–4 GB RAM. So peak active ≈ 5–10k vCPU and 10–20 TB RAM. Sizing the fleet from both dimensions: at ~64–128 vCPU/host the vCPU side needs ~80–150 hosts; at ~256–512 GB RAM/host the memory side needs ~25–60 hosts. The vCPU side dominates, so ~80–150 CPU hosts at peak (plus headroom for fragmentation and warm pools). GPU capacity is sized separately off GPU demand.
The idle/suspended tail must cost near-zero (no running CPU/RAM), which drives the suspend-to-snapshot strategy below.
Cold-start target: a fresh session should be interactive in single-digit seconds (p95), which rules out booting a full guest OS on the request path and motivates warm pools + snapshot restore.

2) High-level architecture

Two planes: a multi-tenant control plane and a per-session data plane. The browser never talks directly to a runtime.

 Browser (Monaco editor + notebook UI + xterm)
   │  HTTPS (control)        │  WSS (output stream)
   ▼                         ▼
┌─────────────── Control Plane ───────────────┐   ┌──── Streaming Gateway ────┐
│ API Gateway / AuthN-AuthZ                    │   │ sharded WS terminators,   │
│ Workspace Service   (metadata, files index)  │   │ auth, fan-out, replay,    │
│ Session Manager     (lifecycle, quotas)      │   │ backpressure              │
│ Scheduler/Capacity  (bin-pack, autoscale)    │   └────────────┬──────────────┘
│ Policy Engine       (images, egress, limits) │                │ gRPC
└──────────────┬───────────────────────────────┘                │
               │ provision / control (gRPC)                      │
               ▼                                                  ▼
   ┌──── CPU Host (bare metal) ────┐            per-session microVM
   │ Host Agent ── Firecracker/    │   ───►     ┌──────────────────────────┐
   │   jailer; warm microVM pool   │            │ Guest kernel (locked down)│
   └───────────────────────────────┘            │ Runtime Agent (exec, fs,  │
                                                 │   log capture, health)    │
   ┌──── GPU Host (single-tenant) ─┐            │ user kernel (Jupyter-like)│
   │ Host Agent ── full VM w/ PCIe │            └──────────────────────────┘
   │   passthrough OR hardened     │
   │   NVIDIA-stack container       │
   └───────────────────────────────┘
   Object store (workspace files, snapshots, large rich-output payloads)
   Log store (durable output + audit)

Control plane services are stateless (any instance serves any request), backed by a metadata DB (Postgres) and a queue for async provisioning.
Data plane is two host fleets: CPU hosts running a warm pool of microVMs, and GPU hosts running the GPU substrate (see §3). Each host runs a Host Agent that manages its runtimes and launches/snapshots them.
Streaming Gateway is a separate, stateful tier (it holds per-session replay buffers and routing) — see §8 for how it's sharded.
Runtime Agent runs inside each runtime as the only privileged-ish process the platform controls; it execs user code, captures output, serves file ops, and reports health. The same agent runs in both the CPU and GPU substrates.

3) Compute substrate: VM vs container vs gVisor vs microVM

The core decision. For untrusted code, a shared host kernel is the attack surface — a single kernel CVE means a full host compromise across tenants. That pushes toward kernel-level isolation.

Option	Isolation	Cold start	Density	Notes
Plain container (namespaces + cgroups)	Weak — shared host kernel, huge syscall surface	~ms	Highest	Unacceptable alone for untrusted code; one kernel escape = host-wide breach
Hardened container (seccomp + AppArmor + dropped caps + read-only rootfs)	Better, still shared kernel	~ms	High	Reduces surface but doesn't eliminate the shared-kernel risk
gVisor (userspace kernel intercepts syscalls)	Strong-ish — sandboxed kernel, no real VM	tens to hundreds of ms	High	Great middle ground; some syscall-compat gaps and CPU overhead
microVM (Firecracker / Cloud Hypervisor)	Strong — hardware-virtualized, own guest kernel, tiny VMM device surface	~100 ms boot, faster via snapshot	High (low per-VM overhead)	Default choice for CPU-only untrusted execution
Full VM (QEMU/cloud image)	Strongest, but heavy device model	seconds	Low	Heavy/slow for the common case, but the only path that supports GPU PCIe passthrough

Default (CPU) substrate: microVMs. Firecracker gives near-VM isolation with a minimal device model (virtio-net, virtio-block, a serial console — no BIOS, no PCI), ~5 MB of memory overhead per VM, and ~125 ms boot. A common production pattern is Kata Containers using Firecracker (or Cloud Hypervisor) as the VMM so each "pod" is actually a microVM while keeping a Kubernetes-native control surface — a reasonable way to manage the fleet. gVisor is a good secondary tier for cheaper/short-lived "scratch" runs where the perf overhead is acceptable.

GPU substrate (explicit, separate path). Firecracker's minimal device model is exactly why it cannot host a GPU: it has no PCI bus and no PCIe/VFIO passthrough, so a GPU device cannot be attached to a Firecracker microVM. GPU sessions therefore run on a different substrate, on a separate pool of single-tenant GPU hosts:

Preferred: a full VM (QEMU/Cloud Hypervisor) with VFIO PCIe passthrough of a dedicated GPU (or a MIG slice). This keeps the per-tenant-kernel isolation property while exposing the device; it pays a heavier boot/device-model cost, mitigated with a small warm pool of pre-attached GPU VMs.
Alternative where passthrough VMs aren't available: a hardened container with the NVIDIA container stack (seccomp + AppArmor + dropped caps + read-only rootfs) on a host that is never co-tenanted with another customer for the GPU's lifetime — accepting the weaker (shared-kernel) isolation explicitly, and confining the blast radius by single-tenanting the whole host.
Either way, GPU hosts are physically separate from the CPU microVM fleet, scheduled from their own pool (§11), and a GPU session never shares a host with another tenant.

So the rule is: microVM by default; GPU sessions are routed to the GPU pool and run as passthrough full-VMs (or single-tenant hardened containers). The substrate is chosen per session from the requested gpu count.

Operationally (CPU fleet):

Run Firecracker under jailer (chroot, dedicated uid/gid, cgroup, separate network namespace) so even a VMM compromise is contained on the host.
Hosts are bare-metal (or nested-virt enabled) since Firecracker needs KVM.
One runtime per session, never shared across tenants — on both fleets.

4) Isolation model (filesystem, process, network, credentials)

This is the security core; isolation is layered so no single bypass is fatal. The controls below apply to both substrates; the GPU substrate additionally relies on single-tenant host placement to compensate for its weaker (when container-based) kernel isolation.

Process / kernel: each session = its own guest kernel (microVM, or passthrough full VM on GPU). Inside, the user process runs unprivileged with seccomp (deny dangerous syscalls), dropped Linux capabilities, and cgroups for CPU/mem/PID/IO limits. The host never trusts the guest.

Filesystem:

Guest rootfs is a read-only base image (language toolchain + agent) plus a writable overlay for the user's workspace and installed packages.
Disk quota on the overlay (per-tier cap) to stop a user filling the host.
No host paths are bind-mounted into the guest beyond the controlled block device.

Network (most-attacked path):

Each runtime gets its own network namespace + veth pair + NAT; no L2 adjacency between tenants.
Default-deny egress. Outbound goes through an egress proxy that allowlists package repos (PyPI, npm, conda, apt mirrors). Everything else is dropped.
Block the cloud metadata endpoint (169.254.169.254 and the link-local range) at the host firewall — this is the classic SSRF→credential-theft path on cloud providers. As defense-in-depth, also enforce IMDSv2 (token-required, session-oriented, with a hop limit of 1) on every host so that even if the firewall block regressed, a guest can't trivially reach the metadata service via SSRF.
No inbound from the internet; the runtime is reachable only by the control plane and streaming gateway over an internal network.

Credentials / identity:

The guest holds no node-level or cloud credentials. The Host Agent never injects long-lived secrets.
If a session needs to reach an internal resource (e.g., the user's object-store prefix), it uses a short-lived, narrowly-scoped token minted per session, audited, and expiring on idle/terminate.
Snapshots restored from a warm pool must re-seed entropy and per-session identity so two sessions don't start from identical RNG state or share a token.

Supply chain: base images are signed and scanned; package installs flow through the proxy where they can be scanned/cached; abusive runtime behavior (sustained max CPU/GPU, known crypto-miner signatures, mass-egress attempts) is detected and the session throttled or killed.

5) Session lifecycle

States and transitions:

Provisioning ──► Running ──► Idle ──► Suspended ──► Terminated
                   ▲           │          │
                   └───────────┴──────────┘  (resume)

Provisioning: Session Manager admits the request (quota check), Scheduler picks a host from the right pool (CPU vs GPU, per the requested gpu count), Host Agent claims a warm runtime and attaches the workspace overlay. Because the runtime is pre-booted (or snapshot-restored), this is sub-second to a few seconds rather than a full boot. (GPU passthrough VMs warm-pool more coarsely since they're heavier and scarcer.)
Running: Runtime Agent serves exec/fs; output streams out.
Idle: no active client WS and low CPU for N minutes → mark Idle (still resident, but a candidate for suspend).
Suspend (key cost lever, CPU substrate): snapshot the microVM (memory + disk state) to the host/object store and free the host's CPU and RAM. This is what makes the 50k-idle tail cheap. Firecracker's snapshot/restore persists full VM memory; resume restores it. Note GPU passthrough VMs don't snapshot-resume cleanly (device state isn't captured), so idle GPU sessions are instead checkpointed to the workspace and torn down — GPU is too expensive to leave resident anyway.
Resume (CPU substrate): restore the snapshot (fast — memory is loaded back, no re-boot, no re-import), re-attach networking with fresh credentials/entropy. Resume from a local snapshot is much faster than from object store, so keep recent snapshots warm-local and tier older ones off.
Terminate: flush final files/logs, destroy the runtime, release the overlay (after a durable checkpoint), revoke tokens.

Cold-start strategy: maintain warm pools of pre-booted base microVMs per language/image so the request path is "claim + attach", not "boot". Pool size is autoscaled off arrival rate. For popular notebook stacks, keep post-import snapshots (kernel already started, heavy libs imported) so the user's first cell is instant.

Quotas / limits: per-user/org caps on concurrent sessions, vCPU, RAM, GPU count, and wall-clock for free tier; hard idle timeouts to reclaim resources.

6) Data persistence

Three distinct durability classes — conflating them is the common mistake:

Class	Where	Durability	Resume cost
Workspace files (code, notebooks) — source of truth	Versioned object store (optionally git-backed)	Durable, survives session death	n/a — fetched on attach
Runtime overlay (installed pkgs, caches)	Local ephemeral disk + periodic snapshot	Best-effort; rebuildable	Restore snapshot (fast) or re-install (slow)
Live VM memory (CPU suspend/resume)	Firecracker snapshot, local then tiered	Best-effort	Restore = near-instant interactivity

Strategy:

Object store is the durable home of workspace files. The Runtime Agent autosaves notebooks/files and checkpoints to object storage on a timer and on suspend/terminate. So even if a host dies, the user loses at most the last few seconds of unsaved edits — never their project.
The overlay (packages, caches) is treated as a cache: persisted via snapshot for fast resume, but reconstructable from the workspace's environment spec if lost.
Paid tier can get a persistent network volume mounted per workspace for large datasets, accepting the throughput/contention tradeoff vs. cheap object-store + local-disk for free tier.

Tradeoff to state explicitly: network volumes make resume trivial but can bottleneck and add per-IOPS cost; snapshot-to-object-store is cheaper at steady state but adds resume latency. Default to object-store + local ephemeral + snapshots; offer volumes as an upgrade.

7) Execution model

Inside each runtime, the Runtime Agent is the platform's trusted entrypoint. It supervises a language kernel (a Jupyter-style kernel protocol internally is a clean fit — execute requests, stream results, interrupt) but the design stays kernel-agnostic. It exposes an internal (control-plane-only) API:

POST /execute — run a cell/command; returns an execution_id, streams output via the agent→gateway channel.
POST /interrupt — SIGINT the running cell (Ctrl-C).
GET /health — kernel + resource status (for idle detection and liveness).
GET /fs/{path} / PUT /fs/{path} / DELETE /fs/{path} — file ops (upload/download, list).

Output is captured as a typed stream so the UI can render rich output: stdout, stderr, structured events (cell_started, cell_finished, exit_code), and MIME-typed payloads (text, image/png, HTML) for plots and tables.

Rich-output sizing (don't inline large blobs). Plots/tables can be large, so they're not shoved raw onto the same low-latency text stream. The agent applies a per-payload threshold (e.g., ~256 KB): below it, the MIME payload rides the stream inline; above it, the agent writes the blob to object store and emits a small reference event ({mime, object_key, size, seq}) that the browser fetches out-of-band. This keeps a single large figure from blowing the streaming buffer or stalling stdout, and is also why the truncation logic in §8 is framed around the text/event stream, not multi-megabyte images.

8) Log / output streaming (near-real-time, resumable)

Goal: low-latency stdout/stderr + structured events to the browser, survivable across reconnects, with bounded buffering.

Path:

Runtime Agent captures kernel output into a local bounded ring buffer, tagging each chunk with (execution_id, stream, seq) where seq is a monotonic per-stream sequence number.
Agent pushes chunks to the Streaming Gateway over gRPC (internal network only — runtimes are never exposed to the internet).
Gateway authenticates the browser's WSS connection (scoped session token), then fans out chunks to the client. It also asynchronously persists output to a durable log store (debugging + audit), decoupled from the live path so log-store latency never stalls the user.
Browser renders incrementally; ack'd seq lets the gateway advance.

Gateway scaling, state, and reconnect routing. The gateway is the one stateful tier (it holds each session's replay buffer and the agent↔client routing), so the "stateless control plane" claim in §2 explicitly does not cover it. It scales horizontally with consistent-hash sharding by session_id: a session's agent stream and its browser WSS both map to the same gateway shard, so fan-out and replay are local to one instance. The browser is given the shard-stable endpoint (via the session record / a routing layer that hashes session_id), so a reconnecting client lands back on the same shard that holds its ring buffer. If that shard has restarted or the buffer was evicted, the gateway rehydrates from the durable log store using last_seq before resuming live fan-out — so a shard failover degrades to a slightly slower replay, never lost output. At ~5k concurrent WSS this fits comfortably in a handful of shards; the shard ring is the autoscaling unit.

Delivery semantics & robustness:

Reconnect/replay: on reconnect the client sends last_seq; the gateway replays missing chunks from its buffer (or the durable log if evicted). Delivery is at-least-once, idempotent on the client via seq dedup — the user never silently loses output mid-run.
Backpressure: if the client is slow or gone, buffer up to a cap (e.g., a few MB / N seconds) of text+events, then drop the oldest with an explicit "... output truncated ..." marker rather than OOMing the gateway or stalling execution. Execution keeps running. (Large rich-output blobs don't count against this cap — they're sideloaded per §7.)
Output floods: the agent rate-limits/caps total output per execution (a while True: print() loop must not DOS the pipeline) and emits a truncation event.
Multiplexing: one WSS per session with logical channels (stdout, stderr, events) so the client demuxes cleanly.
Decoupling running from watching: if the user disconnects, the cell keeps running (policy choice — matches Colab-style behavior); output buffers and the idle timer governs reclamation.

A gateway (vs. direct runtime→browser) centralizes auth, rate limiting, TLS termination, protocol translation, and keeps runtimes off the public internet.

9) Data model (control plane)

workspace(id, owner_id, org_id, name, env_spec, default_image, created_at, last_active_at)
session(id, workspace_id, host_id, runtime_id, substrate /*microvm|gpu_vm|gpu_container*/,
        state, image, cpu, mem, gpu, gateway_shard,
        token_id, created_at, idle_since, expires_at)
snapshot(id, session_id, kind /*disk|mem*/, location, size_bytes, created_at)
file_index(workspace_id, path, object_key, version, size, checksum, updated_at)
quota(subject_id /*user|org*/, max_concurrent, max_vcpu, max_mem, max_gpu, tier)
audit_event(id, actor, action, session_id, image, net_policy, ts)

(substrate records which compute path served the session; gateway_shard lets a reconnecting client be routed back to the gateway instance holding its buffer.)

10) API sketch (control plane + streaming)

# Workspaces & files
POST   /v1/workspaces                         -> create workspace
GET    /v1/workspaces/{id}/files              -> list files
PUT    /v1/workspaces/{id}/files/{path}       -> upload
GET    /v1/workspaces/{id}/files/{path}       -> download

# Sessions (lifecycle)
POST   /v1/workspaces/{id}/sessions           -> create/attach runtime {cpu,mem,gpu,image}
                                                  (gpu>0 routes to the GPU substrate/pool)
GET    /v1/sessions/{sid}                      -> state + connection info (incl. gateway endpoint)
POST   /v1/sessions/{sid}:suspend             -> snapshot + free host resources (CPU substrate)
POST   /v1/sessions/{sid}:resume              -> restore snapshot
POST   /v1/sessions/{sid}:terminate           -> checkpoint + destroy

# Execution (proxied through gateway to runtime agent)
POST   /v1/sessions/{sid}/execute   {code}    -> {execution_id}
POST   /v1/sessions/{sid}/interrupt

# Output stream (WebSocket; endpoint is shard-stable for this session)
GET    wss://gw/v1/sessions/{sid}/stream?token=...&last_seq=...
        server frames: {type: stdout|stderr|event, execution_id, seq, mime, data}
        large rich-output frames carry {object_key,size} instead of inline data

11) Scheduling, autoscaling, and fairness

Admission control in Session Manager: reject/queue at quota boundaries before any host work.
Substrate routing: the requested gpu count selects the pool. gpu == 0 → CPU microVM fleet; gpu > 0 → GPU host pool (passthrough VM or single-tenant hardened container). This is decided at admission so a GPU request never lands on a Firecracker host that can't satisfy it.
Scheduler bin-packs sessions onto hosts by (cpu, mem, gpu) with separate pools — CPU-only, GPU (single-tenant), high-mem — so a GPU session doesn't strand CPU capacity and vice versa. Prefer hosts with a warm runtime of the requested image (locality cuts cold start).
Autoscaling: scale each host pool off its own active+queued demand and warm-pool depletion. Suspended CPU sessions consume no host CPU/RAM (only snapshot storage), so the CPU fleet scales with active load, not registered users. GPU capacity is scaled and budgeted independently since idle GPU sessions are torn down rather than suspended.
Fairness: weighted fair sharing per org; for free tier, preempt-to-suspend the longest-idle CPU sessions when capacity is tight (their state is snapshotted, so preemption is non-destructive). GPU is gated by hard quotas rather than oversubscribed. Hard wall-clock and idle caps on free tier.

12) Observability

Metrics (SLIs): session-start latency p50/p95 (warm vs. cold, per substrate), warm-pool hit rate, resume latency, streaming lag and dropped-chunk rate, running/idle/suspended counts, CPU vs GPU host utilization, noisy-neighbor / preemption events, snapshot size & age, gateway shard fan-out and rehydration rate.

Tracing: end-to-end span on create session → admit → choose substrate → schedule → claim runtime → attach overlay → agent-ready, so cold-start regressions are attributable to a stage.

Audit logs (security-critical): who started which session with which image, substrate, and network policy; denied egress attempts, metadata-endpoint hits, token mints/uses, suspicious-syscall signals. These feed abuse detection and incident response.

13) Failure modes & edge cases

Cold start too slow → warm pools + post-import snapshots + small base images + image caching on hosts (GPU warm pools are smaller and coarser due to cost).
GPU requested but no Firecracker support → handled at admission: gpu > 0 is routed to the GPU substrate/pool (passthrough VM or single-tenant container), never to a microVM.
Host dies mid-session → workspace files are already durable in object store (autosave + checkpoint); user reconnects and a new session re-attaches the workspace. In-flight unsaved output is best-effort from the durable log.
Gateway shard dies / buffer evicted → reconnecting client is re-routed to the (restarted or replacement) shard for its session_id, which rehydrates the replay buffer from the durable log using last_seq before resuming live fan-out.
User installs huge deps / fills disk → overlay disk quota + cached package layers; OOM-kill the cell, not the host.
Infinite output / fork bombs / crypto-mining → output caps, cgroup CPU/PID limits, egress denial + abuse signals → throttle or terminate.
Disconnected client → execution continues; idle timer governs suspend; output replayable on reconnect.
Snapshot reuse leaking entropy/identity → re-seed RNG and mint fresh per-session credentials on every restore.
Secrets leakage → never mount broad cloud credentials; scoped per-session tokens only; metadata endpoint blocked and IMDSv2-enforced.

14) Optional: collaboration

Separate the two concerns:

Document state (who's typing what) → a CRDT (or OT) sync service over WebSocket, independent of the runtime. This scales to many viewers/editors cheaply.
Execution → keep a single owner kernel per notebook (one runtime) to avoid contention and permission ambiguity; collaborators see streamed output via the same gateway fan-out but exec requests are owner-gated (or queued). Trying to share one runtime symmetrically across users creates resource-contention and trust problems, so the clean model is "shared editing, single-owner execution," with per-user runtimes if true parallel execution is needed.

Summary: microVMs (Firecracker, run under jailer) as the default CPU untrusted-execution substrate, with GPU sessions routed to a separate single-tenant pool running full VMs with PCIe passthrough (or hardened NVIDIA-stack containers) because Firecracker has no PCI bus; layered isolation (own guest kernel, read-only base + quota'd overlay, default-deny egress with the metadata endpoint blocked and IMDSv2-enforced, no node credentials + short-lived scoped tokens); warm pools + snapshot/restore for sub-second-to-few-second start and near-zero-cost idle on the CPU fleet; object-store as the durable home for workspace files (with large rich-output blobs sideloaded there too); and a resumable, backpressure-aware, sequence-numbered streaming pipeline through a session-sharded internal gateway so output is real-time, lossless-on-reconnect, and the runtimes never face the internet.

Community answers

Nothing here yet.

Join in on CSWork →