Ray Serve Interview Questions: Deployments, Request Queues, and Replica Failures

A Ray Serve interview question becomes useful when a request has nowhere to go: every replica is busy, one caller's queue is full, and a worker dies after starting an external operation. A strong answer identifies where the request waits, which limit applies, and what evidence survives the failure. “Add replicas and retry” leaves all three unresolved.

This guide traces deployment handles, calculates a queue example and separates replica recovery from business recovery. Start with Review a Batched GPU Inference API Design Doc: Flaws, Throughput, Mixed-Model GPUs, then add the explicit Serve limits below.

Platform statements come from official Ray documentation, checked October 4, 2026; the current latest pages identify Ray 2.59.0. The numerical ledgers and failure cases are original exercises.

Ray Serve interview questions: deployment, replica or application?

The official key concepts distinguish a deployment's logic from its running replicas, which are Ray actors. An application binds one or more deployments together, with an ingress deployment receiving inbound traffic. A DeploymentHandle calls a deployment at runtime; it does not pin the caller to one particular replica.

Use those nouns precisely. Two deployments can scale independently within one application. Two replicas of a deployment execute copies of the same class, but their ordinary instance fields are separate. An in-memory counter inside one replica is therefore not automatically a deployment-wide counter.

The composition guide shows .bind() building an application and downstream deployments being passed as handles. Calling .remote() produces a deployment response that an async caller can await. Describe the dependency graph before choosing how many replicas each stage needs.

An original interview example has Ingress → Embedder → Classifier. If embedding saturates while classification remains idle, tripling classifier replicas addresses the wrong stage. Ask for stage-level queue delay, processing time and resource availability. “The app is slow” does not identify the constrained deployment.

Trace the request without putting the controller on the hot path

The architecture documentation separates the controller's lifecycle responsibilities from proxies, routing and replicas. For HTTP traffic, a proxy receives the request and routes it toward the relevant deployment. A direct handle call enters through the handle's routing path instead.

Explain an original six-event trace: accepted at the edge, admitted by the caller, waiting for a replica, dispatched, executing user code, and response delivered. Those events are diagnostic markers, not six durable states supplied by Serve. Some require application instrumentation.

For request R17, suppose the proxy logs receipt but no replica logs handler entry. Investigate routing availability and caller queueing before profiling model inference. Conversely, handler entry followed by a slow downstream call points beyond admission. A response generated by a replica still does not prove that the client received it.

Keep request identity, deployment name, model version and attempt number together. If a retry uses a new transport request ID, retain the same business operation ID where deduplication is required. This lets you distinguish one user operation from multiple attempts and avoid counting retries as fresh demand.

What do max_ongoing_requests and max_queued_requests limit?

According to production load-shedding guidance, max_ongoing_requests controls how many requests may be outstanding at a replica. max_queued_requests bounds the requests waiting in each caller's DeploymentHandle, including the proxy. It is not one global waiting-room size shared by every caller.

An ongoing request is not necessarily consuming a CPU core or running a GPU kernel at that instant. It can be awaiting I/O or otherwise admitted to the replica. Likewise, a queue limit counts requests, not their bytes, tokens or expected processing costs.

This distinction changes the capacity answer. Adding calling replicas or proxies can add independent queues. Setting a small per-handle limit helps bound that local backlog, but does not by itself establish a global memory budget, tenant fairness or maximum number of outstanding operations throughout a composition.

The Serve API reference identifies BackPressureError as the handle queue-overflow exception. State where you will catch it and what your caller does next. A catch that immediately retries forever converts rejection into more demand.

Work the two-caller queue ledger

Assume this original, deliberately simplified operating point: one downstream deployment has two ready replicas; each allows two ongoing requests; callers H1 and H2 each have a queue limit of three. No other callers exist, no request completes during the snapshot, and routing has already filled the available replica slots.

The replica side holds up to 2 × 2 = 4 ongoing requests. The two caller queues together can hold 2 × 3 = 6 waiting requests. A snapshot with four ongoing and six waiting contains ten accepted requests. That arithmetic is a model of the stated snapshot, not a universal admission guarantee during routing races, startup or failure.

Original snapshot	Replica-side ongoing	H1 waiting	H2 waiting	Interpretation
Both replicas saturated	4	3	0	H1 is full; H2 still has local room
Both callers saturated	4	3	3	Another waiting request from either caller exceeds its queue limit
A third caller appears	4	3	3	Its independent queue changes aggregate backlog; the old six-request bound no longer applies

The first row is the useful counterexample: H1 may reject a new request while H2 can enqueue one. There is no unused global queue slot that H1 is automatically entitled to borrow.

Now change one assumption: a replica fails. Do not subtract two from the ongoing column and label those requests successful. Their outcomes require failure handling, while surviving capacity and replacement startup affect new admissions. Maintain a separate record for interrupted attempts.

Does a concurrency limit guarantee parallel execution?

No. The asyncio best-practices guide distinguishes async handlers, blocking operations, threadpool configuration and FastAPI dispatch. An async def function that calls a synchronous HTTP client can still block its event loop. Increasing its admission limit can grow contention without increasing useful throughput.

The current guide also distinguishes pure Serve synchronous methods from FastAPI synchronous endpoints. Check the installed Ray version and relevant execution configuration before asserting that every synchronous handler runs in a threadpool, or that all such handlers execute serially under every configuration.

In an original measurement exercise, compare queue wait, user-code duration and completed requests per second at concurrency limits two, four and eight. Hold input sizes and hardware constant. If admitted concurrency doubles but completions remain unchanged while tail latency rises, the larger limit has not created processing capacity.

For CPU-heavy Python, async scheduling alone does not create multicore speedup. For GPU work, model batching, device memory and kernel execution impose separate constraints. Keep the answer focused on the observed bottleneck rather than declaring “async” a performance solution.

How should overload, retries and deadlines interact?

The production guidance documents HTTP 503 as the default backpressure response and a configurable HTTP 429 response with optional Retry-After in current versions. Rejection should be visible in your metrics even if it moves out of the 5xx bucket. An HTTP code alone may not distinguish overload from deployment unavailability.

An original latency budget gives a request 300ms end to end: 40ms for validation and transport, 60ms for queueing, 150ms for execution and 50ms of margin. These are chosen exercise numbers, not measured Ray performance. A request that has already waited 200ms cannot sensibly begin a fresh 150ms attempt while claiming to honor that budget.

Give retries a maximum attempt count, backoff with jitter and the remaining overall deadline. Coordinate retry layers: a client, gateway and downstream caller can multiply attempts if each independently retries. Record the original operation and its attempts so that dashboards expose that amplification.

Prefer a bounded rejection over accepting unlimited work that cannot meet its deadline. Explain whether an interactive request should fail promptly, degrade to a cheaper model, or become a durable asynchronous job. Those are product contracts; a DeploymentHandle queue is not an application job ledger.

What survives a replica failure?

The fault-tolerance guide describes replica health checks and restarting unhealthy replicas. A downstream deployment recovers independently; the parent deployment need not restart merely because a child failed. Replacement actors restore serving capacity, not proof of completion for every interrupted business operation.

The architecture guide also notes that transient router/replica state, including internal queues and connections, can be lost during machine failure. Separate that from persisted control-plane configuration. “Serve restarts the replica” does not imply that the original queue is a durable broker.

Consider an original scoring service that also records a billable usage event. Replica A sends the event to an external database, the database commits, and A dies before returning its response. The caller sees failure but cannot infer that no effect occurred. Retrying on replica B needs a stable operation key and an external deduplication contract.

For read-only deterministic inference, repeating computation may be acceptable, although it still costs capacity. For sampled generation, a retry might produce a different output. For streaming, an already delivered prefix requires an explicit resume or restart policy; actor replacement alone cannot erase bytes the client has seen.

Does cancellation mean that no work happened?

The HTTP cancellation documentation describes dropping requests not yet dispatched, attempting cancellation of dispatched handlers and propagating cancellation downstream. Async cancellation is cooperative: a blocking segment without a subsequent await may continue before the response is discarded.

An original case has a client disconnect after an external write completes but before the handler's next await. Cancellation cannot retroactively roll back that write. Keep cleanup, operation outcome and transport outcome separate. Handle cancellation at the component that owns a resource, and do not swallow it indiscriminately to continue expensive unwanted work.

Your failure drill should therefore cancel while queued, cancel during an await, kill a replica before an effect, and kill it after a committed effect but before acknowledgment. Specify the expected client result and external state for each case. This drill is a proposed practice experiment, not a claim that it ran against a Ray cluster here.

Which evidence would justify adding replicas?

Use Serve monitoring to inspect deployment/replica status and application metrics. Pair rejection counts and queueing with user-code duration, resource saturation and constructor failures. A desired replica count does not prove that those replicas are ready and serving.

The autoscaling guide describes decisions based on ongoing load and queued demand. In an original incident, queues grow because model loading takes twenty seconds and the burst lasts five. Reactive scaling may arrive after the burst; investigate warm capacity and startup behavior before merely raising the maximum.

Add replicas when processing capacity is constrained and resources exist to host them. Repair a slow shared database, blocked event loop or broken constructor when that is the actual constraint. Explain what result would falsify your diagnosis.

Sources and Further Reading

• Ray Serve: Key concepts
• Ray Serve: Architecture
• Ray Serve: Model composition
• Ray Serve: Production best practices
• Ray Serve: API reference
• Ray Serve: Asyncio and concurrency
• Ray Serve: Fault tolerance
• Ray Serve: HTTP and cancellation
• Ray Serve: Monitoring
• Ray Serve: Autoscaling

Documentation checked October 4, 2026. Verify options and execution behavior against the Ray version installed in your environment.