/Learn
Modules · Infrastructure · System design

Ray Serve Distributed Actor Deployment

PracticeSystem designTier 3 · Edge casesCreate35 minInfrastructure

You will be able to: design a Ray Serve deployment with autoscaling, health checks and safe rollouts.

Source: Ray Serve documentation

AI engineeringModules

Study with AI

Part 1 of 5

Basics

The whole idea first, then the terms the rest of this page uses.

1.1Whole idea

Ray Serve runs each replica as a long-lived actor holding a model on a GPU. A proxy routes requests, and the controller adds or removes replicas based on queued work. Your code stays Python, while scaling and health handling become configuration.

Why it matters

Runs LLM replicas as autoscaled actors, with health checks and per-replica request limits.

1.2Key concepts

One idea per card, in plain words.

01Deployment

A class plus its replica count, resources and autoscaling rules.

02Replica

An actor process with its own model in GPU memory.

03Ongoing requests

Autoscaling's main signal: the requests a replica is handling or holding.

04Application

Deployments bound together and upgraded as one unit.

Part 2 of 5

Question

What exactly you must answer, and what to settle before you start.

2.1Question

Interviewer

Serve an 8B model on 2–20 GPUs, with traffic that triples at lunch. Model load takes 90 seconds. Design scaling, health checks and upgrades.

2.2Clarify first

Each answer removes a guess.

You askAnswer
Cold start tolerance?Keep latency under target during spikes, so keep warm spares.
Spot instances?Yes for overflow, with on-demand for the baseline.
Part 3 of 5

Detailed answer

How it works, and how to build or explain it.

3.1Architecture

The components and how they connect.

Load balancerHTTP entryServe proxyroutes to replicasControllerautoscaling · healthReplicavLLM on 1 GPUReplicavLLM on 1 GPUReplicascaling upMetricsqueue · TTFT · errorsrequeststartsmetricssignal
ServiceData storeAI componentExternal
Boxes are components; arrows show calls and data flow.

3.2Step by step

The mechanism, in the order you would explain it.

  1. Pick the concurrency

    Load test one replica to find how many requests it holds within your latency target. Set target_ongoing_requests from that.

  2. Keep warm capacity

    A 90-second load means scaling always lags, so min_replicas must cover normal peaks.

  3. Scale up fast, down slowly

    Set a short upscale delay and a long downscale delay, because reloading costs more than idling.

  4. Check real health

    Run a one-token generation in check_health. A live process with a dead GPU must still fail.

  5. Roll out safely

    Serve starts new replicas before stopping old ones. Canary with a separate application, then shift traffic.

3.3Code

Working code for the core of it.

Serving with Ray Serve

This scales replicas with load and restarts unhealthy ones.

ray_serve_app.pyPython
1from ray import serve2from starlette.requests import Request3from vllm import LLM, SamplingParams4 5 6@serve.deployment(7    ray_actor_options={"num_gpus": 1},8    autoscaling_config={"min_replicas": 2, "max_replicas": 20, "target_ongoing_requests": 16},9    max_ongoing_requests=32,10    health_check_period_s=10,11)12class Generator:13    def __init__(self, model: str):14        self.llm = LLM(model=model)15 16    def check_health(self) -> None:17        self.llm.generate(["ping"], SamplingParams(max_tokens=1))18 19    async def __call__(self, request: Request) -> dict:20        body = await request.json()21        output = self.llm.generate([body["prompt"]], SamplingParams(max_tokens=body.get("max_tokens", 256)))22        return {"text": output[0].outputs[0].text}23 24 25app = Generator.bind(model="meta-llama/Llama-3.1-8B-Instruct")
25 linesUTF-8Python
target_ongoing_requests
Ray adds replicas when each one holds more than this.
max_ongoing_requests
Back-pressure: beyond this, requests queue rather than run out of GPU memory.
check_health
Generates one token, which proves the GPU works, and not merely the process.

Syntax-checked. The GPU libraries it needs don't run in our tests.

Part 4 of 5

Q&A

What interviewers ask next. Answer out loud, then open.

4.1Interview questions

Why scale on ongoing requests rather than GPU utilisation?
GPU utilisation sits high under any load. Queue depth reflects what users actually feel.
How do you handle cold starts?
Pre-pull weights to fast local disk, keep warm spares and reduce model load time.
Ray Serve or Kubernetes alone?
Kubernetes scales pods but knows little about requests. Ray adds request-aware routing and scaling, and many teams run Ray on Kubernetes.
Part 5 of 5

Extra depth

Edge cases, counterpoints and what to read next.

5.1Edge cases and counterpoints

Includes the strongest argument against this approach.

Show detailsHide details
Multiplexing
Load several adapters on one replica, and route by model id to reduce cold loads.
Counterpoint
A fixed fleet behind a simple load balancer stays simpler when traffic barely varies.
AI Engineer · Ray Serve Distributed Actor Deployment