Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
Ray Serve runs each replica as a long-lived actor holding a model on a GPU. A proxy routes requests, and the controller adds or removes replicas based on queued work. Your code stays Python, while scaling and health handling become configuration.
Runs LLM replicas as autoscaled actors, with health checks and per-replica request limits.
1.2Key concepts
One idea per card, in plain words.
A class plus its replica count, resources and autoscaling rules.
An actor process with its own model in GPU memory.
Autoscaling's main signal: the requests a replica is handling or holding.
Deployments bound together and upgraded as one unit.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Serve an 8B model on 2–20 GPUs, with traffic that triples at lunch. Model load takes 90 seconds. Design scaling, health checks and upgrades.
2.2Clarify first
Each answer removes a guess.
| You ask | Answer |
|---|---|
| Cold start tolerance? | Keep latency under target during spikes, so keep warm spares. |
| Spot instances? | Yes for overflow, with on-demand for the baseline. |
Detailed answer
How it works, and how to build or explain it.
3.1Architecture
The components and how they connect.
3.2Step by step
The mechanism, in the order you would explain it.
- Pick the concurrency
Load test one replica to find how many requests it holds within your latency target. Set target_ongoing_requests from that.
- Keep warm capacity
A 90-second load means scaling always lags, so min_replicas must cover normal peaks.
- Scale up fast, down slowly
Set a short upscale delay and a long downscale delay, because reloading costs more than idling.
- Check real health
Run a one-token generation in check_health. A live process with a dead GPU must still fail.
- Roll out safely
Serve starts new replicas before stopping old ones. Canary with a separate application, then shift traffic.
3.3Code
Working code for the core of it.
Serving with Ray Serve
This scales replicas with load and restarts unhealthy ones.
1from ray import serve2from starlette.requests import Request3from vllm import LLM, SamplingParams4 5 6@serve.deployment(7 ray_actor_options={"num_gpus": 1},8 autoscaling_config={"min_replicas": 2, "max_replicas": 20, "target_ongoing_requests": 16},9 max_ongoing_requests=32,10 health_check_period_s=10,11)12class Generator:13 def __init__(self, model: str):14 self.llm = LLM(model=model)15 16 def check_health(self) -> None:17 self.llm.generate(["ping"], SamplingParams(max_tokens=1))18 19 async def __call__(self, request: Request) -> dict:20 body = await request.json()21 output = self.llm.generate([body["prompt"]], SamplingParams(max_tokens=body.get("max_tokens", 256)))22 return {"text": output[0].outputs[0].text}23 24 25app = Generator.bind(model="meta-llama/Llama-3.1-8B-Instruct")target_ongoing_requests- Ray adds replicas when each one holds more than this.
max_ongoing_requests- Back-pressure: beyond this, requests queue rather than run out of GPU memory.
check_health- Generates one token, which proves the GPU works, and not merely the process.
Syntax-checked. The GPU libraries it needs don't run in our tests.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why scale on ongoing requests rather than GPU utilisation?
How do you handle cold starts?
Ray Serve or Kubernetes alone?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Multiplexing
- Load several adapters on one replica, and route by model id to reduce cold loads.
- Counterpoint
- A fixed fleet behind a simple load balancer stays simpler when traffic barely varies.