/Interview
Interview room · AI system design · 45 minutes

Serving an open LLM on your own GPU fleet

PracticeSystem designTier 3 · Edge casesCreate45 minModel serving

You will be able to: design serving an open LLM on your own GPU fleet in 45 minutes, and defend each trade-off.

AI engineeringDesign rounds

Study with AI

Ready45:00
ClarifyEstimateHigh-level designDeep diveTrade-offs and wrap-up
Part 1 of 5

Basics

Start with the product and the ideas it depends on. If these are clear, the design that follows will feel obvious rather than memorised.

1.1Product in plain words

Who uses it, what a normal day looks like, and why it is harder than it seems.

Running an open-source language model on the company’s own GPUs instead of paying an AI provider per request, for cost and data-residency reasons. It serves the assistant and internal tools at about 200 requests per second at peak.

Who uses it

  • Product teams: call one internal API and expect fast, reliable streaming.
  • The platform team: operates GPUs, models and capacity.
  • Finance: wants cost per request lower than hosted APIs.

A day in the life

  1. An assistant request arrives at the internal gateway.
  2. The router sends it to the replica with the most free memory.
  3. The GPU reads the prompt and starts streaming words back within a second.
  4. Many other requests share the same GPUs at the same moment through batching.
  5. When traffic rises at lunchtime, a warm spare replica joins before users notice.

Why it is hard

The model is huge

70 billion parameters do not fit on one GPU at full precision.

Memory, not compute, runs out

Each active conversation holds a cache in GPU memory.

Slow cold starts

Loading a model onto GPUs takes minutes, so scaling must be proactive.

Expensive hardware

Idle GPUs burn money; overloaded ones hurt users.

Think of it like this

Like a busy restaurant kitchen where each chef cooks many orders at once, and the limit is not the chefs but counter space for orders in progress.

1.2Key concepts

The building blocks this design uses. Each one appears again in the answer.

01Prefill and decode

Prefill reads the whole prompt at once and is limited by compute; decode generates one token at a time and is limited by memory bandwidth. They scale differently.

02KV cache

While generating, the model keeps attention keys and values for every token so far. Its size, not the weights, usually limits how many requests fit on a GPU.

03Continuous batching

Requests join and leave a running batch token by token, so the GPU stays busy instead of waiting for the slowest request.

04Tensor parallelism

A 70B model does not fit on one GPU, so each layer's matrices are split across several GPUs that work in lockstep.

05Quantisation

Running weights in FP8 or INT4 halves or quarters memory and speeds up decode, at a small, measurable quality cost.

06Latency metrics

Time to first token (how fast the answer starts) and tokens per second (how fast it streams) matter more than average request time.

Learn them properly in the deep dives:

Part 2 of 5

Question

Now read the question closely. Agree on what you are building, ask what is unclear, and work out the numbers before you draw anything.

2.1Brief and scope

The question exactly as the interviewer asks it, and what they want to learn from your answer.

Interviewer

For cost and data-residency reasons we will self-host a 70-billion-parameter open model for our assistant and internal tools. Expected load is 200 requests per second at peak, with about 2,000 input and 300 output tokens each. Design the serving platform on GPUs.

What it tests: GPU memory arithmetic, engine concepts (continuous batching, KV cache, parallelism), latency metrics, slow-start autoscaling and cost.

The answer in one sentence. An inference engine with continuous batching and paged KV cache, replicas across several GPUs each, routing by free memory, and warm-pool autoscaling.

How to spend the 45 minutes

  1. Clarify~5 minAsk questions, agree on scope and write down assumptions.
  2. Estimate~5 minRough numbers for users, data and traffic, and what they imply.
  3. High-level design~11 minComponents, data flow and the key API on the board.
  4. Deep dive~16 minThe two or three hardest parts, with real mechanisms.
  5. Trade-offs and wrap-up~9 minDecisions, failure modes, and what you would do next.

2.2Clarify the problem

The first five minutes. Each answer removes a guess from your design.

You askInterviewer answers
Latency targets?Time to first token under 1 second at p95; at least 30 tokens per second per stream.
Which GPUs are available?80 GB H100-class GPUs, 8 per node.
Is some traffic batch or offline?Yes, nightly document processing can use spare capacity.
Quality tolerance for quantisation?Acceptable if evaluation scores drop less than 1%.

2.3Estimate the scale

Say the arithmetic out loud. What matters is the order of magnitude and what it forces you to do.

MetricEstimateHow
Weights at FP16≈ 140 GB70B × 2 bytes; needs at least 2 GPUs
Weights at FP8≈ 70 GBfits on 1 GPU but leaves little room for KV cache
KV cache per token≈ 0.3 MB at FP162 × 80 layers × 8 KV heads × 128 dims × 2 bytes
KV cache per request≈ 0.7 GB2,300 tokens × 0.3 MB
Output tokens per second≈ 60k tok/s at peak200 req/s × 300 tokens
Input tokens per second≈ 400k tok/sprefill is compute-heavy
Part 3 of 5

Detailed answer

The design itself, from the big picture down to the code: components first, then one request end to end, then the parts interviewers push hardest on.

3.1Architecture

A gateway routes requests to model replicas. Each replica is an inference engine spanning a few GPUs with tensor parallelism; an autoscaler keeps warm capacity ahead of demand.

EDGECONTROLGPU FLEETClientsassistant · toolsLLM gatewayauth · quotas ·routingBatch queuenightly jobsMetricsTTFT · TPOT · KVuseRouterleast-loaded,prefix-awareAutoscalerqueue depth · KVuseReplica 1vLLM · TP=4 · FP8Replica NvLLM · TP=4 · FP8Model storeweights on NVMeReplica 2vLLM · TP=4 · FP8Small model pool8B, routinerequestsKubernetesGPU node poolslow priorityrouteeasy requestssignalsadd nodesload weights
ClientServiceData storeQueue / streamAI component
Reference design. Boxes show components; arrows show who calls whom.
Inference engine
vLLM (or TensorRT-LLM, SGLang) provides continuous batching, which adds new requests to the running batch every step, and PagedAttention, which stores the KV cache in pages so memory is not wasted on padding.
Tensor parallelism
Each replica splits the model across 4 GPUs with tensor parallelism over NVLink. That leaves most memory for KV cache, which is what limits how many requests a replica serves at once.
Router
Sends requests to the replica with the most free KV cache, and keeps conversations with the same long system prompt together to benefit from prefix caching.
Autoscaler
Scales on queue depth and KV cache utilisation, not CPU. Loading 70 GB of weights takes minutes, so it keeps a warm pool and scales ahead of known peaks.
Small model pool
Routine requests go to an 8B model at a fraction of the cost; the gateway decides by task type.
Batch traffic
Nightly jobs use spare capacity at low priority and are pre-empted when interactive traffic rises.

3.2Key flow, step by step

A request through the platform.

ClientGatewayRouterReplicaMetricsPOST /v1/chat (stream)auth, quota, pick modelrouteto replica with most free KVprefill 2,000 tokensfirst token (TTFT)decode in continuous batchtokens streamTTFT, TPOT, KV useKV use above 85% for 2 minutesadd replica from warm pool
Solid arrows are requests; dashed arrows are responses.
  1. Prefill (reading the prompt) is compute-bound and sets time to first token; decode (writing tokens) is memory-bandwidth-bound and sets tokens per second.
  2. Continuous batching keeps GPUs busy without making short requests wait for long ones.
  3. The resource that actually runs out drives scaling: KV cache memory.

3.3Front end

The users of this platform are other engineers and their apps. Its front end is the client SDK that apps use to stream from it reliably, and the operations console the platform team uses to see and control the fleet.

Screen map

Client SDKstream · retry ·cancelPlaygroundprompt · model ·paramsFleet dashboardTTFT · TPOT · KVReplica detailGPU · queue ·errorsUsage by teamtokens · costModel catalogueversions · statusRollout controlcanary · promoteAlertsSLO breachesdrill down
ClientService
Screens and how users move between them.
Client SDK
Wraps streaming with timeouts, cancellation, retry on 429 and 503 with Retry-After, and returns usage with every response.
Playground
Lets teams try prompts against each model and see tokens, latency and cost before writing code.
Fleet dashboard
Live charts of time to first token, time per output token, KV cache use and queue depth per model and replica, with p95 lines.
Rollout control
Moves a new engine or model version from canary to full fleet only when evaluation and latency gates pass.

Client layers

LayerTechnologyResponsibility
Client SDKTypeScript and Kotlin librariesStreaming, retries, cancellation, usage
Console UIReact + charting libraryFleet, usage, rollouts
Live dataWebSocket or server-sent events from metrics APIUpdate charts every few seconds
AuthSSO with rolesOnly platform engineers can roll out models

Every state the user can see

StateWhat the user seesHow it works
HealthyGreen status, p95 under targetAll SLOs met
DegradedAmber with the breaching metricTTFT or error rate above SLO
Scaling‘Adding 2 replicas’ progressAutoscaler action in progress
CanarySplit view: canary vs stableNew version on a fraction of traffic
Rate limited (client)Retry after N seconds429 from the gateway

Code

Client SDK: stream with cancel and polite retry (TypeScript)

client-sdk-stream.tsTypeScript
1export async function* streamChat(req: ChatRequest, opts: { signal?: AbortSignal; maxRetries?: number } = {}) {2  for (let attempt = 0; ; attempt++) {3    const res = await fetch(GATEWAY + '/v1/chat', { method: 'POST', body: JSON.stringify({ ...req, stream: true }), signal: opts.signal });4    if ((res.status === 429 || res.status === 503) && attempt < (opts.maxRetries ?? 2)) {5      const wait = Number(res.headers.get('Retry-After') ?? 1) * 1000;6      await sleep(wait * (0.5 + Math.random()));                // jittered, server-paced7      continue;8    }9    if (!res.ok) throw new LlmError(res.status, await res.text());10    yield* parseSse(res.body!);                                   // tokens, then a usage event11    return;12  }13}
13 linesUTF-8TypeScript

Live fleet chart subscription (React)

live-fleet-chart.tsxTypeScript JSX
1function useFleetMetrics(model: string) {2  const [points, setPoints] = useState<MetricPoint[]>([]);3  useEffect(() => {4    const es = new EventSource('/ops/metrics?model=' + encodeURIComponent(model));5    es.onmessage = e => setPoints(p => [...p.slice(-299), JSON.parse(e.data)]);   // keep last 300 points6    return () => es.close();7  }, [model]);8  return points;9}
9 linesUTF-8TypeScript JSX

3.4Data model and APIs

The tables and contracts that make the design concrete.

Capacity estimate

capacity-estimate.txtPlain text
1Per replica: 4 × 80 GB = 320 GB2  weights FP8            ≈ 70 GB3  activations/overhead   ≈ 30 GB4  KV cache budget        ≈ 220 GB  (FP8 KV ≈ 0.15 MB/token)5  → ≈ 1.4M cached tokens ≈ 600 concurrent requests of 2,300 tokens6Measured decode throughput per replica (benchmark!)  ≈ 4–6k tok/s7Peak need 60k tok/s → 10–15 replicas → 40–60 GPUs, plus warm pool
7 linesUTF-8Plain text

Engine launch

engine-launch.shShell
1vllm serve meta-llama/Llama-3.1-70B-Instruct \2  --tensor-parallel-size 4 --quantization fp8 --kv-cache-dtype fp8 \3  --max-model-len 8192 --enable-prefix-caching --gpu-memory-utilization 0.92
3 linesUTF-8Shell

3.5Tested code, explained

The core of this design as working code, with what each part does and why.

How much memory one token costs

This arithmetic sets how many requests a GPU can serve at once. Weights are fixed, and the KV cache grows with every token.

kv-size.tsTypeScript
1/** Bytes of KV cache one token costs: a key and a value, per layer, per KV head. */2export function kvBytesPerToken(m: { layers: number; kvHeads: number; headDim: number; bytesPerValue: number }): number {3  return 2 * m.layers * m.kvHeads * m.headDim * m.bytesPerValue;4}5 6/** Requests that fit in the memory left after loading the weights. */7export function concurrentRequests(freeGiB: number, tokensPerRequest: number, bytesPerToken: number): number {8  return Math.floor((freeGiB * 1024 ** 3) / (tokensPerRequest * bytesPerToken));9}
9 linesUTF-8TypeScript
2 * m.layers * ...
A key and a value, for each layer, each head and each dimension.
freeGiB
What's left after loading the weights. Divide by the cost of a whole request, and you have your concurrency limit.

Tested by npm run test:modules.

Static versus continuous batching

One long request shouldn't hold up everyone else. Run both functions on the same jobs, and the difference is plain.

continuous-batching.tsTypeScript
1export interface Job {2  arrival: number;3  tokens: number;4}5 6const byArrival = (jobs: Job[]) => jobs.map((_, i) => i).sort((a, b) => jobs[a].arrival - jobs[b].arrival);7 8/** Static batching: the whole batch waits for its longest request. Returns each job's finish step. */9export function staticBatching(jobs: Job[], size: number): number[] {10  const done = new Array<number>(jobs.length);11  const queue = byArrival(jobs);12  let t = 0;13  let next = 0;14  while (next < queue.length) {15    t = Math.max(t, jobs[queue[next]].arrival);16    const batch: number[] = [];17    while (batch.length < size && next < queue.length && jobs[queue[next]].arrival <= t) batch.push(queue[next++]);18    t += Math.max(...batch.map((i) => jobs[i].tokens));19    for (const i of batch) done[i] = t;20  }21  return done;22}23 24/** Continuous batching: finished jobs leave after any step, and waiting jobs join at once. */25export function continuousBatching(jobs: Job[], size: number): number[] {26  const done = new Array<number>(jobs.length);27  const queue = byArrival(jobs);28  const left = jobs.map((j) => j.tokens);29  const running = new Set<number>();30  let t = 0;31  let next = 0;32  while (running.size > 0 || next < queue.length) {33    while (running.size < size && next < queue.length && jobs[queue[next]].arrival <= t) running.add(queue[next++]);34    if (running.size === 0) {35      t = jobs[queue[next]].arrival;36      continue;37    }38    t++; // one step: one token for everyone in the batch39    for (const i of [...running]) {40      if (--left[i] === 0) {41        done[i] = t;42        running.delete(i);43      }44    }45  }46  return done;47}
47 linesUTF-8TypeScript
t += Math.max(...)
Static batching: everyone waits for the longest request in the batch.
running.delete(i)
Continuous batching: a finished request leaves after any step.
running.add(queue[next++])
A waiting request joins at the very next step, which is where the throughput gain comes from.

Tested by npm run test:modules.

Reusing a shared prompt prefix

Many requests start with the same system prompt, so its KV entries can be reused.

prefix-cache.tsTypeScript
1/** vLLM-style prefix cache: each full block's key chains its parent's hash, its tokens and a salt. */2export class PrefixCache {3  private blocks = new Map<string, true>(); // Map order doubles as LRU order4  private blockTokens: number;5  private capacity: number;6 7  constructor(blockTokens = 16, capacity = 1024) {8    this.blockTokens = blockTokens;9    this.capacity = capacity;10  }11 12  private keys(tokens: number[], salt: string): string[] {13    // At least one token must still run through the model, so never cache the very last one.14    const whole = Math.floor((tokens.length - 1) / this.blockTokens);15    const out: string[] = [];16    let parent = salt;17    for (let b = 0; b < whole; b++) {18      parent = `${parent}|${tokens.slice(b * this.blockTokens, (b + 1) * this.blockTokens).join(",")}`;19      out.push(parent);20    }21    return out;22  }23 24  /** Number of prompt tokens the engine can skip. */25  lookup(tokens: number[], salt = ""): number {26    let hits = 0;27    for (const key of this.keys(tokens, salt)) {28      if (!this.blocks.has(key)) break;29      this.blocks.delete(key); // move to most recently used30      this.blocks.set(key, true);31      hits++;32    }33    return hits * this.blockTokens;34  }35 36  insert(tokens: number[], salt = ""): void {37    for (const key of this.keys(tokens, salt)) {38      this.blocks.delete(key);39      this.blocks.set(key, true);40      if (this.blocks.size > this.capacity) this.blocks.delete(this.blocks.keys().next().value as string);41    }42  }43}
43 linesUTF-8TypeScript
parent = `${parent}|...`
Each block's key includes its parent's, so a match means the whole prefix matches.
salt
Keeps tenants apart, because shared prefixes can otherwise leak through timing.
if (!this.blocks.has(key)) break
Stops at the first miss. Later blocks are useless without the earlier ones.
tokens.length - 1
At least one token must run through the model to produce the next one.

Tested by npm run test:modules.

Serving with Ray Serve

This scales replicas with load and restarts unhealthy ones.

ray_serve_app.pyPython
1from ray import serve2from starlette.requests import Request3from vllm import LLM, SamplingParams4 5 6@serve.deployment(7    ray_actor_options={"num_gpus": 1},8    autoscaling_config={"min_replicas": 2, "max_replicas": 20, "target_ongoing_requests": 16},9    max_ongoing_requests=32,10    health_check_period_s=10,11)12class Generator:13    def __init__(self, model: str):14        self.llm = LLM(model=model)15 16    def check_health(self) -> None:17        self.llm.generate(["ping"], SamplingParams(max_tokens=1))18 19    async def __call__(self, request: Request) -> dict:20        body = await request.json()21        output = self.llm.generate([body["prompt"]], SamplingParams(max_tokens=body.get("max_tokens", 256)))22        return {"text": output[0].outputs[0].text}23 24 25app = Generator.bind(model="meta-llama/Llama-3.1-8B-Instruct")
25 linesUTF-8Python
target_ongoing_requests
Ray adds replicas when each one holds more than this.
max_ongoing_requests
Back-pressure: beyond this, requests queue rather than run out of GPU memory.
check_health
Generates one token, which proves the GPU works, and not merely the process.

Syntax-checked. The GPU libraries it needs don't run in our tests.

3.6Hard parts, in depth

Where interviewers spend the second half of the round.

The numbers that matter

Time to first token (TTFT) and time per output token (TPOT) are the user-facing latencies. Throughput in tokens per second per GPU is the cost metric. Track all three per model and replica; averages hide the tail.

Benchmark with your real prompt lengths. A 2,000-token prompt changes the balance between prefill and decode completely compared with chat benchmarks.

Getting more from each GPU

FP8 weights and KV cache roughly halve memory with small quality loss on current GPUs. Prefix caching avoids recomputing a long shared system prompt. Speculative decoding uses a small draft model to propose tokens the large model verifies in one pass, raising tokens per second.

Separate prefill and decode onto different replicas only at large scale; it adds complexity.

Cold starts and reliability

Keep weights on local NVMe or a fast shared cache so a new replica loads in minutes, not tens of minutes. Health-check with a real generation, not a TCP ping. Drain replicas gracefully: stop new requests, finish streams, then stop.

Part 4 of 5

Q&A

The questions that come after the design, with model answers. Answer each one out loud before opening it.

4.1Follow-up questions

What the interviewer asks once the main design is on the board.

When is self-hosting worth it?
When volume is steady and high enough to keep GPUs busy, or data residency requires it. At low or spiky volume, hosted APIs are usually cheaper.
How do you serve many fine-tuned variants?
LoRA adapters loaded on top of one base model, so one replica serves many variants.
How do you charge teams internally?
Meter input and output tokens per team at the gateway and allocate GPU cost by tokens.

4.2Front-end follow-ups

Questions aimed at the client: rendering, state and devices.

What belongs in the client SDK rather than each app?
Streaming parsing, cancellation, server-paced retries, timeouts and usage reporting, so every team behaves well under load without reimplementing it.
What would you show on the main ops screen?
TTFT and TPOT at p95, error rate, KV cache use and queue depth per model, plus active rollouts and spend today.

4.3Incidents, probes and traps

Realistic production incidents and pointed probes, grouped by topic.

The product 1

ProbeWhen is self-hosting a bad idea?
When volume is low or spiky, or the team lacks GPU operations experience. Hosted APIs are then cheaper and safer.

Requirements 1

TrapWhich latency numbers do you ask for?
Time to first token and time per output token at p95, not averages. They map to prefill and decode and lead to different fixes.

Scale 1

ProbeWalk me through the KV cache maths.
Bytes per token = 2 × layers × KV heads × head dimension × bytes per value. For a 70B-class model with grouped-query attention that is about 0.3 MB per token at FP16, so 2,300 tokens is about 0.7 GB per request.

Architecture 2

IncidentLatency spikes every afternoon; GPUs show 95% utilisation and the queue grows.
The KV cache is full, so requests are pre-empted and recomputed. Scale on KV use and queue depth, cap context per tenant, and add warm replicas ahead of the daily peak.
ProbeWhy route by free KV cache rather than round robin?
Requests differ hugely in length. Free KV memory is the real capacity signal; round robin overloads replicas holding long conversations.

The key flow 1

IncidentA few very long prompts make everyone slow.
Large prefills stall the batch. Use chunked prefill, cap prompt length, or send long-context traffic to its own pool.

Front end 2

IncidentClient apps retry aggressively during overload and make it worse.
Ship a client SDK that honours Retry-After with jitter, limits retries and uses a circuit breaker, and make the gateway return 429 early.
ProbeWhat does the ops console show first?
TTFT and TPOT at p95, error rate, KV use and queue depth per model, active rollouts, and spend today.

Data and APIs 1

TrapWhy store the KV cache in FP8?
It roughly doubles concurrent requests per GPU. Verify quality on your evaluation set before relying on it.

The hard parts 2

IncidentA new engine version doubled benchmark throughput but production quality dropped.
Numerics or default sampling settings changed. Pin versions, canary with the evaluation suite and compare outputs before fleet rollout.
ProbeHow do you serve 30 fine-tuned variants?
LoRA adapters on one base model with multi-adapter serving, so one replica serves many variants.

Trade-offs 1

ProbeTensor parallel 8 or 4?
TP=8 frees more memory per replica but adds communication and gives fewer replicas. Benchmark both on your prompt lengths.

Failures 2

IncidentA node failed during peak and 300 streams dropped.
Spread replicas across nodes, health-check with real generations, remove failed replicas quickly, and let clients retry safely on another replica.
IncidentScaling up takes 25 minutes because of cold starts.
Each start downloaded the weights from object storage. Cache weights on local NVMe or peer nodes, keep a warm pool and scale ahead of predictable peaks.
Part 5 of 5

Extra depth

What turns a good answer into a strong one: the decisions you rejected, how the system fails, and the signals of seniority.

5.1Trade-offs and decisions

Every choice has a cost. Naming it is what interviewers listen for.

Show detailsHide details
DecisionChosenRejectedWhy
PrecisionFP8FP16Fits more KV cache per GPU; verify the quality drop on your evaluation set.
ParallelismTP=4 per replicaTP=2 or TP=8Balances KV headroom and per-replica throughput; benchmark to confirm.
Scaling signalQueue depth + KV useCPU or GPU utilisationGPUs look busy even when they could take more; KV memory is the true limit.
Model mixRouter to 8B and 70B70B for everythingMost requests don’t need the large model.

5.2Failure modes

What breaks in production, and how the design keeps working when it does.

Show detailsHide details
What breaksHow the design copes
GPU node failureReplicas spread across nodes; router removes unhealthy replicas within seconds; streams retry on another replica.
Traffic spike beyond warm poolQueue with a short timeout, shed lowest-priority traffic, route overflow to a hosted provider.
Long prompts exhaust KV cachePer-tenant context limits and pre-emption of batch jobs.
Quality regression after an engine upgradeCanary replica with evaluation suite before fleet rollout.

5.3Performance and accessibility

The client-side details that separate a demo from a product.

Performance

  • Clients open the stream immediately and render the first token as soon as it arrives.
  • Dashboards downsample to a few hundred points per chart.
  • The SDK reuses connections (HTTP/2) to avoid handshake latency per request.

Accessibility and localisation

  • Charts have data tables and text summaries for screen readers.
  • Status uses icons and words, not colour alone.

5.4Senior versus principal

The same design, answered at two levels. Use the rubric to score yourself.

Senior answer

A senior answer deploys a model on GPUs behind an API.

Principal answer

A principal answer does the memory arithmetic, explains prefill versus decode, continuous batching and KV cache limits, picks parallelism and precision with evidence, scales on the right signal with warm pools, and frames when self-hosting pays off.

Score yourself

5.5Whiteboard

Draw the design here, in this order. It stays in this browser.

  1. Gateway → router → replicas
  2. One replica drawn as 4 GPUs with weights and KV cache
  3. Autoscaler fed by KV use and queue depth
  4. Small-model pool beside the large one
  5. Batch queue using spare capacity
Tip: draw boxes first, then arrows, then label them with Text.
AI Engineer · interview room