AI systems reels
The whole idea
vLLM runs one loop: schedule, allocate, execute, free. Each pass generates one token for every running request. A scheduler picks the requests, a block manager gives them KV cache memory in small fixed blocks, and a worker runs one batched forward pass. Memory, not compute, usually caps concurrency, so vLLM spends its cleverness there.
The question
Your inference fleet returns 429s at 40% GPU utilisation, and nvidia-smi shows memory full. Explain what limits throughput, and how an engine like vLLM changes it.
Queue the request
The API server tokenises the prompt and adds it to a waiting queue. No GPU memory moves yet.
Schedule every step
Before each forward pass the scheduler walks the running list, gives a request a new block if its last one filled, then admits waiting requests while blocks remain. The default policy serves first come, first served.
Preempt on shortage
No free block? The scheduler evicts the most recently admitted request, then recomputes it later or swaps its blocks to CPU memory. Frequent preemption signals overload, so alert on it.
Execute one batched step
The worker flattens all running requests into one batch and passes the block tables alongside. The attention kernel reads keys and values block by block, so physical blocks need not sit together.
Stream and free
Each new token streams back. A finished request returns its blocks at once, and the next step admits someone new.
Why does PagedAttention raise throughput but not per-token speed?
It packs more requests into the same memory. Each token still takes one forward step, but the batch grows. The paper reports 2–4× the throughput of TGI and Orca-style engines on the same hardware.
The whole idea
Attention needs every key of a sequence, but paged storage scatters them across memory. Two parts solve it: a block manager that owns memory, and a kernel that follows each sequence's block table while it computes scores.
The question
Implement the memory manager behind PagedAttention: add a token, fork a sequence for parallel sampling, release a finished one. Then sketch the GPU kernel that computes attention scores through the block table.
Keep a free list
Free blocks sit in a stack, so allocation and release take O(1).
Add one token
Allocate only when the last block fills. A partly used block costs at most one block of waste per sequence.
Fork with reference counts
Fork copies the table and raises each block's count. No tensor data moves.
Copy on write
Before writing to a shared last block, take a new one. A real engine also copies the tensor contents; this code tracks ownership.
Release by count
Release lowers each count and frees blocks that reach zero.
Read through the table
The kernel runs one program per logical block, loads the physical id from the table, computes the pool offset and takes dot products with the query.
Why return false instead of evicting inside addToken?
Eviction needs policy: arrival order, priority, tenant. Keeping that in the scheduler leaves the allocator simple and testable.
The whole idea
A static batch finishes when its longest request finishes, so short requests sit idle. Continuous batching schedules at every token step: finished requests leave, and waiting ones join.
The question
Short chat replies wait behind 2,000-token report jobs on the same GPUs. Product wants p95 time to first token under 500 ms with no new hardware. What do you change?
Find the waste
Under static batching the batch holds its slots until the longest request ends. GPU slots idle, and new arrivals queue.
Admit between steps
Check the queue after every token. A slot freed at step 10 serves a new request at step 11.
Watch the prefill cost
A new request's prefill shares a step with running decodes, and everyone waits for it. Long prompts now stall short replies.
Chunk the prefill
Give each step a token budget. Long prompts prefill across several steps, so decodes keep moving.
Separate the classes
Put reports in a lower-priority queue with their own concurrency limit. Reject or defer when the waiting queue grows.
Does continuous batching raise tokens per second?
Yes, under load, because slots never idle. The Orca paper reported large throughput gains at the same latency.