/Practice

AI systems reels

AI systems1 / 48
1 / 8

The whole idea

vLLM runs one loop: schedule, allocate, execute, free. Each pass generates one token for every running request. A scheduler picks the requests, a block manager gives them KV cache memory in small fixed blocks, and a worker runs one batched forward pass. Memory, not compute, usually caps concurrency, so vLLM spends its cleverness there.

Why it mattersPagedAttention treats the KV cache like virtual memory, so one GPU holds 2–4× more concurrent requests.
2 / 8

The question

Your inference fleet returns 429s at 40% GPU utilisation, and nvidia-smi shows memory full. Explain what limits throughput, and how an engine like vLLM changes it.

You will be able toExplain why memory caps LLM throughput, walk through one vLLM scheduling step, and size a KV cache on a whiteboard.
3 / 8

Queue the request

The API server tokenises the prompt and adds it to a waiting queue. No GPU memory moves yet.

4 / 8

Schedule every step

Before each forward pass the scheduler walks the running list, gives a request a new block if its last one filled, then admits waiting requests while blocks remain. The default policy serves first come, first served.

5 / 8

Preempt on shortage

No free block? The scheduler evicts the most recently admitted request, then recomputes it later or swaps its blocks to CPU memory. Frequent preemption signals overload, so alert on it.

6 / 8

Execute one batched step

The worker flattens all running requests into one batch and passes the block tables alongside. The attention kernel reads keys and values block by block, so physical blocks need not sit together.

7 / 8

Stream and free

Each new token streams back. A finished request returns its blocks at once, and the next step admits someone new.

8 / 8

Why does PagedAttention raise throughput but not per-token speed?

It packs more requests into the same memory. Each token still takes one forward step, but the batch grows. The paper reports 2–4× the throughput of TGI and Orca-style engines on the same hardware.

CounterpointPaged blocks add an indirection to every attention read. When request lengths barely vary and memory has room, simpler engines match vLLM. Measure on your workload before switching.
vLLM High-Throughput Engine Architecture
#kv-cache#long-read
AI systems2 / 48
1 / 9

The whole idea

Attention needs every key of a sequence, but paged storage scatters them across memory. Two parts solve it: a block manager that owns memory, and a kernel that follows each sequence's block table while it computes scores.

Why it mattersBuild the block table and the kernel that reads it, so multi-tenant batches never fragment GPU memory.
2 / 9

The question

Implement the memory manager behind PagedAttention: add a token, fork a sequence for parallel sampling, release a finished one. Then sketch the GPU kernel that computes attention scores through the block table.

You will be able toImplement a block allocator with copy-on-write, and write a Triton kernel that reads keys through a block table.
3 / 9

Keep a free list

Free blocks sit in a stack, so allocation and release take O(1).

4 / 9

Add one token

Allocate only when the last block fills. A partly used block costs at most one block of waste per sequence.

5 / 9

Fork with reference counts

Fork copies the table and raises each block's count. No tensor data moves.

6 / 9

Copy on write

Before writing to a shared last block, take a new one. A real engine also copies the tensor contents; this code tracks ownership.

7 / 9

Release by count

Release lowers each count and frees blocks that reach zero.

8 / 9

Read through the table

The kernel runs one program per logical block, loads the physical id from the table, computes the pool offset and takes dot products with the query.

9 / 9

Why return false instead of evicting inside addToken?

Eviction needs policy: arrival order, priority, tenant. Keeping that in the scheduler leaves the allocator simple and testable.

Next stepAdd a hash per block and this allocator becomes a prefix cache.
Block-Allocated PagedAttention Triton Kernel
#triton-kernel#machine-coding
AI systems3 / 48
1 / 8

The whole idea

A static batch finishes when its longest request finishes, so short requests sit idle. Continuous batching schedules at every token step: finished requests leave, and waiting ones join.

Why it mattersIteration-level scheduling ends the wait for a batch's longest request; prefill stalls become the next bottleneck.
2 / 8

The question

Short chat replies wait behind 2,000-token report jobs on the same GPUs. Product wants p95 time to first token under 500 ms with no new hardware. What do you change?

You will be able toCompare static and continuous batching with numbers, and choose scheduler settings for a latency target.
3 / 8

Find the waste

Under static batching the batch holds its slots until the longest request ends. GPU slots idle, and new arrivals queue.

4 / 8

Admit between steps

Check the queue after every token. A slot freed at step 10 serves a new request at step 11.

5 / 8

Watch the prefill cost

A new request's prefill shares a step with running decodes, and everyone waits for it. Long prompts now stall short replies.

6 / 8

Chunk the prefill

Give each step a token budget. Long prompts prefill across several steps, so decodes keep moving.

7 / 8

Separate the classes

Put reports in a lower-priority queue with their own concurrency limit. Reject or defer when the waiting queue grows.

8 / 8

Does continuous batching raise tokens per second?

Yes, under load, because slots never idle. The Orca paper reported large throughput gains at the same latency.

CounterpointOffline jobs of uniform length lose little under static batching. Skip the scheduler work for nightly batches.
Continuous Batching & Iteration-Level Scheduling
#scheduler#deep-dive
Tensor Parallelism & Ring-AllReduce Communication
Speculative Decoding Target-Draft Pair Verifier
vLLM Automatic Prefix Caching Architecture
Fused Flash-Softmax Triton Kernel
Hierarchical Navigable Small World (HNSW) Graphs
Reciprocal Rank Fusion (RRF) Hybrid Search
Parent-Child Hierarchical Context Retrieval
Entity-Relation Graph Mining & GraphRAG Pipeline
Redis Vector Semantic Response Caching
Inverted Indexing & BM25 Scoring Engine
BGE-Reranker Multi-Stage Retrieval Pipeline
Selective Context Window Token Compression
Low-Rank Adaptation (LoRA) Weight Decomposition
PyTorch LoRA Linear Layer Implementation
QLoRA NormalFloat4 & Double Quantization
Direct Preference Optimization (DPO) Loss Function
DeepSpeed ZeRO-3 Memory Partitioning
PyTorch Fully Sharded Data Parallel (FSDP)
Merging LoRA Adapters into Base Model Weights
Unsloth Gradient Calculation Acceleration
LLM-as-a-Judge Evaluation & Calibration Architecture
Llama-Guard Jailbreak & Safety Classifier
Faithfulness, Answer Relevance & Context Recall Metrics
Guardrails AI Structured Pydantic Output Validation
Evaluating Multimodal LLMs on Visual Reasoning
Automated Adversarial Red-Teaming Suites
SLURM GPU Cluster Job Scheduling & Failure Recovery
Async Zero-Redundancy Optimizer Checkpointing
Triton Compiler JIT Codegen Pipeline
NCCL Inter-Node Transport Protocols & Infiniband
Ray Serve Distributed Actor Deployment
Prompt Patterns: Chain of Thought, Few-Shot, Structured Output
CLIP and ViT Multimodal Embeddings
App Startup: Cold Start From Tap to First Frame
Mobile Observability: Crashes, ANRs, Traces and Rollout Gates
Mobile Security: Tokens, Keystore, Pinning and Tampering
Live Location Tracking for a Driver App
Push Notification Platform at Scale
Feature Flag and Experimentation SDK
React Native New Architecture: JSI, Fabric, TurboModules and Threads
React Native State: Server Cache, Client Store and Render Storms
Modular React Native for Many Apps: The Modularity Ladder
Memory Leaks in React Native: Three Heaps, One Staircase
Offline Sync in React Native: What to Persist and Where
The 95/5 Rule and Performance Budgets in CI