Design every step so it is safe to kill and resume.
if retries < 5 { backoff(attempt) }
AI handbook reels
AI handbook1 / 7
1 / 6
What it is
A large language model is a function that takes a sequence of tokens and returns a probability for every possible next token. Generation repeats that function, appending one chosen token at a time. Almost every engineering concern, from latency and cost to hallucination and context limits, follows from this loop.
2 / 6
Follow one request
A user asks the booking assistant: “Move my massage to Friday evening.” The orchestrator builds a prompt of about 2,400 tokens: system instructions, tool definitions, the last few turns and retrieved facts. The tokenizer turns that text into 2,400 integer IDs using a byte-pair-encoding vocabulary of roughly 100,000 to 200,000 entries. An embedding table turns each ID into a vector of a few thousand numbers.
3 / 6
Inside it
Tokenizer. Byte-pair encoding splits text into frequent sub-word pieces. Token counts drive price, latency and context limits, so always measure them with the provider’s tokenizer. Self-attention. Each token produces a query, key and value. Scores are queries times keys, scaled by √d, turned into weights by softmax, and used to mix values. Causal masking stops a token from seeing the future. Grouped-query attention shares keys and values across heads to shrink the KV cache. MLP and mixture of experts. A feed-forward network transforms each position independently. In mixture-of-experts models, a router sends each token to a few expert MLPs, so only part of the parameters run per token.
4 / 6
Generating a reply token by token
Prefill is compute-bound and grows with prompt length; long prompts mean slower first tokens. Decode is memory-bandwidth-bound: each step reads all weights and the KV cache to produce one token. This is why output tokens usually cost more than input tokens, and why streaming improves perceived speed.
5 / 6
Tools to know
Dense vs mixture of experts: Dense: every parameter runs per token. MoE: a few experts per token. MoE gives large-model quality at lower compute per token, at the cost of more memory. Context length: 8k · 32k · 128k · 1M tokens. Longer windows simplify prompts but cost more and do not replace retrieval for large corpora. Reasoning models: Spend extra tokens thinking before answering. Better on multi-step problems; slower and more expensive per answer.
6 / 6
Interview question
“Why does a long prompt slow the first token but not later tokens as much?” Prefill processes all prompt tokens at once, which is compute-heavy. Later tokens reuse the KV cache and only process one new position each.
1 / 6
What it is
A large language model is a function that takes a sequence of tokens and returns a probability for every possible next token. Generation repeats that function, appending one chosen token at a time. Almost every engineering concern, from latency and cost to hallucination and context limits, follows from this loop.
2 / 6
Follow one request
A user asks the booking assistant: “Move my massage to Friday evening.” The orchestrator builds a prompt of about 2,400 tokens: system instructions, tool definitions, the last few turns and retrieved facts. The tokenizer turns that text into 2,400 integer IDs using a byte-pair-encoding vocabulary of roughly 100,000 to 200,000 entries. An embedding table turns each ID into a vector of a few thousand numbers.
3 / 6
Inside it
Tokenizer. Byte-pair encoding splits text into frequent sub-word pieces. Token counts drive price, latency and context limits, so always measure them with the provider’s tokenizer. Self-attention. Each token produces a query, key and value. Scores are queries times keys, scaled by √d, turned into weights by softmax, and used to mix values. Causal masking stops a token from seeing the future. Grouped-query attention shares keys and values across heads to shrink the KV cache. MLP and mixture of experts. A feed-forward network transforms each position independently. In mixture-of-experts models, a router sends each token to a few expert MLPs, so only part of the parameters run per token.
4 / 6
Generating a reply token by token
Prefill is compute-bound and grows with prompt length; long prompts mean slower first tokens. Decode is memory-bandwidth-bound: each step reads all weights and the KV cache to produce one token. This is why output tokens usually cost more than input tokens, and why streaming improves perceived speed.
5 / 6
Tools to know
Dense vs mixture of experts: Dense: every parameter runs per token. MoE: a few experts per token. MoE gives large-model quality at lower compute per token, at the cost of more memory. Context length: 8k · 32k · 128k · 1M tokens. Longer windows simplify prompts but cost more and do not replace retrieval for large corpora. Reasoning models: Spend extra tokens thinking before answering. Better on multi-step problems; slower and more expensive per answer.
6 / 6
Interview question
“Why does a long prompt slow the first token but not later tokens as much?” Prefill processes all prompt tokens at once, which is compute-heavy. Later tokens reuse the KV cache and only process one new position each.
How an LLM works inside
#ai#deep dive
❚❚
AI handbook2 / 7
1 / 6
What it is
Inference speed and cost come down to three limits: GPU compute during prefill, memory bandwidth during decode, and memory capacity for weights and KV cache. Every acceleration technique relaxes one of them: smaller numbers, fewer tokens, better batching, or guessing ahead.
2 / 6
A real optimisation story
A support summariser on a self-hosted 70B model costs too much and p95 latency is 9 seconds. The team measures first: average prompt 6,000 tokens, output 250, GPU utilisation 35% because each replica processes one request at a time. Step one moves to vLLM with continuous batching and paged KV cache: throughput per GPU roughly triples and p95 drops, because the GPU now decodes many requests per step. Step two trims the prompt: summarising ticket history once and caching it cuts input to 2,500 tokens and halves TTFT.
3 / 6
Inside it
Routing and distillation. The cheapest token is one processed by a smaller model. Classify requests and send routine ones to a small model, often distilled from the large one on your task. Context trimming and caching. Input tokens dominate cost for retrieval-heavy apps. Summarise history, retrieve only relevant chunks, and keep stable prefixes identical so provider prompt caching or engine prefix caching applies. Continuous batching. Instead of waiting for a batch to finish, the server adds and removes requests every decode step, keeping the GPU full.
4 / 6
Speculative decoding step
Speed-up depends on the acceptance rate, which is high for predictable text such as code and templated answers. Because the target model verifies every token, the output matches what it would have produced on its own.
5 / 6
Tools to know
Serving engines: vLLM · SGLang · TensorRT-LLM · Hugging Face TGI · llama.cpp. vLLM or SGLang for general GPU serving; TensorRT-LLM for maximum NVIDIA performance; llama.cpp for CPU and edge. Quantisation: FP8 · AWQ · GPTQ · GGUF (llama.cpp). FP8 first on supported GPUs; INT4 when memory is the hard limit. Kernels: FlashAttention · fused kernels in engines. Usually provided by the engine; check versions match your GPU.
6 / 6
Interview question
“Your LLM feature is too slow. Where do you start?” Measure TTFT versus TPOT. Slow first token means long prompts or queueing: trim context, cache prefixes, add capacity. Slow streaming means decode limits: batching, quantisation, speculative decoding or a smaller model.
1 / 6
What it is
Inference speed and cost come down to three limits: GPU compute during prefill, memory bandwidth during decode, and memory capacity for weights and KV cache. Every acceleration technique relaxes one of them: smaller numbers, fewer tokens, better batching, or guessing ahead.
2 / 6
A real optimisation story
A support summariser on a self-hosted 70B model costs too much and p95 latency is 9 seconds. The team measures first: average prompt 6,000 tokens, output 250, GPU utilisation 35% because each replica processes one request at a time. Step one moves to vLLM with continuous batching and paged KV cache: throughput per GPU roughly triples and p95 drops, because the GPU now decodes many requests per step. Step two trims the prompt: summarising ticket history once and caching it cuts input to 2,500 tokens and halves TTFT.
3 / 6
Inside it
Routing and distillation. The cheapest token is one processed by a smaller model. Classify requests and send routine ones to a small model, often distilled from the large one on your task. Context trimming and caching. Input tokens dominate cost for retrieval-heavy apps. Summarise history, retrieve only relevant chunks, and keep stable prefixes identical so provider prompt caching or engine prefix caching applies. Continuous batching. Instead of waiting for a batch to finish, the server adds and removes requests every decode step, keeping the GPU full.
4 / 6
Speculative decoding step
Speed-up depends on the acceptance rate, which is high for predictable text such as code and templated answers. Because the target model verifies every token, the output matches what it would have produced on its own.
5 / 6
Tools to know
Serving engines: vLLM · SGLang · TensorRT-LLM · Hugging Face TGI · llama.cpp. vLLM or SGLang for general GPU serving; TensorRT-LLM for maximum NVIDIA performance; llama.cpp for CPU and edge. Quantisation: FP8 · AWQ · GPTQ · GGUF (llama.cpp). FP8 first on supported GPUs; INT4 when memory is the hard limit. Kernels: FlashAttention · fused kernels in engines. Usually provided by the engine; check versions match your GPU.
6 / 6
Interview question
“Your LLM feature is too slow. Where do you start?” Measure TTFT versus TPOT. Slow first token means long prompts or queueing: trim context, cache prefixes, add capacity. Slow streaming means decode limits: batching, quantisation, speculative decoding or a smaller model.
Making inference fast and cheap
#ai#deep dive
❚❚
AI handbook3 / 7
1 / 6
What it is
An embedding model turns text, images or audio into a vector so that similar meanings land close together. Vector search finds the nearest vectors quickly using approximate indexes. Together they power semantic search, recommendations, deduplication and the retrieval half of RAG.
2 / 6
Building semantic search for a service catalogue
The wellness app has 40,000 services across 3,000 clinics. Keyword search fails on ‘something for a stiff neck after long flights’ because no listing contains those words. A short document per service feeds the embedding: title, category, description and benefits. Postgres stores the 768-dimension vectors with pgvector and an HNSW index, alongside city, price and clinic ID for filtering.
3 / 6
Inside it
Embedding model. Maps input to a fixed-length vector. Use the same model, version and preprocessing for documents and queries; changing models means re-embedding everything. Similarity. Usually cosine similarity, or dot product on normalised vectors. Approximate index. HNSW builds a navigable graph for fast, high-recall search; IVF partitions vectors into clusters; product quantisation compresses vectors to save memory at some recall cost.
4 / 6
A hybrid query
Filters run inside both searches so results never include other cities. The reranker is where most of the precision comes from.
5 / 6
Tools to know
Embedding models: Hosted embedding APIs · open models (e.g. BGE, E5, multilingual variants). Benchmark on your queries; check multilingual quality, dimension and cost. Vector stores: pgvector · OpenSearch · Qdrant · Weaviate · Milvus · Pinecone. Start in the database you run; move when scale or filtering demands. Rerankers: Cross-encoder models · hosted rerank APIs. Almost always worth it for top-k quality.
6 / 6
Interview question
“Why combine keyword and vector search?” Vectors miss exact tokens like product codes; keywords miss paraphrases. Fusion gets both, and a reranker sorts the final list.
1 / 6
What it is
An embedding model turns text, images or audio into a vector so that similar meanings land close together. Vector search finds the nearest vectors quickly using approximate indexes. Together they power semantic search, recommendations, deduplication and the retrieval half of RAG.
2 / 6
Building semantic search for a service catalogue
The wellness app has 40,000 services across 3,000 clinics. Keyword search fails on ‘something for a stiff neck after long flights’ because no listing contains those words. A short document per service feeds the embedding: title, category, description and benefits. Postgres stores the 768-dimension vectors with pgvector and an HNSW index, alongside city, price and clinic ID for filtering.
3 / 6
Inside it
Embedding model. Maps input to a fixed-length vector. Use the same model, version and preprocessing for documents and queries; changing models means re-embedding everything. Similarity. Usually cosine similarity, or dot product on normalised vectors. Approximate index. HNSW builds a navigable graph for fast, high-recall search; IVF partitions vectors into clusters; product quantisation compresses vectors to save memory at some recall cost.
4 / 6
A hybrid query
Filters run inside both searches so results never include other cities. The reranker is where most of the precision comes from.
5 / 6
Tools to know
Embedding models: Hosted embedding APIs · open models (e.g. BGE, E5, multilingual variants). Benchmark on your queries; check multilingual quality, dimension and cost. Vector stores: pgvector · OpenSearch · Qdrant · Weaviate · Milvus · Pinecone. Start in the database you run; move when scale or filtering demands. Rerankers: Cross-encoder models · hosted rerank APIs. Almost always worth it for top-k quality.
6 / 6
Interview question
“Why combine keyword and vector search?” Vectors miss exact tokens like product codes; keywords miss paraphrases. Fusion gets both, and a reranker sorts the final list.
Embeddings and vector search
#ai#deep dive
❚❚
Prompting, RAG or fine-tuning?Tool use, agents and MCPIntegrating LLMs into productsUsing AI to accelerate engineering teams