/Practice

AI handbook reels

AI handbook1 / 7
1 / 6

What it is

A large language model is a function that takes a sequence of tokens and returns a probability for every possible next token. Generation repeats that function, appending one chosen token at a time. Almost every engineering concern, from latency and cost to hallucination and context limits, follows from this loop.

2 / 6

Follow one request

A user asks the booking assistant: “Move my massage to Friday evening.” The orchestrator builds a prompt of about 2,400 tokens: system instructions, tool definitions, the last few turns and retrieved facts. The tokenizer turns that text into 2,400 integer IDs using a byte-pair-encoding vocabulary of roughly 100,000 to 200,000 entries. An embedding table turns each ID into a vector of a few thousand numbers.

3 / 6

Inside it

Tokenizer. Byte-pair encoding splits text into frequent sub-word pieces. Token counts drive price, latency and context limits, so always measure them with the provider’s tokenizer. Self-attention. Each token produces a query, key and value. Scores are queries times keys, scaled by √d, turned into weights by softmax, and used to mix values. Causal masking stops a token from seeing the future. Grouped-query attention shares keys and values across heads to shrink the KV cache. MLP and mixture of experts. A feed-forward network transforms each position independently. In mixture-of-experts models, a router sends each token to a few expert MLPs, so only part of the parameters run per token.

4 / 6

Generating a reply token by token

Prefill is compute-bound and grows with prompt length; long prompts mean slower first tokens. Decode is memory-bandwidth-bound: each step reads all weights and the KV cache to produce one token. This is why output tokens usually cost more than input tokens, and why streaming improves perceived speed.

5 / 6

Tools to know

Dense vs mixture of experts: Dense: every parameter runs per token. MoE: a few experts per token. MoE gives large-model quality at lower compute per token, at the cost of more memory. Context length: 8k · 32k · 128k · 1M tokens. Longer windows simplify prompts but cost more and do not replace retrieval for large corpora. Reasoning models: Spend extra tokens thinking before answering. Better on multi-step problems; slower and more expensive per answer.

6 / 6

Interview question

“Why does a long prompt slow the first token but not later tokens as much?” Prefill processes all prompt tokens at once, which is compute-heavy. Later tokens reuse the KV cache and only process one new position each.

How an LLM works inside
#ai#deep dive
AI handbook2 / 7
1 / 6

What it is

Inference speed and cost come down to three limits: GPU compute during prefill, memory bandwidth during decode, and memory capacity for weights and KV cache. Every acceleration technique relaxes one of them: smaller numbers, fewer tokens, better batching, or guessing ahead.

2 / 6

A real optimisation story

A support summariser on a self-hosted 70B model costs too much and p95 latency is 9 seconds. The team measures first: average prompt 6,000 tokens, output 250, GPU utilisation 35% because each replica processes one request at a time. Step one moves to vLLM with continuous batching and paged KV cache: throughput per GPU roughly triples and p95 drops, because the GPU now decodes many requests per step. Step two trims the prompt: summarising ticket history once and caching it cuts input to 2,500 tokens and halves TTFT.

3 / 6

Inside it

Routing and distillation. The cheapest token is one processed by a smaller model. Classify requests and send routine ones to a small model, often distilled from the large one on your task. Context trimming and caching. Input tokens dominate cost for retrieval-heavy apps. Summarise history, retrieve only relevant chunks, and keep stable prefixes identical so provider prompt caching or engine prefix caching applies. Continuous batching. Instead of waiting for a batch to finish, the server adds and removes requests every decode step, keeping the GPU full.

4 / 6

Speculative decoding step

Speed-up depends on the acceptance rate, which is high for predictable text such as code and templated answers. Because the target model verifies every token, the output matches what it would have produced on its own.

5 / 6

Tools to know

Serving engines: vLLM · SGLang · TensorRT-LLM · Hugging Face TGI · llama.cpp. vLLM or SGLang for general GPU serving; TensorRT-LLM for maximum NVIDIA performance; llama.cpp for CPU and edge. Quantisation: FP8 · AWQ · GPTQ · GGUF (llama.cpp). FP8 first on supported GPUs; INT4 when memory is the hard limit. Kernels: FlashAttention · fused kernels in engines. Usually provided by the engine; check versions match your GPU.

6 / 6

Interview question

“Your LLM feature is too slow. Where do you start?” Measure TTFT versus TPOT. Slow first token means long prompts or queueing: trim context, cache prefixes, add capacity. Slow streaming means decode limits: batching, quantisation, speculative decoding or a smaller model.

Making inference fast and cheap
#ai#deep dive
AI handbook3 / 7
1 / 6

What it is

An embedding model turns text, images or audio into a vector so that similar meanings land close together. Vector search finds the nearest vectors quickly using approximate indexes. Together they power semantic search, recommendations, deduplication and the retrieval half of RAG.

2 / 6

Building semantic search for a service catalogue

The wellness app has 40,000 services across 3,000 clinics. Keyword search fails on ‘something for a stiff neck after long flights’ because no listing contains those words. A short document per service feeds the embedding: title, category, description and benefits. Postgres stores the 768-dimension vectors with pgvector and an HNSW index, alongside city, price and clinic ID for filtering.

3 / 6

Inside it

Embedding model. Maps input to a fixed-length vector. Use the same model, version and preprocessing for documents and queries; changing models means re-embedding everything. Similarity. Usually cosine similarity, or dot product on normalised vectors. Approximate index. HNSW builds a navigable graph for fast, high-recall search; IVF partitions vectors into clusters; product quantisation compresses vectors to save memory at some recall cost.

4 / 6

A hybrid query

Filters run inside both searches so results never include other cities. The reranker is where most of the precision comes from.

5 / 6

Tools to know

Embedding models: Hosted embedding APIs · open models (e.g. BGE, E5, multilingual variants). Benchmark on your queries; check multilingual quality, dimension and cost. Vector stores: pgvector · OpenSearch · Qdrant · Weaviate · Milvus · Pinecone. Start in the database you run; move when scale or filtering demands. Rerankers: Cross-encoder models · hosted rerank APIs. Almost always worth it for top-k quality.

6 / 6

Interview question

“Why combine keyword and vector search?” Vectors miss exact tokens like product codes; keywords miss paraphrases. Fusion gets both, and a reranker sorts the final list.

Embeddings and vector search
#ai#deep dive
Prompting, RAG or fine-tuning?
Tool use, agents and MCP
Integrating LLMs into products
Using AI to accelerate engineering teams