/Learn
Deep dives · AI handbook · internals

How an LLM works inside

LearnDeep diveTier 2 · MechanismsAnalyze20 minModel serving

You will be able to: explain how an LLM works inside and answer its interview questions.

Prove it: Speculative Decoding Target-Draft Pair Verifier

AI engineeringDeep dives

Study with AI

Part 1 of 5

Basics

What this part of the system is, what it is responsible for, and the vocabulary you need before the mechanics.

1.1Definition

The one-paragraph summary, then where its responsibility starts and stops.

LLMs map a token sequence to a probability for every possible next token. Generation repeats that function, appending one token at a time. Latency, cost, hallucination and context limits all follow from this loop.

What to understand
  • Text becomes tokens; cost and limits are counted in tokens
  • Each layer mixes information between tokens (attention) and transforms it (MLP)
  • Prefill reads the whole prompt at once; decode writes one token per step
  • The KV cache stores past keys and values so decode is not quadratic
  • Sampling settings shape the output, not the model’s knowledge
Common misconceptions
  • The model does not look facts up; it predicts likely text
  • A bigger context window does not mean it uses all of it equally well
  • Temperature 0 is not a guarantee of identical output across providers or hardware
  • Tokens are not words; Hindi or code can cost more tokens per word than English

1.2Key terms

The words used in the rest of this page.

TermMeaning
TokenA sub-word unit. Roughly ¾ of an English word on average; varies by language and content.
Context windowMaximum tokens of prompt plus output the model accepts in one call.
Prefill / decodeReading the prompt in parallel / generating one token per step.
LogitsRaw scores over the vocabulary before sampling.
ParametersLearned weights; 7B, 70B and so on. Memory ≈ parameters × bytes per weight.
Pre-training / post-trainingLearning to predict text at scale / then tuning to follow instructions and preferences.

1.3Key concepts

Ideas this deep dive relies on, in one sentence each.

01KV cache

Stored attention keys and values per token during generation; it limits concurrent requests per GPU.

02Embedding

A vector representing meaning so similar items are close together.

1.4Follow one request

A concrete example first, so the mechanics have something to attach to.

A user asks the booking assistant: “Move my massage to Friday evening.” The orchestrator builds a prompt of about 2,400 tokens: system instructions, tool definitions, the last few turns and retrieved facts.

The tokenizer turns that text into 2,400 integer IDs using a byte-pair-encoding vocabulary of roughly 100,000 to 200,000 entries. An embedding table turns each ID into a vector of a few thousand numbers.

Prefill: all 2,400 vectors pass through every transformer layer in parallel. In each layer, attention lets every token weigh earlier tokens; the MLP transforms each position. The KV cache saves the keys and values from every layer. This step decides time to first token.

Decode: the final position produces a score (logit) for every token in the vocabulary. The sampler applies temperature and top-p and picks one token, perhaps a tool-call marker. That single token is fed back in, attending to the cached keys and values, and the loop repeats until a stop token. Each step decides the time per output token.

Part 2 of 5

Question

Know what you are reading for. These are the questions this deep dive prepares you to answer, and the rounds where they come up.

2.1Target questions

Keep them in mind while you read the detail.

  1. Why does a long prompt slow the first token but not later tokens as much?
  2. What is the KV cache and why does it matter for serving?
  3. What does temperature actually change?
  4. Why can Hindi text cost more than English?

Model answers are in part 4, after the detail that supports them.

2.2Interview rounds

System design rounds where this component is part of a strong answer.

Part 3 of 5

Detailed answer

How it actually works: the components, the states it moves through, a full run, and the code.

3.1The pipeline inside the model

The parts and how they connect.

INPUTTRANSFORMER × N LAYERSOUTPUTPrompt textsystem · history ·toolsTokenizerBPE → token IDsEmbedding tableID → vectorSelf-attentionQKᵀ/√d · softmax ·VNorm + residualstabilise layersKV cachekeys + values perlayerMLP / MoEper-token transformPositionsRoPERepeat N timese.g. 32–120 layersLM headvector → logitsSamplertemperature · top-pNext tokenappend and repeatvectorsstore K,Vlast positionlogits
ClientServiceData storeAI component
Boxes are components; arrows show calls and data flow.
Tokenizer
Byte-pair encoding splits text into frequent sub-word pieces. Token counts drive price, latency and context limits, so always measure them with the provider’s tokenizer.
Self-attention
Each token produces a query, key and value. Scores are queries times keys, scaled by √d, turned into weights by softmax, and used to mix values. Causal masking stops a token from seeing the future. Grouped-query attention shares keys and values across heads to shrink the KV cache.
MLP and mixture of experts
A feed-forward network transforms each position independently. In mixture-of-experts models, a router sends each token to a few expert MLPs, so only part of the parameters run per token.
Positions
Rotary position embeddings (RoPE) encode order inside attention; context-extension techniques stretch them to longer windows.
KV cache
Saved keys and values for every past token in every layer. It makes decoding linear instead of quadratic, and it is the main memory cost of serving long prompts.
Sampler
Temperature rescales logits; top-p keeps the smallest set of tokens whose probability adds up to p. Lower values make output more predictable, not more correct.

3.2Generating a reply token by token

One run, step by step.

AppTokenizerTransformerKV cacheSamplerprompt (2,400 tokens)token IDsprefill all positionsstore K,V for 2,400 tokenslogits for next tokentoken 1 (TTFT)Decode loop: one token per stepfeed token backread cached K,Vlogitstoken 2, 3, …
Solid arrows are calls; dashed arrows are responses.
  1. Prefill is compute-bound and grows with prompt length; long prompts mean slower first tokens.
  2. Decode is memory-bandwidth-bound: each step reads all weights and the KV cache to produce one token.
  3. This is why output tokens usually cost more than input tokens, and why streaming improves perceived speed.

3.3Code

Working code for the core of it.

Attention in a few lines (TypeScript, one head)

attention-few-lines.tsTypeScript
1type Matrix = number[][];2 3const matmul = (a: Matrix, b: Matrix): Matrix =>4  a.map((row) => b[0].map((_, j) => row.reduce((s, x, k) => s + x * b[k][j], 0)));5const transpose = (m: Matrix): Matrix => m[0].map((_, j) => m.map((row) => row[j]));6 7function softmax(row: number[]): number[] {8  const max = Math.max(...row);                       // subtract the max: exp() never overflows9  const e = row.map((x) => Math.exp(x - max));10  const sum = e.reduce((a, b) => a + b, 0);11  return e.map((x) => x / sum);12}13 14// One causal attention head. x: T tokens × D dims; Wq, Wk, Wv: D × d.15// Multi-head attention runs H of these with d = D / H and concatenates the outputs.16function causalAttention(x: Matrix, Wq: Matrix, Wk: Matrix, Wv: Matrix): Matrix {17  const q = matmul(x, Wq), k = matmul(x, Wk), v = matmul(x, Wv);18  const scale = Math.sqrt(q[0].length);19  const scores = matmul(q, transpose(k)).map((row, t) =>20    row.map((s, u) => (u > t ? -Infinity : s / scale)));   // no peeking at later tokens21  return matmul(scores.map(softmax), v);                     // T × d22}
22 linesUTF-8TypeScript

Sampling with temperature and top-p (TypeScript)

sampling-temperature-topp.tsTypeScript
1// Temperature reshapes the distribution; top-p keeps the smallest set of tokens2// whose probability reaches p. Uses softmax from the attention example.3function sample(logits: number[], temperature = 0.7, topP = 0.9, rand = Math.random): number {4  const probs = softmax(logits.map((l) => l / Math.max(temperature, 1e-5)));5  const ranked = probs.map((p, i) => [p, i] as const).sort((a, b) => b[0] - a[0]);6  const keep: (readonly [number, number])[] = [];7  let mass = 0;8  for (const pair of ranked) {9    keep.push(pair);10    mass += pair[0];11    if (mass >= topP) break;                        // nucleus reached12  }13  let r = rand() * mass;                            // sample within the kept tokens only14  for (const [p, i] of keep) if ((r -= p) <= 0) return i;15  return keep[keep.length - 1][1];16}
16 linesUTF-8TypeScript

KV cache size for any model

kv-cache-size.txtPlain text
1bytes per token = 2 (K and V) × layers × kv_heads × head_dim × bytes_per_value2Llama-3-70B-class: 2 × 80 × 8 × 128 × 2 bytes ≈ 0.33 MB per token at FP1634,000-token conversation ≈ 1.3 GB of GPU memory, per request
3 linesUTF-8Plain text

3.4Tested code, explained

A working implementation, then what each part does and why.

How much memory one token costs

This arithmetic sets how many requests a GPU can serve at once. Weights are fixed, and the KV cache grows with every token.

kv-size.tsTypeScript
1/** Bytes of KV cache one token costs: a key and a value, per layer, per KV head. */2export function kvBytesPerToken(m: { layers: number; kvHeads: number; headDim: number; bytesPerValue: number }): number {3  return 2 * m.layers * m.kvHeads * m.headDim * m.bytesPerValue;4}5 6/** Requests that fit in the memory left after loading the weights. */7export function concurrentRequests(freeGiB: number, tokensPerRequest: number, bytesPerToken: number): number {8  return Math.floor((freeGiB * 1024 ** 3) / (tokensPerRequest * bytesPerToken));9}
9 linesUTF-8TypeScript
2 * m.layers * ...
A key and a value, for each layer, each head and each dimension.
freeGiB
What's left after loading the weights. Divide by the cost of a whole request, and you have your concurrency limit.

Tested by npm run test:modules.

Managing the KV cache in blocks

A sequence's final length is unknown, so reserving contiguous memory wastes most of it. Blocks work like virtual memory pages.

paged-blocks.tsTypeScript
1/** PagedAttention's memory manager: fixed-size blocks, one table per sequence, copy-on-write forks. */2export class BlockManager {3  private free: number[];4  private refs: number[];5  private blockTokens: number;6  private seqs = new Map<string, { blocks: number[]; tokens: number }>();7 8  constructor(totalBlocks: number, blockTokens = 16) {9    this.free = Array.from({ length: totalBlocks }, (_, i) => totalBlocks - 1 - i);10    this.refs = new Array<number>(totalBlocks).fill(0);11    this.blockTokens = blockTokens;12  }13 14  get freeBlocks(): number {15    return this.free.length;16  }17 18  table(seq: string): number[] {19    return [...(this.seqs.get(seq)?.blocks ?? [])];20  }21 22  /** Reserves room for one more token. False means no free block: the scheduler must preempt. */23  addToken(seq: string): boolean {24    const s = this.seqs.get(seq) ?? { blocks: [], tokens: 0 };25    this.seqs.set(seq, s);26    const lastIsFull = s.tokens % this.blockTokens === 0;27    if (lastIsFull) {28      const block = this.free.pop();29      if (block === undefined) return false;30      this.refs[block] = 1;31      s.blocks.push(block);32    } else {33      const last = s.blocks[s.blocks.length - 1];34      if (this.refs[last] > 1) {35        // A forked sequence shares this block: copy before writing.36        const copy = this.free.pop();37        if (copy === undefined) return false;38        this.refs[last]--;39        this.refs[copy] = 1;40        s.blocks[s.blocks.length - 1] = copy;41      }42    }43    s.tokens++;44    return true;45  }46 47  /** Parallel sampling or beam search: the child shares every block until it writes. */48  fork(parent: string, child: string): void {49    const p = this.seqs.get(parent);50    if (!p) throw new Error(`unknown sequence ${parent}`);51    p.blocks.forEach((b) => this.refs[b]++);52    this.seqs.set(child, { blocks: [...p.blocks], tokens: p.tokens });53  }54 55  release(seq: string): void {56    for (const b of this.seqs.get(seq)?.blocks ?? []) {57      if (--this.refs[b] === 0) this.free.push(b);58    }59    this.seqs.delete(seq);60  }61}
61 linesUTF-8TypeScript
s.tokens % this.blockTokens === 0
A new block is needed only when the last one is full, so waste is under one block per sequence.
return false
No free block. The scheduler must preempt something, which is why this returns a value rather than throwing.
this.refs[last] > 1
Copy on write: forked sequences share blocks until one of them writes.
--this.refs[b] === 0
A shared block returns to the free list only when its last user finishes.

Tested by npm run test:modules.

3.5Why internals matter to engineers

Cost and latency

  • You pay per input and output token; output tokens are slower and usually pricier.
  • Long system prompts cost on every call unless the provider caches the prefix.

Quality

  • Models attend unevenly over long contexts; put the most important instructions and facts near the start or end, and retrieve rather than paste everything.
  • Sampling settings change style and variance; evaluation decides them, not intuition.

Capacity planning

  • GPU memory = weights + KV cache; KV cache grows with concurrent users × context length.
  • Throughput depends on batching many decodes together.
Part 4 of 5

Q&A

The questions from part 2, now with model answers. Try each one out loud before opening it.

4.1Interview questions

Why does a long prompt slow the first token but not later tokens as much?
Prefill processes all prompt tokens at once, which is compute-heavy. Later tokens reuse the KV cache and only process one new position each.
What is the KV cache and why does it matter for serving?
Stored keys and values for past tokens in every layer. It avoids recomputation, but its memory grows with context and concurrency, so it limits how many users a GPU can serve.
What does temperature actually change?
It sharpens or flattens the probability distribution before sampling. It changes variety, not knowledge.
Why can Hindi text cost more than English?
Teams train tokenizers mostly on English, so other scripts split into more tokens per word.
Part 5 of 5

Extra depth

Practical context for real projects: which technology to pick, what successful teams do, what to measure and the edge cases to plan for.

5.1Model families and knobs

Show detailsHide details
NeedOptionsWhen to pick
Dense vs mixture of expertsDense: every parameter runs per token. MoE: a few experts per tokenMoE gives large-model quality at lower compute per token, at the cost of more memory.
Context length8k · 32k · 128k · 1M tokensLonger windows simplify prompts but cost more and do not replace retrieval for large corpora.
Reasoning modelsSpend extra tokens thinking before answeringBetter on multi-step problems; slower and more expensive per answer.
Open vs hostedLlama, Mistral, Gemma, Qwen families · hosted frontier modelsOpen weights for control and residency; hosted for top quality and no infrastructure.

5.2Numbers to know

Show detailsHide details
MetricWhy it matters
Time to first token (TTFT)Driven by prompt length and queueing.
Time per output token (TPOT)Driven by model size, batch size and memory bandwidth.
Tokens per request (in / out)The core cost driver.
Context utilisationHow much of the window you actually send; trimming it is the easiest saving.

5.3Failure modes and their causes

Show detailsHide details
CaseHow it is handled
Confident wrong factsThe model predicts plausible text; ground answers with retrieval and tool results.
Instruction ignored in a long promptToo much context dilutes attention; shorten and restructure.
Output cut offHit max output tokens; raise the limit or ask for shorter structure.
Same prompt, different answersSampling randomness and batching effects; lower temperature and evaluate on sets, not single runs.
AI Engineer · How an LLM works inside