Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
The context window holds system prompt, history, retrieved text and the reply. Everything competes, and every token costs money and latency. Compression keeps what the answer needs and drops the rest.
Spend the token budget on evidence: prune sentences and summarise history before they reach the model.
1.2Key concepts
One idea per card, in plain words.
Divides the window between system prompt, history, evidence and reply.
Keeps the most relevant original sentences. Nothing gets rewritten, so citations survive.
Rewrites text shorter. Saves more, but can introduce errors.
Replaces old chat turns with a running summary.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Your assistant sends 12,000 tokens per request and costs too much. Answers cite only a few sentences. Cut the prompt in half without hurting answer quality.
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Measure the split
Log tokens per prompt part. History often outweighs evidence.
- Cap each part
Give system, history, evidence and reply explicit limits.
- Rerank first
Fewer, better passages save more than clever compression.
- Prune sentences
Score sentences against the question, keep the best within budget, and restore reading order.
- Roll up history
Summarise old turns, and keep the last few verbatim.
- Verify on evals
Compare faithfulness and answer quality before and after.
3.2Code
Working code for the core of it.
Fitting context into a budget
Retrieval usually returns more than fits. This keeps the best sentences, in the order they were written.
1export interface Sentence {2 text: string;3 score: number;4}5 6/** Keeps the highest-scoring sentences that fit the budget, then restores reading order. */7export function fitContext(sentences: Sentence[], budget: number, count: (s: string) => number): string[] {8 const ranked = sentences.map((s, i) => ({ ...s, i })).sort((a, b) => b.score - a.score);9 const kept: { text: string; i: number }[] = [];10 let used = 0;11 for (const s of ranked) {12 const cost = count(s.text);13 if (used + cost > budget) continue;14 used += cost;15 kept.push(s);16 }17 return kept.sort((a, b) => a.i - b.i).map((s) => s.text);18}sort((a, b) => b.score - a.score)- Picks by relevance.
continue- Skips one that doesn't fit and tries smaller ones, instead of stopping.
sort((a, b) => a.i - b.i)- Restores reading order, because shuffled sentences confuse a model.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
Why restore reading order?
Does prompt caching change this?
What about model-based compression?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Lost conditions
- Pruning can drop an 'unless' clause. Keep a sentence's neighbours when it contains conditions.
- Counterpoint
- Windows keep growing and prices keep falling. Don't compress before you've measured what it saves.