Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
Users ask the same things in different words. A semantic cache embeds each question, searches earlier questions, and returns the stored answer if one sits close enough. A hit costs milliseconds instead of an LLM call. A false hit serves a wrong answer, so the threshold decides everything.
Answers repeat questions from a vector cache, cutting LLM cost, as long as false hits stay rare.
1.2Key concepts
One idea per card, in plain words.
Sets how close a cached question must be. Too low serves wrong answers; too high never hits.
Treats two different questions as the same: 'cancel my order' against 'cancel my subscription'.
Limits hits to answers this user may see, usually by tenant and sometimes by role.
Expires entries so stale answers leave the cache.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Your support assistant handles 2 million LLM calls a day at $0.004 each, and you suspect many repeat. Design a cache that cuts cost without serving wrong answers.
2.2Clarify first
Each answer removes a guess.
| You ask | Answer |
|---|---|
| Are answers personalised? | Some use account data. Those must bypass the cache. |
| How fresh must they be? | Policy answers can age for an hour; order status cannot be cached. |
| What repeat rate? | Measure first by replaying a day of logs against the cache offline. |
Detailed answer
How it works, and how to build or explain it.
3.1Architecture
The components and how they connect.
3.2Step by step
The mechanism, in the order you would explain it.
- Decide what to cache
Classify first. Cache general questions; bypass anything that uses personal data or live state.
- Embed and search
Embed the question and search within the tenant's tag. This adds 5–15 ms to every request.
- Serve or call
Above the threshold, return the stored answer. Below it, call the LLM and store the result with a TTL.
- Set the threshold from data
Label a few hundred question pairs, plot false hits against hit rate, and pick the point you can defend.
- Invalidate by source
Tag entries with the documents behind them, and purge when a document changes.
- Shadow-check hits
Sample hits and also call the LLM. Compare the answers to catch threshold drift.
3.3Savings at a 30% hit rate
| Item | Per day |
|---|---|
| LLM calls avoided | 600,000 |
| LLM cost saved | $2,400 |
| Added embeddings | 2 million small calls, a few dollars |
3.4Code
Working code for the core of it.
A cache keyed by meaning
Similar questions can share an answer, if you're careful about who shares with whom.
1const cosine = (a: number[], b: number[]) => {2 const dot = a.reduce((s, x, i) => s + x * b[i], 0);3 return dot / (Math.hypot(...a) * Math.hypot(...b));4};5 6interface Entry<V> {7 vector: number[];8 value: V;9 expires: number;10}11 12/** Answers a question from a previous similar question. Tenants never share entries. */13export class SemanticCache<V> {14 private entries = new Map<string, Entry<V>[]>();15 private embed: (text: string) => Promise<number[]>;16 private threshold: number;17 private ttlMs: number;18 private now: () => number;19 20 constructor(embed: (text: string) => Promise<number[]>, opts: { threshold?: number; ttlMs?: number; now?: () => number } = {}) {21 this.embed = embed;22 this.threshold = opts.threshold ?? 0.92;23 this.ttlMs = opts.ttlMs ?? 3_600_000;24 this.now = opts.now ?? Date.now;25 }26 27 async get(tenant: string, query: string): Promise<V | null> {28 const v = await this.embed(query);29 const live = (this.entries.get(tenant) ?? []).filter((e) => e.expires > this.now());30 this.entries.set(tenant, live);31 let best: Entry<V> | null = null;32 let bestScore = this.threshold;33 for (const e of live) {34 const score = cosine(v, e.vector);35 if (score >= bestScore) {36 best = e;37 bestScore = score;38 }39 }40 return best?.value ?? null;41 }42 43 async put(tenant: string, query: string, value: V): Promise<void> {44 const entry = { vector: await this.embed(query), value, expires: this.now() + this.ttlMs };45 this.entries.set(tenant, [...(this.entries.get(tenant) ?? []), entry]);46 }47}threshold ?? 0.92- Set this too low, and one question gets the answer to a different one. Tune it on real data.
this.entries.get(tenant)- Tenants never share entries, because an answer may contain private data.
e.expires > this.now()- Answers go stale, so everything has a time limit.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
How do you handle negation?
Where can it go wrong for privacy?
How do you measure success?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Prompt changes
- A new prompt version changes answers. Add the version to the cache scope.
- Streaming
- Replay cached answers at a steady pace, so hits and misses look the same in the UI.
- Counterpoint
- Below about a 10% repeat rate the cache adds cost, latency and risk for little return. An exact-match cache may be enough.