Basics
The whole idea first, then the terms the rest of this page uses.
1.1Whole idea
Fast retrieval casts a wide net but ranks roughly. A cross-encoder reads the question and each candidate together and ranks far better, but too slowly to run on everything. So retrieve many, then rerank a few.
Retrieve 100 with fast search, rerank to 5 with a cross-encoder, and degrade safely on timeout.
1.2Key concepts
One idea per card, in plain words.
Recall comes from the first stage, precision from the second.
Scores one question-passage pair per forward pass. Accurate, but costs scale with candidates.
Bounds how long the reranker may run before the request moves on.
Question
What exactly you must answer, and what to settle before you start.
2.1Question
Recall@100 reaches 95%, but precision@5 sits at 40%. Add a reranker within 150 ms at p95, and never fail a request because of it.
Detailed answer
How it works, and how to build or explain it.
3.1Step by step
The mechanism, in the order you would explain it.
- Retrieve widely
Take the top 100 from hybrid search.
- Score in batches
Batch question-passage pairs on a GPU, and truncate passages to the model's limit.
- Enforce the budget
Race the reranker against a timeout.
- Fall back safely
On timeout or error, keep the retrieval order and record that the rerank failed.
- Measure the lift
Compare precision@5 with and without reranking on your eval set.
3.2Code
Working code for the core of it.
Retrieve widely, rerank precisely
A cross-encoder is accurate but slow, so it only sees the top candidates, and it's allowed to fail.
1export interface Candidate {2 id: string;3 text: string;4}5 6interface Options {7 candidates?: number;8 top?: number;9 timeoutMs?: number;10}11 12/** Retrieve widely, rerank precisely. If the reranker fails or runs late, keep the retrieval order. */13export async function retrieveAndRerank(14 query: string,15 retrieve: (q: string, n: number) => Promise<Candidate[]>,16 rerank: (q: string, docs: Candidate[]) => Promise<number[]>,17 { candidates = 100, top = 5, timeoutMs = 300 }: Options = {}18): Promise<{ results: Candidate[]; reranked: boolean }> {19 const docs = await retrieve(query, candidates);20 const timeout = new Promise<never>((_, reject) => setTimeout(() => reject(new Error("rerank timeout")), timeoutMs));21 try {22 const scores = await Promise.race([rerank(query, docs), timeout]);23 const order = docs.map((d, i) => ({ d, s: scores[i] })).sort((a, b) => b.s - a.s);24 return { results: order.slice(0, top).map((x) => x.d), reranked: true };25 } catch {26 return { results: docs.slice(0, top), reranked: false };27 }28}candidates = 100- Cast the net wide first, because reranking can't recover what retrieval missed.
Promise.race([rerank(...), timeout])- Puts a limit on latency.
reranked: false- On failure, fall back to retrieval order and say so, rather than failing the request.
Tested by npm run test:modules.
Q&A
What interviewers ask next. Answer out loud, then open.
4.1Interview questions
How many candidates?
Why not rerank everything?
Should scores act as thresholds?
Extra depth
Edge cases, counterpoints and what to read next.
5.1Edge cases and counterpoints
Includes the strongest argument against this approach.
Show detailsHide details
- Silent degradation
- Alert on the fallback rate. A permanently failing reranker looks like a small quality drop.
- Counterpoint
- If retrieval already places answers in the top 5, a reranker adds cost and latency for little gain.