RAG Architectures

The popular cheat sheets present these as a flat list of parallel systems. They are not — and the flattening is why the same system keeps appearing under several names.

Three axes, not one list

Control flow: How many times retrieval fires, and what decides.
Index structure: What the retrieval unit is — a flat chunk, an entity graph, a summary.
Representation: What a document is in the index — one vector, many, or an image.

Read along those axes, most of the catalogue collapses. Below, each architecture is given the mathematics that constitutes it, the condition under which it wins, and — the part the cheat sheets leave out — the condition under which it provably fails.

The eight stages

Every system below is a path through these. The architectures differ by a handful of edges over one shared skeleton.

01 Ingest

Segmenting a corpus into retrievable units — the choice that fixes what can ever be retrieved.
02 Index

Building the structure searched at query time: a flat vector store, a quantized codebook, a navigable graph, or a community-summarized entity graph.
03 Retrieve

Scoring the corpus against a probe and returning candidates — one similarity functional, evaluated sublinearly.
04 Fuse

Combining several independent rankings into one, on ranks rather than scores so the combination is scale-invariant.
05 Rerank

Rescoring a shortlist with a model too expensive to run over the corpus — the cascade that buys precision with compute.
06 Select

Choosing which candidates actually enter the context window, and which strategy to spend on this query at all.
07 Generate

Conditioning the answer on the selected context — the step whose failures retrieval metrics cannot see.
08 Evaluate

Measuring what the system did, as an estimator with a standard error rather than a number on a slide.

The architectures

Each of these changes the structure of the system: how many times retrieval fires, what the retrieval unit is, or what a document is in the index. They are not interchangeable, and none of them dominates.

Naive RAG

Composed from several

also called Vanilla RAG · single-pass RAG · standard RAG

Ingest
Index
Retrieve
Fuse
Rerank
Select
Generate
Evaluate

ingest → index → retrieve → generate — Retrieval fires once; stages run in order.

Mechanism: Retrieval is a fixed preprocessing step that fires exactly once: the context is the top-k of a single similarity functional, chosen before any generation begins.
Wins when: The answer lives in one passage the query names more or less directly — a question phrased in the corpus’s own vocabulary, with a single local answer.
Fails when: Two failures, and they are different. A query phrased unlike its source documents lands off the document manifold, so the nearest neighbors are the wrong ones. And an answer that requires composing evidence no single passage carries is unreachable at any k, because more context does not manufacture a relation the corpus never stated.

The mathematics it rests on

foundational retrieval-foundations

The Retrieval Problem: Relevance, Similarity, and the Geometry of Scores

Retrieval as ranking by a relevance functional — and the three similarity scores that agree on the sphere and diverge off it

Retrieval is ranking: given a query, score every document by a relevance functional rel(q, d) and return the top k, a set-valued operator on the resulting order. Because only the order matters, relevance is ordinal even though scores are cardinal — the ranking is invariant under any strictly monotone transform of the score, a fact we will lean on repeatedly. We then study the three similarity functions retrieval actually uses — Euclidean distance, the dot product, and cosine similarity — through the single identity ||a-b||^2 = ||a||^2 + ||b||^2 - 2<a,b>. On the unit sphere this identity collapses the three into one: ranking by Euclidean distance, by dot product, and by cosine all induce the same order. Off the sphere they diverge, because magnitude matters for the dot product but is quotiented away by cosine — the divergence that motivates normalization throughout the rest of the curriculum. We separate which of these are true metrics (Euclidean is; cosine distance violates the triangle inequality; the dot product is not a metric at all) because the triangle inequality is exactly the structure that tree- and graph-based approximate-nearest-neighbor indexes later exploit. The level sets make the picture geometric: equal-score loci are hyperplanes for the dot product, spheres for Euclidean distance, and cones for cosine. An interactive similarity playground and a tested, deterministic implementation accompany the derivation, with a finance example showing the same query ranking documents differently under dot product and cosine when document norms vary.

intermediate embedding-geometry

Chunking as a Segmentation and Optimization Problem

Where to cut a document, posed as a coherence-maximizing segmentation with an exact dynamic-programming optimum — and the proxy it secretly optimizes

Chunking is the first thing a retrieval pipeline does to a document and the least mathematized — the default is to split every few hundred tokens and move on. Posed properly it is a one-dimensional segmentation problem: choose boundaries that maximize within-chunk coherence. We show that coherence has a clean closed form — for L2-normalized sentence embeddings the within-segment cost is the segment length minus the norm of the sum of its embeddings, which is the length times one minus the mean resultant length, the von Mises-Fisher concentration statistic from the prerequisite topic — so minimizing total cost carves the document into tight clusters on the sphere. Because the cost is additive across segments, the globally optimal segmentation is computed exactly by an O(n squared) dynamic program, which we prove optimal and verify against brute force; TextTiling's greedy depth scores and fixed-size chunking are heuristics that cannot beat it. We then confront the honest catch that makes this more than an algorithms exercise: coherence is a proxy for downstream retrieval quality, and the harness shows boundary-recovery F1 peaking at the true section count while the coherence cost keeps falling under over-segmentation. An interactive Chunking Laboratory and a tested implementation accompany the derivation, with a synthetic 10-K filing on which the optimal segmentation recovers the section structure that fixed-size chunking misses.

1 prerequisite

advanced neural-retrieval

Dense Retrieval and Dual Encoders: Architecture, Expressivity, and the Cost of Negatives

Why a query tower and a document tower trained to a separable score let you precompute every document, collapse retrieval to a single maximum-inner-product lookup, and represent exactly the relevance patterns of rank at most d by Eckart–Young — and why one batch of 2B encodings secretly buys B² training comparisons

InfoNCE told us how a dual encoder is trained; this topic asks what that architecture can represent and what it costs. A dual encoder is two towers, a query encoder and a document encoder, whose relevance score is the separable inner product of their outputs. We show that separability is the whole game: because the score factorizes, every document vector can be precomputed and stored once, and a query at serving time reduces to a single argmax over inner products — maximum-inner-product search, the problem the MIPS-hardness topic analyzes. We then ask the expressivity question. Stacking the scores into a query-by-document relevance matrix, a d-dimensional dual encoder can realize exactly the matrices of rank at most d; the Eckart-Young-Mirsky theorem makes the truncated SVD the optimal rank-d approximation to any target, so d is a clean upper bound on what the architecture can express, and retrieval accuracy collapses when d falls below the relevance pattern's intrinsic rank. We flag honestly that rank is an upper bound, not the tight sign-rank measure of how many dimensions relevance needs. Finally we account for training cost: a batch of B query-document pairs costs 2B encoder forward passes but produces a B-by-B Gram matrix of similarities, yielding B-squared-minus-B in-batch negatives at no extra encoding cost — the quadratic-utility-from-linear-cost law that makes in-batch-negative training the default. A laboratory reuses the InfoNCE finance encoder to show the precompute-then-MIPS path, the rank-d reconstruction of a relevance matrix, and the in-batch Gram trick; a tested notebook owns every number.

Start here 2 prerequisites

advanced generation-grounding

Retrieval versus Long Context: Attention Complexity and Positional Bias

If the context window is large enough to hold everything, why retrieve at all? Because attention costs (kL)² — quadratic in the tokens read — and because more context is not better: once the answer is in hand, extra passages are redundant at best and same-sector distractors at worst, so answer quality peaks at the smallest covering context and declines as you stuff the window, while a relevant passage buried in the middle is read at attenuated attention — a soft erasure. The right move is not a bigger window but a better-chosen one.

Modern context windows are large enough to hold a whole filing, which invites a tempting shortcut: skip retrieval and stuff everything in. This topic gives the mathematics of why that loses. The first reason is the rate: full self-attention over a context of k passages of L tokens forms a (kL)×(kL) score matrix, so the arithmetic cost is Θ((kL)²) — doubling the context quadruples the compute, and FlashAttention lowers the memory to O(n) but leaves the FLOPs untouched. The second reason is the distortion. We read the top-k retrieved passages under a finite attention budget — softmax weights w_j that sum to one — and combine their evidence additively, generalizing the von Mises–Fisher answer model the PMI and noisy-channel topics built: p(a|q,C_k) = softmax((⟨q,μ_a⟩ + Σ_j w_j⟨d_j,μ_a⟩)/τ). On a finance corpus of sectors of confusable companies, each answer carries several relevant passages plus a shell of same-sector distractors. Even with the answer reliably retrieved (recall@1 ≈ 1), answer quality Q(k) = E[p(a*|q,C_k)] is highest at the smallest covering context and declines monotonically: while the top-k is all-relevant the extra passages are redundant — a second filing of the same company moves belief almost not at all, the diminishing-returns result imported from the PMI topic — and once same-sector distractors enter, they steal attention budget and inject wrong-company evidence, so precision falls and the answer entropy H(A|context) rises toward the prior. Recall climbs while precision falls, and answer quality tracks precision, not recall: chasing recall@k by enlarging the window degrades the answer. Read against cost, the focused-retrieval point Pareto-dominates stuffing — quadratically more compute for strictly worse answers. The third pillar is positional: a relevant passage buried in the middle of a long context is read at attenuated attention (lost-in-the-middle), a soft erasure that drops quality even though the passage was retrieved. The answer model is the same synthetic softmax stand-in the previous topics used, so every quality number is exact for the model and illustrative of a real generator; a tested notebook owns every number, and the quadratic cost, the diminishing returns, and the monotone decline are asserted, not asserted-about.

1 prerequisite

Hybrid RAG

Composed from several

also called hybrid search · sparse + dense · Advanced RAG

Ingest
Index
Retrieve
Fuse
Rerank
Select
Generate
Evaluate

ingest → index → retrieve → fuse → generate — Retrieval fires once; stages run in order.

Mechanism: Two or more retrieval legs with different failure modes run independently and are combined on ranks rather than scores, which makes the combination invariant to each leg’s scale.
Wins when: The legs are partial views that miss different documents. Fusion gain grows as the legs de-correlate, because what one leg misses another has already ranked.
Fails when: When the legs form a quality ladder — several monotone approximations of one underlying score — the best leg dominates and fusion only adds noise, so the gain is negative. "Fused beats best leg" is not a theorem. Even the dominated-leg flip needs a false positive endorsed by both legs: under the usual constant, one top vote is worth less than two mediocre ones.

The mathematics it rests on

intermediate probabilistic-ir

BM25 and the Binary Independence Model

How probability theory turns term counts into a ranking function — and why saturation and length normalization are the two ideas that matter

BM25 is the strongest lexical retrieval baseline in information retrieval, and it is not an arbitrary formula: its inverse-document-frequency factor falls out of the Binary Independence Model with a Jeffreys prior, and its term-frequency saturation is motivated by the 2-Poisson eliteness model. We derive the Robertson–Spärck-Jones weight from the Probability Ranking Principle, show how smoothed IDF emerges, motivate the saturating tf transform and the document-length normalization, assemble the BM25 scoring function, and prove its limit behavior (k₁→0 recovers the binary model, k₁→∞ recovers length-normalized raw term frequency, b interpolates length normalization). An interactive scoring laboratory and a from-scratch NumPy implementation whose tests verify these limits accompany the derivation, with a worked finance example over earnings-call transcripts and 10-K filings.

3 prerequisites

advanced ranking-fusion

Rank Fusion: Reciprocal Rank Fusion and the Geometry of Rank Aggregation

Why combining ranked lists by position beats combining them by score — and how close a cheap heuristic gets to the optimal consensus

Hybrid retrieval runs two retrievers — a lexical one like BM25 and a dense one — and must combine their rankings. The two live on incompatible score scales, so adding scores (CombSUM) lets one swamp the other; Reciprocal Rank Fusion (RRF) instead combines by position, reading only each document's rank in each list. We show that this makes RRF invariant to any strictly monotone rescaling of either retriever's scores, that as its constant k grows the RRF order converges to the Borda count, and that rank fusion opens onto the real mathematics of rank aggregation: permutations as points in a metric space under the Kendall-τ distance, the Spearman footrule, the Diaconis–Graham inequality relating them, the Kemeny consensus as the median permutation, and the footrule-optimal aggregate as a polynomial-time 2-approximation of the NP-hard Kemeny optimum. An interactive Fusion Laboratory and a from-scratch implementation whose tests verify every claim accompany the derivation, with a worked finance example fusing a BM25 ranking over 10-K text with a dense ranking over earnings-call passages.

Start here 1 prerequisite

advanced neural-retrieval

Dense Retrieval and Dual Encoders: Architecture, Expressivity, and the Cost of Negatives

2 prerequisites

advanced rag-information-theory

Capstone: The Mathematics of a Production Multimodal Financial RAG System

The retrieval stack arrived as a dozen separate topics — lexical scoring, dense MIPS, IVF and PQ and HNSW, late interaction served by PLAID, reciprocal-rank fusion; this capstone composes the published stack into one finance pipeline and proves the three laws that govern it end to end — cascade recall multiplies across stages, hybrid fusion gains exactly when its legs de-correlate, and a fixed compute budget is allocated across heterogeneous per-modality legs by water-filling — with the single exact statement that a full-budget pipeline recovers brute-force retrieval, and everything above it a heuristic speed-for-recall trade measured honestly on a finance frontier

The retrieval half of a production RAG system arrived as a dozen topics; this capstone composes the published stack — lexical scoring, dense MIPS, IVF/PQ/HNSW candidate generation, PLAID-served late interaction, reciprocal-rank fusion — into one finance pipeline and proves the three laws that govern it end to end. Cascade recall multiplies across independent stages, a conservative lower bound the correlated real legs sit above; hybrid fusion beats its best leg by a margin that grows as the legs de-correlate, and fails when a false positive is co-endorsed by two legs; and a fixed compute budget is allocated across heterogeneous per-modality legs by water-filling, equalizing the marginal recall-per-cost. The one exact statement is the collapse anchor — a full-budget pipeline is brute-force retrieval — and a tested notebook that imports the fusion, PLAID, IVF, and over-fetch code owns every number on the finance frontier. The generation and grounding layer above retrieval is named as future work, not derived.

5 prerequisites

Multimodal RAG

Composed from several

also called cross-modal RAG · vision RAG

Same mechanism as Hybrid RAG. The legs differ; the topology does not. It is listed separately because the name is used separately, not because it is a distinct design.

Ingest
Index
Retrieve
Fuse
Rerank
Select
Generate
Evaluate

ingest → index → retrieve → fuse → generate — Retrieval fires once; stages run in order.

Mechanism: Hybrid’s fusion mechanism applied over heterogeneous encoders: each modality contributes its own leg, and the legs meet either in one shared embedding space or at the rank level.
Wins when: Evidence is genuinely distributed across modalities — a figure in a table, a qualification in a transcript, a trend only a chart shows.
Fails when: The modality gap. Separately trained encoders place text and images in disjoint cones, so a cross-modal cosine is dominated by the gap direction rather than by relevance. The gap is invisible to inner-product ranking, which is exactly why it goes unnoticed until the scores are used for anything but sorting.

The mathematics it rests on

advanced neural-retrieval

Cross-Modal Contrastive Alignment and the Modality Gap

Train a text tower and a chart tower with the same symmetric CLIP loss and the two modalities settle into disjoint cones on the sphere — a modality gap between the centroids that contrastive training shrinks but never quite closes. We prove the gap is the coherent, rank-one part of cross-modal misalignment (an orthogonal split L_align = gap² + dispersion) and that it is INVISIBLE to maximum-inner-product ranking: a shared offset is a per-query constant, so recall is exactly gap-invariant. The gap is a calibration artifact — it shifts absolute similarities, never the order — and lower training temperature preserves a larger residual gap.

A multimodal retrieval system trains two encoders — a text tower and a chart tower — with the same symmetric contrastive (CLIP) loss, so a text query can retrieve a chart of the same company. Even after training, the two modalities occupy disjoint cones on the sphere: a measurable modality gap between the text and chart centroids. This topic reads the gap three ways. First, an orthogonal decomposition: the cross-modal alignment loss splits exactly as L_align = gap² + dispersion, a Frobenius-Pythagoras decomposition of the per-pair difference matrix into its coherent rank-one part (the gap) and its incoherent complement, so the gap is bounded by the alignment loss and contrastive training can shrink the loss while a coherent gap survives. Second, the headline: the gap is invisible to maximum-inner-product ranking. Shifting one modality by the shared gap vector changes every cross-modal score by a per-query constant, leaving the argsort — and recall@k — exactly invariant, so the gap is a calibration artifact on absolute similarities, not a ranking defect. Cosine retrieval, which renormalizes, is the honest exception. Third, the cone effect: a deterministic full-batch descent on the symmetric loss closes the gap at moderate temperature, and lower temperature preserves a larger residual gap. The finance thread runs throughout: a desk embedding 10-K text and price charts of the same companies retrieves correctly through the gap but miscalibrates a fixed relevance threshold across it.

Start here 2 prerequisites

advanced rag-information-theory

Capstone: The Mathematics of a Production Multimodal Financial RAG System

5 prerequisites

advanced ranking-fusion

Rank Fusion: Reciprocal Rank Fusion and the Geometry of Rank Aggregation

Why combining ranked lists by position beats combining them by score — and how close a cheap heuristic gets to the optimal consensus

1 prerequisite

HyDE

One topic is this

also called Hypothetical Document Embeddings · query transformation · query rewriting

Ingest
Index
Retrieve
Fuse
Rerank
Select
Generate
Evaluate

generate → retrieve → generate — Retrieval fires once; stages run in order.

Mechanism: Draft a hypothetical answer document, embed that instead of the query, and retrieve with it — moving the probe off query space and onto the document manifold. It is the only common architecture in which generation runs upstream of retrieval.
Wins when: Query–document distribution shift: questions phrased unlike the corpus they must match. The correction is independent of how far off-manifold the query started, because the bare query’s position is discarded rather than adjusted.
Fails when: It spends a generation call before any retrieval, and buys nothing when the query was already on-manifold — so it falls off the cost frontier first. A generator that drafts the wrong entity at some rate imposes a recall ceiling that averaging more drafts cannot break, because the error is a bias, not variance.

The mathematics it rests on

intermediate probabilistic-ir

Relevance Feedback and Query Expansion: Rocchio and RM3

Closing the loop — using the documents you just retrieved to repair the query, and why doing too much of it drifts off topic

A query and its relevant documents often use different words — a filing's risk disclosure says 'outlook' and 'forecast' where the analyst typed 'guidance' — and lexical retrieval cannot match what is not there. Relevance feedback repairs the query using the documents already retrieved. We derive Rocchio, the vector-space update that moves the query toward the centroid of the relevant documents, and the language-model relevance models RM1 and RM3, which estimate an expanded query distribution from the top documents weighted by their query likelihood and re-score by the KL/cross-entropy view. We prove Rocchio's centroid-optimality and RM3's interpolation limits, and we make the central honest point executable: pseudo-relevance feedback assumes the top documents are relevant, so a little feedback bridges the vocabulary gap and lifts recall, but too much pulls in off-topic terms and the query drifts. An interactive feedback laboratory and a from-scratch implementation whose tests assert both the gain and the drift accompany the derivation, with a finance case study over filings and transcripts.

2 prerequisites

advanced generation-grounding

Query Transformation and HyDE: Correcting Distribution Shift in Embedding Space

A query is question-shaped and a document is answer-shaped, so a bare query sits off the document manifold and its nearest documents are mediocre. HyDE writes a hypothetical answer, embeds that, and retrieves real documents near it — landing back inside the manifold. It is pseudo-relevance feedback with a generated rather than a retrieved centroid, and it trades query-document mismatch for one honest new cost: generation bias, which no amount of Monte-Carlo averaging removes.

In a dual encoder the query and document encoders share an embedding space, but the two distributions do not coincide on the sphere: a query is question-shaped, an answer is answer-shaped, so a bare query lands off the document manifold and retrieves mediocre neighbors. HyDE (Gao et al., 2023) corrects this distribution shift without any relevance labels — it asks a language model for a hypothetical answer document, embeds that, and retrieves real documents near it. We formalize HyDE on the dense-retrieval finance geometry as two things at once: a distribution-shift correction (the generated proxy lands inside the document manifold, so retrieval recovers the answer at any shift) and the neural generalization of pseudo-relevance feedback (Rocchio/RM3 expand a query toward the centroid of retrieved documents; HyDE expands it toward the centroid of generated ones). Averaging k hypotheticals is a Monte-Carlo estimate of the answer centroid whose variance falls toward the 1/k rate; but if the generator hallucinates on a fraction p of queries, the estimate is consistent for the wrong center and recall plateaus at a ceiling near 1 − p. That irreducible generation bias is the honest price HyDE pays for closing the gap.

Start here 2 prerequisites

advanced neural-retrieval

Dense Retrieval and Dual Encoders: Architecture, Expressivity, and the Cost of Negatives

2 prerequisites

Corrective RAG

Composed from several

also called CRAG · Self-RAG · self-reflective RAG · evidence grading

Ingest
Index
Retrieve
Fuse
Rerank
Select
Generate
Evaluate

retrieve → evaluate → retrieve → generate — A stage sends control backwards — the number of retrievals depends on the data.

Mechanism: A grader scores the retrieved evidence before generation; evidence that fails triggers re-retrieval or a fallback. The architecture is a cycle whose gate is a classifier, so the system is only as good as that classifier is calibrated.
Wins when: A query mix containing retrieval failures that are actually detectable, together with a fallback that helps. Both halves are required, and the second is the one usually assumed.
Fails when: Three, all quiet. A grader that separates perfectly makes correction vacuous — it is just always doing the right thing, and the comparison measures nothing. An uncalibrated grader fires on good retrievals, and a false-positive correction moves the query off target, so correction strictly hurts. And the confidence being thresholded is typically far more confident than it is accurate.

The mathematics it rests on

advanced generation-grounding

Faithfulness and Groundedness as Measurable Quantities

HyDE modeled the quality of a generated answer with a single hallucination rate p. Here we stop modeling it with a knob and measure it on the text itself — as two numbers, not one. Faithfulness is the precision of an answer's atomic claims against the retrieved context (what fraction of what you said is supported); groundedness is the recall (what fraction of the supportable facts you used). They diverge, a noisy judge measures both with a bias we debias and a confidence we calibrate, and trading coverage for guaranteed faithfulness is the abstention frontier.

HyDE modeled the quality of its generated hypothetical with one hallucination rate p and showed no amount of averaging could remove it. This topic stops modeling generation quality with a knob and measures it on the generated text — as two numbers, not one. We decompose an answer into atomic claims and define faithfulness as the precision of those claims against the retrieved context and groundedness as the recall of the supportable facts; together they are the precision–recall curve of generation, the evaluation layer's pair of metrics read over claims instead of documents. The two diverge: a terse answer is faithful but thin, a verbose one covers everything but invents figures, and a single factuality score hides the trade. Because the measurement is an LLM judge — a noisy Bernoulli instrument — the raw faithfulness number is biased, so we debias it with Rogan–Gladen and calibrate the judge's confidence with ECE, Platt, and isotonic recalibration before that confidence can drive a cut. Raising the cut drops low-confidence claims, lifting precision and lowering recall along the frontier, and trading coverage for a guaranteed faithfulness under a distribution-free conformal back-off is exactly the abstention frontier the next topic studies. Finally we read grounding in bits: a supported claim is one whose pointwise mutual information with the context is positive, a hallucination one whose PMI is non-positive. A tested notebook owns every number.

Start here 2 prerequisites

advanced retrieval-evaluation

Conformal Factuality: Distribution-Free Correctness Guarantees for Generation

The evaluation layer produced numbers with error bars; the judge produced a calibrated confidence that a generated claim is supported. This topic turns that confidence into a guarantee — conformal prediction converts any nonconformity score into finite-sample coverage under exchangeability alone. We back off unsupported claims with a split-conformal threshold, control the false-claim rate with conformal risk control, and watch the guarantee break under drift and be repaired by reweighting. The finance thread: a retrieval-augmented system over filings that abstains rather than hallucinate a figure, at an error rate an auditor could sign off on.

The evaluation layer treated every retrieval number as an estimator and the LLM judge as a noisy instrument whose verdicts we debias and whose confidence we calibrate. That leaves a sharper question for generation: not how good a number is, but whether we can sign a guarantee on the answer we emit. Conformal prediction answers it. Given any nonconformity score — here the calibrated judge confidence that a claim is supported — split conformal sets a threshold from a held-out calibration set so that a genuinely faithful claim survives with probability at least one minus alpha, a finite-sample coverage guarantee that assumes only exchangeability, no model of when the language model hallucinates. We build the score, the per-claim back-off that removes the least-confident claims, and the recall guarantee that follows verbatim from the marginal-coverage theorem. But recall is not precision: a lenient judge endorses unsupported claims at high confidence, so the recall guarantee lets the false-claim rate run uncontrolled — on this corpus nearly a quarter of retained claims are hallucinations at the strictest level. Controlling the false-claim rate needs a guarantee on a monotone loss, which is conformal risk control: the per-slot false-claim rate, with its fixed denominator, is non-increasing in the back-off threshold, and the risk-control threshold holds its expectation at or below alpha. The fraction-of-retained loss, by contrast, is not monotone and silently voids the guarantee — a trap we exhibit numerically. A perfect judge collapses the whole construction onto precision at k, the metric the evaluation layer was built on. Finally the guarantee is marginal and rests on exchangeability, which covariate shift breaks: the threshold calibrated on yesterday's query mix under-covers a shifted deployment, and weighted conformal with the known likelihood ratio restores coverage. The finance thread is a retrieval-augmented system over filings that backs off unsupported claims and abstains rather than hallucinate a figure, at a guaranteed error rate. A tested notebook owns every number.

2 prerequisites

advanced retrieval-evaluation

LLM-as-Judge and Faithfulness: RAGAS as a Family of Estimators

The evaluation track measured retrieval against fixed relevance labels. Generation has no label, so we hire an LLM to judge — and the judge is a noisy instrument with its own sensitivity, specificity, bias, and variance. We rebuild every RAGAS metric as an estimator you must correct for the instrument: debias the verdict with known error rates, agree on the protocol with chance-corrected reliability, calibrate the confidence, price the irreducible judge-variance floor, and recover the error rates with no gold labels at all.

Retrieval evaluation rested on fixed relevance labels; generation has none. The standard workaround hires a large language model as a judge, and RAGAS turns its verdicts into metrics — faithfulness, answer relevance, context precision. This topic treats the judge as what it is: a noisy measurement instrument, and every RAGAS metric as an estimator built from its verdicts. Faithfulness is the sample mean of the judge's per-claim support verdicts, and because the judge has imperfect sensitivity and specificity, that mean is a biased estimator of the latent grounding prevalence. The Rogan–Gladen correction inverts the error model to recover an unbiased estimate, but its variance is inflated by the inverse square of the judge's Youden index, so a near-useless judge cannot be debiased — only amplified. Trusting a judge first means agreeing with it: Cohen's kappa chance-corrects raw agreement, but the kappa paradox shows high agreement collapsing to near-zero kappa under the skewed marginals typical of 'most claims are supported', so we read Gwet's AC1 and Krippendorff's alpha alongside it, and decompose the variance to expose an intraclass-correlation reliability and a judge-variance floor that more queries cannot lower. We calibrate the judge's stated confidence with the prerequisite's reliability-diagram machinery, detect position bias with a paired swap test, and recover each judge's error rates with no gold set through Dawid–Skene latent-class EM — closing the loop with the correction that opened the topic. A tested notebook owns every number.

1 prerequisite

advanced retrieval-evaluation

Score Calibration, Drift Detection, and Significance Testing for Retrieval

The prerequisites measured a metric as an estimator and left one question open: is an observed gap real? We close it with the right instrument — a paired significance test — and then generalize the same two-distribution comparison to two more production questions: is a retrieval score calibrated as a probability, and has the distribution drifted since we last looked? One idea runs through all three: difference two distributions of per-query quantities and ask whether the difference is zero.

Set metrics and NDCG framed every retrieval number as a sample mean with a standard error, and both left the same question hanging: when two systems' confidence intervals overlap, is the gap real or sampling noise? The crude overlapping-intervals read is conservative because it ignores that the two systems are measured on the SAME queries. The paired significance test exploits that pairing — differencing query-by-query cancels the shared per-query difficulty, so the variance of the difference is far below the sum of the two variances, and the test resolves a gap at a fraction of the queries. We build the paired t-test, cross-check it against the permutation and bootstrap tests that make no normality assumption, correct for the three simultaneous leg comparisons, and replace the prerequisites' crude query-count with a power calculation — closing the NDCG cliffhanger exactly: the closest pair needs 116 queries for 80% power, not the 185 the overlapping intervals suggested. The same two-distribution comparison then answers two more questions. Calibration asks whether a retrieval score behaves like a probability: the reliability diagram plots empirical relevance against confidence, the expected calibration error is the area off the diagonal, and raw cosine and MaxSim scores are wildly over-confident — which is exactly why rank fusion discards scores and fuses ranks. Platt scaling and isotonic regression pull the scores onto the diagonal while leaving the ranking untouched, so calibration is orthogonal to the ranking metrics. Drift asks whether the distribution has moved since a reference window: the Kolmogorov–Smirnov statistic is the sup-gap between two empirical CDFs and the population stability index is a symmetrized KL divergence, the credit-risk standard. A silent quality decay the aggregate mean cannot distinguish is caught by the paired test; a pure covariate shift fires the input drift alarm with no quality loss at all, so input monitoring alone cannot diagnose decay. A tested notebook owns every number.

1 prerequisite

Graph RAG

One topic is this

also called GraphRAG · knowledge-graph RAG · entity-graph retrieval

Ingest
Index
Retrieve
Fuse
Rerank
Select
Generate
Evaluate

ingest → index → retrieve → generate — Retrieval fires once; stages run in order.

Mechanism: The retrieval unit stops being a flat chunk. Entities and relations are extracted into a graph, partitioned into communities by maximizing modularity, and summarized offline — so most of the work moves from query time to index time.
Wins when: Global or thematic queries that require aggregating over a community, where no single passage carries the answer and the summary is the only object that does.
Fails when: It loses to plain retrieval on simple fact lookup, at up to tens of thousands of tokens per query against a few hundred. And community detection has an information-theoretic threshold: below it, no algorithm recovers the planted structure, so the failure is not one a better implementation can fix.

The mathematics it rests on

intermediate embedding-geometry

Chunking as a Segmentation and Optimization Problem

Where to cut a document, posed as a coherence-maximizing segmentation with an exact dynamic-programming optimum — and the proxy it secretly optimizes

1 prerequisite

advanced rag-information-theory

GraphRAG: Community Detection and the Modularity of Knowledge

Multi-hop retrieval found a path; the relation 'retrievable-from' it traced has by now drawn a graph over the whole corpus, and a global, sensemaking question — what are the dominant themes across all of these filings? — has an answer that is a property of that graph's partition, not of any document or path. This is the mathematics of the partition: modularity scores it against a degree-preserving null and the modularity matrix's leading eigenvector relaxes the optimal split; the resolution limit caps the scale modularity can see; the stochastic block model's detectability threshold says when a thematic decomposition exists to be found at all; and Louvain and Leiden are the NP-hard-forced heuristics, with Leiden's refinement guaranteeing the connected communities Louvain cannot.

Multi-hop retrieval answered a local question — a path from a query to a specific answer document. A global, sensemaking question — what are the dominant themes across the whole corpus? — has no such answer: it is a property of the corpus's partition into communities, and no single chunk and no single path can carry it. GraphRAG builds an entity graph from the corpus and partitions it; this topic is the mathematics of that partition. Newman–Girvan modularity scores a partition against the degree-preserving configuration-model null, the excess of within-community edge weight over chance; on the finance entity graph the planted sectors score 0.778 against a random partition's −0.04. For a bipartition the modularity matrix B = A − kkᵀ/2m has rows summing to zero, so Q = (1/4m)sᵀBs, and the relaxed optimum is B's leading (largest-algebraic) eigenvector, rounded by sign — Newman's spectral method, which matches a brute-force argmax exactly on a small graph and reports a clique indivisible when that eigenvalue is non-positive. The integer optimum is out of reach: modularity maximization is NP-hard, so spectral rounding, Louvain, and Leiden are all heuristics. Modularity also has a resolution limit (Fortunato–Barthélemy): on a ring of thirty cliques the optimal partition merges adjacent cliques even though each is a genuine community, because the global √(2m) normalization makes a small clique invisible; the resolution parameter γ only moves the scale, it does not remove the limit. The deep result is the stochastic block model's detectability transition: for the symmetric two-block sparse SBM, a partition correlated with the planted one is recoverable if and only if (c_in − c_out)² > 2(c_in + c_out), the Kesten–Stigum threshold — below it, no algorithm beats a coin flip, an information-theoretic converse and the community-detection analogue of the channel capacity the noisy-channel topic priced. We demonstrate it numerically: above the line spectral recovery overlaps the truth at 0.93, below it at 0.01. Louvain (local-moving plus aggregation, with a closed-form ΔQ) and Leiden (adding a refinement phase that guarantees internally-connected communities) are the workhorses; a disconnected community can be a Louvain local optimum its local moving cannot repair, and Leiden's refinement is exactly what forbids it. The modularity of knowledge is literal: a corpus decomposes into themes worth summarizing precisely when modularity is high, the resolvable scale is coarse enough, and the block signal clears Kesten–Stigum — below threshold there is no decomposition to summarize, and GraphRAG degrades to flat retrieval. The entity graph is a synthetic planted partition on the finance vMF geometry, exact for that model and illustrative of a real knowledge graph; a tested notebook owns every number and the lab recomputes only the closed forms.

Start here 1 prerequisite

advanced rag-information-theory

Multi-Hop and Iterative Retrieval as Search over an Evidence Space

A compositional question — the revenue of the company that acquired Company A's primary supplier — hides its answer in a document the query cannot reach in one retrieval: near-orthogonal to the query on the embedding sphere, yet a short reformulation away through a bridge filing that names the supplier. Retrieval becomes a search over an evolving evidence space, where each hop is the previous topic's retrieve-and-select step, end-to-end recall is the product of per-hop recalls, an information-theoretic stopping rule decides when the marginal evidence no longer pays its cost, and — the climax — single-shot selection provably cannot shortcut the path because the bridge and answer form a supermodular synergy.

Single-hop retrieval answers a question by finding documents near the query on the embedding sphere. A compositional question — the revenue of the company that acquired Company A's primary supplier — defeats it: the answer document is near-orthogonal to the query (their cosine is below the edge threshold) and reachable only through a bridge document that names the supplier. On the retrieval graph, whose nodes are documents and whose edges are the relation 'retrievable-from', the answer sits at graph distance two, so single-hop recall is essentially zero while multi-hop recall is essentially one — the compositional gap. This topic frames multi-hop and iterative retrieval as search over an evidence space: a state is a belief over answers, an action is a reformulated query, a trajectory is a path through the graph, and each hop is exactly the retrieve-and-select operator the previous topic certified. The reformulation operator q' = normalize(d − ⟨d,q⟩q) extracts the new entity a read filing names, the part orthogonal to the current query — and that is what turns a bridge into the next query. The greedy hop maximizes expected marginal information; the optimal policy satisfies a Bellman recursion we name but do not solve, because it is intractable over the belief simplex. We then pay the bill of chaining: end-to-end recall is the product of per-hop recalls, so it decays geometrically, and to hold an end-to-end target ρ each of k hops must over-retrieve to ρ^(1/k) — though positive dependence between hops, by the same FKG argument the capstone used for cascades, makes the independent product a conservative lower bound, the safe direction to provision. We close on stopping and the climax. A hop's marginal information is the new direction its filing opens; the rule is to stop when that residual collapses — when the read filing names no one new — which makes the realized hop count equal the chain depth. Greedy hopping earns no 1−1/e guarantee, and the reason is the heart of the topic: information gain is not submodular, and the compositional question is its supermodular, synergistic case — the bridge alone says nothing about the answer, the answer document alone cannot be recognized as relevant, yet together they resolve it, the exact XOR witness the context-selection topic used. So single-shot selection provably cannot reach the answer — it is never in the one-hop pool — and a myopic stopping rule that watches the belief would halt at the worthless-looking bridge; only reformulating from the bridge's content harvests the synergy. The synthetic von Mises–Fisher answer model is exact for the model and illustrative of a real retriever, and the compositional-gap headline is one constructed corpus, not a universal law; a tested notebook owns every number and the lab recomputes only the closed forms.

1 prerequisite

Agentic RAG

One topic is this

also called multi-hop RAG · iterative RAG · retrieval-as-tool · agentic search

Ingest
Index
Retrieve
Fuse
Rerank
Select
Generate
Evaluate

retrieve → generate → retrieve → generate — A stage sends control backwards — the number of retrievals depends on the data.

Mechanism: Retrieval stops being a pipeline stage and becomes an action: the system reformulates its query from what it just read and retrieves again, until a stopping rule fires. The number of retrievals is determined by the data, not fixed in advance.
Wins when: Compositional queries whose answer is near-orthogonal to the query and reachable only through a bridge document — where a single-shot retrieval pool provably does not contain the answer at any k.
Fails when: The stopping rule, and it fails in a way that looks correct. Belief movement is tiny at the bridge and enormous at the answer, so a myopic "stop when the belief stops moving" rule halts at the worthless-looking bridge and never reaches the answer. Per-hop retention also compounds multiplicatively, so a chain is only as good as the product of its stages.

The mathematics it rests on

advanced rag-information-theory

Multi-Hop and Iterative Retrieval as Search over an Evidence Space

Start here 1 prerequisite

advanced rag-information-theory

Context Selection: Submodular Coverage, MMR, and Determinantal Point Processes

Once retrieval returns more candidate passages than the window can hold, which subset do you keep? Ranking by relevance packs the context with near-duplicates that each add almost nothing — the diminishing returns PMI measured in bits — so the right objective is monotone submodular coverage, for which greedy is provably within 1−1/e of the optimum (Nemhauser–Wolsey–Fisher), while MMR trades relevance against redundancy as a popular heuristic with no such guarantee, and a determinantal point process makes diversity a probability P(S) ∝ det(L_S), the squared volume of the selected feature vectors, so two near-duplicates span a flat parallelepiped and are almost never drawn together — and at a fixed budget every one of these diversity-aware rules beats plain top-k on the answer it ultimately produces.

Retrieval returns a ranked list; the generator's window holds a budget of k passages. Filling that budget with the top-k by relevance is the obvious move and the wrong one: the highest-scoring passages of a confusable company are near-duplicates, and the previous topics showed each duplicate adds almost no information — PMI measured it in bits, the long-context topic watched answer quality go flat over the redundant set. This topic gives the mathematics of choosing the subset that maximizes coverage of the query rather than per-item relevance. The natural objective is monotone submodular: facility-location coverage f(S) = Σ_i w_i max_{j∈S} sim(i,j) rewards a selection for standing in for every candidate the query cares about, and submodularity — the marginal value of adding a passage to a larger set is no greater than adding it to a smaller one — is exactly the diminishing-returns property the saturation curve exhibits. Maximizing a monotone submodular function under a cardinality constraint is NP-hard, but the greedy algorithm that adds the highest-marginal-gain passage at each step returns a set worth at least (1−1/e) ≈ 0.632 of the optimum, the Nemhauser–Wolsey–Fisher bound, proved here in full; Minoux's lazy-greedy speeds it up by exploiting submodularity itself. Maximal Marginal Relevance, λ·rel(d,q) − (1−λ)·max_{d'∈S} sim(d,d'), is the field's workhorse diversity heuristic, but — the rigor flag — it carries no submodular guarantee: its penalty is taken against the evolving chosen set, and the closest fixed objective is submodular but non-monotone, exactly the hypothesis the theorem needs. The determinantal point process gives a probabilistic answer: P(S) ∝ det(L_S), where the determinant is the squared volume of the parallelepiped the selected feature vectors span, so near-duplicates span a near-flat solid of near-zero volume and are almost never sampled together; the quality–diversity factorization L = diag(q) S diag(q) separates per-item relevance from a similarity kernel, exact MAP is NP-hard, and the certified object is greedy on the monotone surrogate log det(I + L_S), which inherits the same 1−1/e. We close on the payoff and the honest hinge. The objective we actually want is the answer-information gain I(A;D_S|Q), but mutual information of a set with a target is not submodular in general — not even on this corpus, and provably not for a synergistic pair whose marginal gain increases with conditioning — so coverage, not info gain, is the backbone that earns the theorem. On a finance pool where the most query-relevant passages are sector-generic near-duplicates that support the gold company and its peer equally, while the disambiguating passage is less query-similar, top-k spends its budget on the redundant cluster and leaves the answer split; coverage- and diversity-aware selection reach the disambiguator and sharpen the answer. We read every method through the imported answer_posterior_topk, so the numbers chain with the rest of the arc, and the headline — diversity beats top-k at a fixed budget — is a comparison across methods, pinned to the observed run. A tested notebook owns every number; the lab recomputes only the closed forms.

3 prerequisites

advanced rag-information-theory

Pointwise Mutual Information: What Retrieval Adds to Generation, in Bits

Generation begins with an answer prior p(a|q) and a retrieved document sharpens it to a posterior p(a|q,d); the log-ratio of the two is the pointwise mutual information of that document for that answer — positive for a relevant filing, negative for a distractor that costs bits — and its average is the conditional mutual information I(A;D|Q), the entropy retrieval removes, the same quantity the dense encoder's InfoNCE objective maximized a lower bound on

Retrieval is usually scored by whether the right documents came back — recall, average precision, the set-metric family. This topic scores it by a different, information-theoretic question: how many bits does a retrieved document add to the generator's answer? Generation begins with a prior over answers conditioned on the query alone, p(a|q), which is uncertain, with entropy H(A|Q). A retrieved document d sharpens it to a posterior p(a|q,d). The pointwise mutual information pmi(a;d|q) = log p(a|q,d)/p(a|q) is, for a particular answer a, the log-factor by which the document moved the odds — the literal log of a Radon–Nikodym density ratio. Its expectation over the posterior is the per-document information gain, the KL divergence KL(p(·|q,d) ‖ p(·|q)) ≥ 0, and averaging over documents and queries gives the conditional mutual information I(A;D|Q) = H(A|Q) − H(A|Q,D): the entropy retrieval removes, the bits it adds, verified three ways. We then make four points the set-metric layer could not see. A relevant filing has positive pmi on the true answer; a plausible distractor has negative pmi — it costs bits, dragging the posterior toward a wrong answer. A second document that merely repeats the first adds almost nothing — diminishing returns are the chain rule of mutual information, not a heuristic. The dense encoder we trained was, all along, maximizing an InfoNCE lower bound on this very I(Q;D), so retrieval value and contrastive training measure the same quantity from two directions. And bits-added is orthogonal to recall: on an easy corpus where every gold filing is retrieved, recall reads 'perfect' for every query while the bits actually delivered vary widely. The answer model is a synthetic von Mises–Fisher exponential-family stand-in, not a transformer, so every bit is exact for the model and illustrative of a real generator; a tested notebook owns every number.

2 prerequisites

Adaptive RAG

One topic is this

also called routing RAG · query routing · strategy selection

Ingest
Index
Retrieve
Fuse
Rerank
Select
Generate
Evaluate

select → retrieve → generate — A decision picks among alternative paths before committing the effort.

Mechanism: A router chooses which retrieval strategy to run for each query, using only features available before retrieval, trading answer quality against the cost of the arm it picks.
Wins when: Queries that genuinely differ in the effort they require, together with real cost asymmetry across the arms. Without the second, the router’s only available win is cost, and it is worth nothing.
Fails when: Routing is worthless — regardless of how good the classifier is — when the arms’ advantage ordering does not vary across queries. The achievable gain is exactly a Jensen gap between the expected maximum and the maximum expectation, and that gap is zero when one arm is uniformly best. A router tuned and scored on the same queries will also look roughly twice as good as it is.

The mathematics it rests on

advanced generation-grounding

Adaptive Retrieval Routing: Choosing a Strategy per Query

The published gate decided whether to answer; this one decides how hard to try — a router over retrieval strategies whose optimum is Chow's rule with more than two actions, whose achievable region is a convex hull, and whose entire possible gain is a Jensen gap that vanishes exactly when the strategies never change places

Selective generation ended with a binary gate: emit the answer or abstain, cut at Chow's threshold. This topic is that gate with more than two actions. A router chooses among retrieval strategies — answer from the query alone, retrieve once, or iterate — using only features available before any retrieval fires, and the pointwise optimum is the same argmax Chow's rule was a special case of, reducing to it exactly when the actions are emit and abstain. Sweeping the cost weight traces a family of policies whose convex hull is the achievable cost–quality region, and the most any router can win over the best single strategy is a Jensen gap between the expected maximum and the maximum expectation — strictly positive if and only if the strategies actually change places across queries. The honest half is what a realizable router collects: on a corpus where the oracle gap is positive at every operating point, a fitted router captures 58% of it at one, 3.5% at another, and less than nothing at a third, where it loses to always-retrieving-once. Only one of four gains is distinguishable from zero. A gap that exists is not a gap you can collect.

Start here 2 prerequisites

advanced generation-grounding

Selective Generation: When a RAG System Should Abstain

The faithfulness topic ended at a back-off frontier — coverage traded for a conformally guaranteed faithfulness — and left a question: what should the system do when the certified-faithful answer is too thin to be worth emitting? It should abstain. The two prior topics decided which claims to keep inside an answer; this one decides whether to answer at all. That answer-level decision has its own mathematics: Chow's cost-optimal reject rule, the risk–coverage curve and its area, the gap a real confidence signal pays against the oracle, a distribution-free guarantee on the wrong-emission rate, and a cost model for abstaining versus erring.

The faithfulness topic measured a generated answer as two numbers and traded coverage for a conformally guaranteed faithfulness along a back-off frontier — deciding which claims to keep. This topic takes the next step the frontier set up: the per-query decision of whether to answer at all. We define the answer-level selective risk (the error rate among the answers emitted) and coverage (the fraction of queries answered), and derive Chow's rule — the cost-optimal reject threshold, abstain iff the probability of being wrong exceeds the cost ratio of abstaining to erring. Sweeping the threshold traces the risk–coverage curve, whose area, the AURC, is the answer-level mirror of average precision; the achievable curve, ordered by a noisy judge's confidence, sits above the oracle curve ordered by truth, and the excess closes only as the signal's discrimination approaches perfect. We then lift conformal risk control from the claim level to the answer level: a monotone wrong-emission loss whose distribution-free threshold controls the rate of confident wrong answers at a chosen level. The whole construction composes with the prior topic's back-off into a two-stage gate — certify the faithful claims, then abstain if the answer is too risky or too thin — and a cost model picks the operating point where a finance RAG defers to a human analyst rather than guess. A tested notebook owns every number, and the build-and-run discipline catches the trap: a too-weak signal makes abstention no better than refusing to answer at all.

2 prerequisites

advanced generation-grounding

Retrieval versus Long Context: Attention Complexity and Positional Bias

1 prerequisite

advanced rag-information-theory

Multi-Hop and Iterative Retrieval as Search over an Evidence Space

1 prerequisite

Operators, not architectures

This bolts onto any pattern above without changing its topology. It is worth separating out, because most of what gets marketed as "Advanced RAG" is this — a maturity level rather than a design.

Reranking Cascade

One topic is this

also called cross-encoder reranking · two-stage retrieval · Advanced RAG

Ingest
Index
Retrieve
Fuse
Rerank
Select
Generate
Evaluate

retrieve → rerank → generate — Retrieval fires once; stages run in order.

Mechanism: A model too expensive to run over the corpus rescores a shortlist that a cheap retriever produced. Joint query–document attention escapes the rank ceiling a dual encoder is bound by, because the score is no longer an inner product of independent representations.
Wins when: The first stage has high recall at a depth the second stage can afford. Under a known-item judgment the arithmetic is exact: an oracle reranker’s precision at one equals stage one’s recall at the cutoff, which is the entire reason the cascade works.
Fails when: This is an operator, not an architecture — it bolts onto any of the patterns above and changes none of their topology, which is why "Advanced RAG" is a maturity level rather than a design. Recall is monotone in the shortlist depth only for an exact reranker; a lossy one can rank a confident false positive above a true neighbor, so a deeper shortlist makes it worse.

The mathematics it rests on

advanced neural-retrieval

Cross-Encoders and the Reranking Cascade

A dual encoder's separable score has a rank-d ceiling, and a learned bilinear q⊤Wd does not escape it — S = QWG⊤ is still rank at most d. Only a nonlinear joint encoder h([q;d]) breaks the wall, at a cost that forbids it from ever being the first stage. So the cross-encoder becomes a reranker: retrieve K cheaply, rescore K expensively. An oracle rerank is recall-monotone and pinches recall@1 to the candidate pool's recall@K, but a lossy cross-encoder can dip below the first stage by being confidently wrong

The dual-encoder topic kept invoking the cross-encoder as the rank-free, un-precomputable counterpoint; this topic spends that power. A cross-encoder scores a fused pair h([q;d]), letting query and document attend at every layer, and the first result is why the nonlinearity is the whole point: a learned bilinear form q-transpose-W-d does NOT escape the rank ceiling, because S = QWG-transpose = (QW)G-transpose is still a product through a d-dimensional bottleneck and has rank at most d. Only a nonlinear joint encoder has no such ceiling, so it escapes the sign-rank wall the embedding-dimension topic proved — and pays for it with a joint forward pass per pair, which forbids precomputation and makes it unusable as a first stage. The architecture is therefore a retrieve-then-rerank cascade: a cheap dual encoder retrieves a candidate pool of size K, the cross-encoder rescores only those K. We prove the cascade's two governing facts. First the recall pinch: an oracle rerank of the top-K makes recall@1 equal the candidate pool's recall@K, so the reranker can never recover a true neighbor the first stage dropped — the cost model c_ret + K times c_ce against the brute corpus-size times c_ce sets the Pareto knee. Second, oracle rerank is recall-monotone in K because a larger pool is a superset, but a lossy cross-encoder can dip below the first stage by being confidently wrong, demoting a true top-1. We close on expressivity versus generalization: the cross-encoder's extra capacity overfits small training sets where the dual encoder's inner-product inductive bias wins, the up-link to VC dimension. A laboratory shows the rank-ceiling split, the cascade frontier and its cost knee, and the per-query rerank buckets with their hard-negative fixes and confident-wrong dips; a tested notebook owns every number; and the finance thread is the production reranker that separates same-sector hard negatives a single embedding confuses.

Start here 1 prerequisite

advanced ranking-fusion

LambdaRank, LambdaMART, and Listwise Objectives

RankNet gave each document a gradient force but is position-blind — a swap at the top of the list costs the same as a swap in the tail. This topic makes the objective position-aware three ways: LambdaRank weights each pairwise force by the ΔNDCG a swap would cause, LambdaMART boosts those forces into trees, and listwise objectives replace the heuristic with a proper Plackett–Luce loss — anchored by the honest theorem that LambdaRank's field is the gradient of no scalar loss, while the listwise loss genuinely is one

RankNet reduced ranking to a pairwise logistic surrogate whose gradient factorizes into a per-document lambda force — the net pull on each document from its preference pairs. That force is position-blind: it weights a swap at ranks 1–2 exactly like a swap at ranks 99–100. This topic removes that blindness three ways. LambdaRank multiplies each pairwise force by the absolute change in NDCG that swapping the pair would cause; because the change in NDCG factorizes into a change in gain times a change in discount, and the discount marginal is steep at the head and flat in the tail, the gradient mass concentrates where ranking quality is decided. But this weighting has a price, and the price is the honest core of the topic: the lambda field now depends on the current ranking, so it is the gradient of no scalar loss — locally, within a ranking cell, it is a weighted-RankNet gradient with a symmetric Jacobian, but globally it is discontinuous across swaps and a closed loop integrates to a nonzero value, the classical signature of a non-conservative field. LambdaMART feeds those lambdas as pseudo-residuals to gradient-boosted regression trees, buying a nonlinear scorer demonstrated to escape a rank ceiling no linear model can on a constructed interaction instance. Listwise objectives restore a proper loss: ListMLE is the negative log-likelihood of the Plackett–Luce model over the ideal permutation, convex in the scores, and ListNet its top-one cross-entropy. The arc is pairwise (a loss, position-blind) to LambdaRank (position-aware, not a loss) to listwise (a convex loss). A tested notebook owns every number; the headline NDCG deltas among the methods sit within the confidence interval, so the reported wins are structural.

2 prerequisites

advanced ranking-fusion

LLM Rerankers: Listwise Permutation Objectives and RankGPT

Classical learning-to-rank scored each document from a fixed feature vector, blind to the rest of the list except through the loss. The LLM reranker drops that — feed the whole candidate list into one model and it emits a permutation, scoring documents in each other's context. But context fits only a window of candidates, so RankGPT slides a window and locally re-sorts (a bubble sort), the order is position-biased (lost in the middle), windows must be aggregated (social choice), and the costly teacher is distilled into a cheap student — anchored by the honest caveat that an autoregressive permutation model, like LambdaRank, descends no scalar loss at inference.

The predecessor scored each document from a fixed feature vector. An LLM reranker drops that assumption: feed the whole candidate list into one model and it emits a permutation directly, scoring documents in each other's context — the most listwise objective there is. We model the LLM as a Plackett–Luce sampler at temperature τ (the ListMLE bridge made operational): as τ→0 it emits the ideal order byte-for-byte; as τ→∞ it is uniform. But an LLM context fits only a window of w ≪ n candidates, so RankGPT slides a window back-to-front and locally re-sorts — a bubble sort whose call count is O(n/s) per pass against the O(n²) of all-pairs comparison, with one perfect pass guaranteeing the global best in the top window and recall climbing toward a plateau under noise. The in-window order is position-biased (lost in the middle); presenting each window in several random orders and aggregating flattens it. Multiple windows produce multiple noisy permutations, reconciled by Borda, RRF, the NP-hard Kemeny median, or the Dwork et al. Markov-chain consensus — a comparison random walk's stationary distribution — with the consensus error falling at the 1/√K central-limit rate. Finally the expensive teacher is distilled into a cheap linear listwise student: a perfect teacher's student is exactly the predecessor's own fit, and it answers at zero inference LLM calls. The cost–quality frontier is the systems story. A tested notebook owns every number; the LLM is a simulated noisy oracle, so the provable wins are algorithmic, and the aggregate quality deltas sit inside the confidence interval.

2 prerequisites

advanced neural-retrieval

Knowledge Distillation for Retrieval: Teacher–Student Transfer (MarginMSE)

A cross-encoder is the most accurate scorer and the least deployable — a joint forward pass per query–document pair forbids precomputation. Distillation spends that accuracy cheaply: train a precomputable dual-encoder student to match the teacher's per-query score MARGIN, not its absolute level. The all-pairs MarginMSE reduces to a centered Frobenius distance 2·n_d·‖SC − TC‖², so margins are blind to the teacher's per-query miscalibration (translation-invariance), and the margin-optimal rank-d student is best_rank_d of the per-query-centered teacher — Eckart–Young again. Distillation approaches the teacher's recall at dual-encoder inference cost, but the embedding-dimension rank ceiling still binds it

The cross-encoder topic built the most expressive relevance scorer and proved it could never be a first stage: a joint forward pass per query–document pair forbids precomputation. This topic spends that accuracy where it is affordable — it distills the expensive teacher into a cheap, precomputable dual-encoder student. The transfer loss is MarginMSE (Hofstätter et al., 2020): rather than match the teacher's absolute scores, match its per-query score MARGIN between a positive and a hard negative. The first result is why margins are the right target. The all-pairs MarginMSE between a student S and teacher T reduces, by a variance identity, to a centered Frobenius distance 2 n_d ||SC − TC||² where C row-centers each query's document scores — so adding a per-query offset T ↦ T + b·1ᵀ leaves the loss unchanged. That translation-invariance is the rigorous hinge: cross-encoder scores are per-query miscalibrated, and margins are blind to that miscalibration. The closed-form optima follow from Eckart–Young–Mirsky with no SGD: the pointwise-MSE-optimal rank-d student is best_rank_d(T), and the margin-optimal one is best_rank_d(TC), the per-query-centered teacher. Because the teacher's scores carry a large constant level — its top singular direction — the pointwise student wastes a dimension reproducing it, while the margin student spends every dimension on ranking; at a restricted rank the margin student's recall@1 leads the pointwise student's, and the embedding-dimension rank ceiling binds both below the teacher, which they approach but never exceed. The payoff is the reranking sub-track's whole point: the distilled student gives cross-encoder-quality ranking at dual-encoder inference cost — precompute the documents once, answer by MIPS, never run a per-pair forward pass. A laboratory shows the translation-invariance of the loss under a miscalibration slider, the margin-over-pointwise recall gap across the rank budget with the two spectra that explain it, and the teacher's graded dark-knowledge margins beside the cost speedup; a tested notebook owns every number; and the finance thread is the production dense retriever that inherits a cross-encoder's same-sector discrimination at no extra query-time cost.

2 prerequisites

The contrast class

Not retrieval at all. It belongs here because it is the baseline every architecture above has to beat in order to justify its own complexity.

Long Context

One topic is this

also called Cache-Augmented Generation · CAG · context stuffing · no-RAG

Ingest
Index
Retrieve
Fuse
Rerank
Select
Generate
Evaluate

ingest → generate — Retrieval fires once; stages run in order.

Mechanism: No retrieval at inference: a bounded corpus is placed in context in full. It is on this list because it marks the boundary — the baseline every architecture above has to beat to justify existing.
Wins when: A corpus small enough to fit, where attention over the whole of it is affordable and no selection is needed.
Fails when: Attention cost is quadratic in context length, and accuracy is U-shaped in position, so evidence in the middle is attended least. More context is not better: adding passages dilutes the attention budget and raises the entropy of the answer even when retrieval was perfect.

What this map leaves out

Roughly half the published topics are not named above. Approximate nearest-neighbor index structures, vector quantization, embedding geometry, and the evaluation machinery underpin every architecture here without being any one of them — which is why they are organized by the mathematics instead, on the curriculum roadmap. A comparison that measured these architectures against each other, rather than describing them, would need all of it.