Capstone: The Mathematics of a Production Multimodal Financial RAG System

Overview & motivation

Every prior topic in this series built one organ of a retriever. The probabilistic-IR track built the lexical leg; the dense dual encoder and late interaction built the neural legs; IVF, product quantization, and HNSW built the index; PLAID served late interaction at scale; reciprocal-rank fusion combined heterogeneous rankings; the over-fetch laws priced the cost of recall. A production multimodal financial RAG system is what you get when these organs are wired into one body, and the wiring has mathematics of its own — not a new primitive, but the composition of the ones already built.

This capstone proves the three laws that govern the composed pipeline. They are deliberately honest about their status: one is an exact theorem, the rest are measured propositions, and the difference is the whole point. We have built the retrieval half of the stack; the generation, grounding, and evaluation layer above it — pointwise mutual information between query and answer, conformal factuality, determinantal context selection, GraphRAG, retrieval distillation — is the unbuilt upper curriculum, named here but not yet derived. What we can compose now is everything from ingest through rerank, and that is enough to see the system as a single object.

A · cascade recallB · hybrid fusionC · budget water-fillingD · storage + anchor

copula dependence ρ = 0.3

cascade product ∏rᵢ (lower bound)

0.851

measured end-to-end recall

0.922

over-fetch 1/∏rᵢ

1.18×

R_true at ρ=0.3 (gap +0.049)

0.349

End-to-end retention is the product of the per-stage retentions (1.00 · 0.92 · 0.92 = 0.851), and to deliver k survivors the front end over-fetches by 1/∏rᵢ ≈ 1.18×. That product is exact only under independent per-stage survival; under the positive dependence real queries exhibit, FKG/Harris makes it a conservative lower bound — drag ρ above 0 and the true recall lifts above the dashed product line, below 0 it falls under it, with equality at ρ = 0. The measured pipeline (recall 0.9225) sits above its product 0.851, as the bound predicts.

Notation: the served pipeline and one shared truth

A query $q$ enters a pipeline of stages $S_1, \dots, S_L$ — candidate generation, fusion and pruning, rerank — each a lossy map that keeps some documents and discards others. The exact end-scorer is brute-force MaxSim, $S(q, d) = \sum_i \max_j \langle q_i, d_j\rangle$, and its top-$k$ defines the ground truth: the true neighbors of $q$ are the $k$ documents that the full, uncompressed scorer would return. This choice matters. Because the truth is the exact scorer and not any one cheap leg, no candidate-generation leg is privileged — the oracle is the act of reranking everything, not any single retriever — which is exactly what lets the legs genuinely complement and lets a full-budget pipeline reach recall $1$.

The corpus is one shared multi-view document set. Each document is a bag of unit-norm token vectors (the token view), seen three ways: a lexical leg matches a window of the tokens’ quantized centroid identifiers (BM25 over a centroid vocabulary), a dense leg pools a disjoint window into a single vector (cosine MIPS), and a late-interaction leg scores a third window by cheap centroid-MaxSim. Three partial views of one document — the systems-level shape of multimodality, where a text leg, a transcript leg, and a chart leg each observe part of a filing and fusion recombines them. The cross-modal alignment that would map them into one joint embedding is a separate, future topic; here the legs are independent retrievers feeding fusion.

Movement 1 — cascade recall composition

Fix a query and one true top-$k$ neighbor $x^\*$. Let $A_i$ be the event that $x^\*$ survives stage $i$ given it entered, and $r_i = \Pr[A_i] \in (0, 1]$ the stage’s per-neighbor retention. The neighbor reaches the output exactly when it survives every stage.

The first law is the one every engineer half-knows and half-misstates.

On the measured pipeline the three retentions are $1.000$, $0.922$, $0.922$ — candidate generation loses nothing at this breadth, fusion and rerank each drop a little — so the product is $0.851$ and the over-fetch a modest $1.18\times$. But the product law is exact only under an idealization, the same positional-independence assumption the over-fetch topic flagged for a single stage, now lifted to the whole pipeline. Real queries violate it, and the direction in which they violate it is the load-bearing claim.

The temptation is to guess the wrong sign: surely correlated failures lose more? They do not. Positive correlation piles the failures onto the same hard queries, which fail anyway, so fewer distinct documents are lost than independence predicts and true recall sits above the product. The argument is FKG/Harris — increasing events on a positively-associated law have $\Pr[\bigcap] \ge \prod \Pr$ — but because the intuition is so easy to invert, the notebook does not trust it: it generates dependence with a bivariate-normal survival copula and sweeps the correlation across both signs. The gap $R_{\text{true}} - \prod r_i$ runs from $-0.102$ at strong negative dependence through $-0.002$ at independence to $+0.167$ at strong positive dependence, monotone, with equality pinned at $\rho = 0$ — the first panel makes the slider live. The measured pipeline obliges: its realized recall, $0.922$, sits above its own product, $0.851$, exactly as a lower bound should.

Movement 2 — hybrid fusion gain

The pipeline’s middle stage fuses heterogeneous legs. Fix a query with true-relevant set $\mathcal{R}$ at depth $k$, let $A_L = \mathcal{R} \cap \mathrm{top}_k(L)$ be the relevant documents leg $L$ surfaces, and $\rho_L = |A_L| / |\mathcal{R}|$ its recall. The fused ranking is $\mathrm{RRF}(d) = \sum_L 1/(c + r_L(d))$ with $c = 60$, reciprocal-rank fusion imported verbatim. The question is when fusing helps, and the honest answer is a condition, not a guarantee.

This is the topic’s headline-flip discipline made literal. The naive expectation — a noisy leg drowns out a good one — is right in spirit and wrong in mechanism, and the notebook builds and runs the counterexample before the prose is allowed to claim anything. A lone loud vote is not enough under RRF’s near-uniform weights; it takes a conspiracy of two. The second panel reconstructs the RRF scores live and shows $r_3$ falling past the top-$3$ cut.

What survives the scrutiny is the conditional gain.

The complementarity is not incidental; it is engineered and it is the real multimodal story. When the three legs were three approximations of one scalar truth, they formed a quality ladder — each strictly better than the last — and no fusion could beat the best, because the worse legs added only noise. Making each leg a partial view of the document, blind to what the others see, is what gives fusion something to recombine. That is precisely the value a lexical leg adds over a dense one in a real financial retriever: it surfaces the exact tokens — tickers, line-item names, dollar figures — that a paraphrase-trained embedding smears, documents the dense leg never had.

Movement 3 — end-to-end budget allocation

The whole pipeline runs on a budget. Each stage $i$ buys retention $g_i(c_i)$ by spending compute $c_i$, and the costs must share a unit — distance-computations per query — or a single budget is meaningless. A sparse lexical match touches a few centroid-id terms; a dense leg is one pooled dot product; late-interaction MaxSim scores every query token against every document token, an order of magnitude more per document. The system-design question is how to spend a fixed budget $B$ across these unequal stages.

Two honesties belong with this theorem. First, the equal-marginal condition is not bought by the logarithm. The un-logged product has $\partial R / \partial c_i = g_i' \prod_{j \ne i} g_j = g_i' \,(R / g_i)$, and the common scalar $R$ cancels, so $g_i' / g_i = \lambda$ falls out of the product directly; what the log buys is separability — decoupling the stationarity equations — and, with concavity, a global optimum. Second, the separable model $\prod_i g_i$ is the Movement-1 object: measured stage-by-stage against the global truth and multiplied, it is a conservative lower bound on the true coupled-pipeline recall, which unions the legs and does at least as well. We optimize the bound, honestly labeled.

That anchor is the one exact statement of the entire capstone. Probe every cell, prune nothing, rerank by exact MaxSim, and the pipeline is brute-force retrieval — recall $1.0$, the same probe-all/prune-nothing collapse the PLAID topic proved for its cascade, here for the whole stack. Everything above it — the product law, the fusion gain, the water-filling allocation — is a heuristic speed-for-recall trade we measure rather than guarantee.

Finance case study

Honest accounting