advanced rag-information-theory 30 min read

Pointwise Mutual Information: What Retrieval Adds to Generation, in Bits

Generation begins with an answer prior p(a|q) and a retrieved document sharpens it to a posterior p(a|q,d); the log-ratio of the two is the pointwise mutual information of that document for that answer — positive for a relevant filing, negative for a distractor that costs bits — and its average is the conditional mutual information I(A;D|Q), the entropy retrieval removes, the same quantity the dense encoder's InfoNCE objective maximized a lower bound on

Part of the Information Theory of RAG track · View full curriculum →

Prerequisites: Dense Retrieval and Dual Encoders: Architecture, Expressivity, and the Cost of Negatives Set Metrics: Precision, Recall, MAP, and MRR as Estimators

From “did the right document come back” to “how many bits did it add”

The evaluation layer scored retrieval by presence. A metric is a sample mean of per-query scores (set metrics): recall asks what fraction of the relevant documents were returned, average precision weights them by rank. These are estimators of whether the right documents came back, and they are the right question when the failure mode is a missing document.

But a retrieval-augmented generator does not consume a recall number. It consumes the documents, and it produces an answer. The question that matters for generation is not whether a document was returned but what it did to the answer — and the natural unit for that is the bit. Generation begins, before any retrieval, with a distribution over answers conditioned on the query alone, the prior $p(a \mid q)$ . It is uncertain: an analyst’s question may pin a sector but not the specific company, so the prior spreads its mass and carries entropy $H(A \mid Q)$ . A retrieved document $d$ sharpens that distribution to a posterior $p(a \mid q, d)$ , and the amount of uncertainty it removes is measured in bits.

Retrieval value is entropy reduction: the bits a document removes from the answer’s uncertainty. Per answer it is the pointwise mutual information $\operatorname{pmi}(a; d \mid q) = \log_2 p(a \mid q,d)/p(a \mid q)$ ; on average it is the conditional mutual information $I(A;D \mid Q) = H(A\mid Q) - H(A\mid Q,D)$ .

retrieved document d = doc 1 · company 1 (sector 0) — gold

H(A|Q) prior

2.133 bits

H(A|Q,d) posterior

0.533 bits

bits removed = H(A|Q) − H(A|Q,d)

1.599

pmi(a*; d | q)

+0.995 bits

The prior spreads over the sector's companies (uncertain). The gold filing (doc 1) sharpens the posterior onto a* and gives a positive pmi; the same-sector distractor (doc 0) sharpens onto the wrong company, so pmi at a* turns negative — it costs bits. A far-sector filing barely touches a*.

Finance vMF corpus reused from the dense-retrieval topic (8 companies across 4 sectors, one filing per company), 32 sector-ambiguous queries; the answer model is a synthetic softmax stand-in, not an LLM. I(A;D|Q) = 1.183 bits (H(A|Q) = 2.059 → H(A|Q,D) = 0.876). Numbers mirror pmi_retrieval_value.py; the lab recomputes entropy, pmi, and histogram bins in closed form.

The lab above is the topic in four panels, and we take each in turn: the prior sharpening into a posterior as a document is retrieved, with the bits removed and the pmi at the true answer read off; the histogram where a relevant filing’s pmi is positive and a distractor’s is negative; the diminishing returns of a redundant second document; and the InfoNCE bound the encoder maximized, beside the recall axis that bits are orthogonal to.

The answer model: a prior, a posterior, and the log-ratio between them

To measure bits we need probabilities, and a retriever gives us geometry — query and document directions and the inner products between them. We turn the geometry into an answer model in the way retrieval-augmented generation does: a retrieval distribution over documents, a per-document answer distribution, and a prior that is the marginal of the two. We reuse the dense-retrieval finance geometry — four sectors, two companies each, one filing per company — so the answer prototypes $\mu_a$ are the company filing directions, and the answer to a query is which company it concerns.

Definition 1 (The answer prior, posterior, and pointwise mutual information).

Let the answer space be the $K$ companies with unit prototypes $\mu_a$ , and let $q, d$ be unit query and document vectors. The retrieval distribution is the dual-encoder MIPS softmax

p(d \mid q) = \operatorname{softmax}_d\!\big(\langle q, \mu_d\rangle / \tau_{\mathrm{doc}}\big),

the per-document answer posterior sums the query’s and the document’s evidence in log-space,

p(a \mid q, d) = \operatorname{softmax}_a\!\big((\langle q, \mu_a\rangle + \langle d, \mu_a\rangle)/\tau\big),

and the prior is the RAG marginal over which document is retrieved,

p(a \mid q) = \sum_d p(d \mid q)\, p(a \mid q, d).

The pointwise mutual information of document $d$ for answer $a$ is the log-ratio

\operatorname{pmi}(a; d \mid q) = \log_2 \frac{p(a \mid q, d)}{p(a \mid q)} = \log_2 \Big[\tfrac{dp(\cdot \mid q, d)}{dp(\cdot \mid q)}\Big](a),

the logarithm of the Radon–Nikodym derivative of the posterior answer measure with respect to the prior answer measure. The marginal definition of the prior is load-bearing — it is what makes the mutual information identity below close exactly, rather than approximately.

The orientation of pmi is the whole point: it is positive when a document raises an answer’s probability above its prior and negative when it lowers it. A document is not uniformly informative; it carries different bits about different answers, and — as we will see — it can carry negative bits about the true one.

Information gain and the bits retrieval adds

Average pmi over the posterior and you get the per-document information gain, the KL divergence from prior to posterior; average that over which document is retrieved and over queries and you get the conditional mutual information — the bits retrieval adds. The Shannon entropy and the KL divergence are developed in full on formalML; here we assemble them into one identity and verify it three ways.

Theorem 1 (Retrieval value is entropy reduction).

For a fixed query $q$ , with the prior the marginal of the posterior,

\underbrace{\sum_{a,d} p(a,d\mid q)\,\log_2\frac{p(a,d\mid q)}{p(a\mid q)\,p(d\mid q)}}_{\text{joint sum}} \;=\; \underbrace{\sum_d p(d\mid q)\,\mathrm{KL}\big(p(\cdot\mid q,d)\,\big\|\,p(\cdot\mid q)\big)}_{\text{expected gain}} \;=\; \underbrace{H(A\mid q) - H(A\mid q,D)}_{\text{entropy reduction}} \;\ge\; 0,

and averaging over queries gives $I(A;D\mid Q) = H(A\mid Q) - H(A\mid Q,D) \ge 0$ , the bits retrieval adds.

Proof.

Write $p(a,d\mid q) = p(d\mid q)\,p(a\mid q,d)$ . Because the prior is the marginal, $p(a\mid q) = \sum_d p(d\mid q)\,p(a\mid q,d)$ , the ratio inside the joint sum is $p(a,d\mid q)/(p(a\mid q)p(d\mid q)) = p(a\mid q,d)/p(a\mid q)$ , so the joint sum collapses to $\sum_d p(d\mid q)\sum_a p(a\mid q,d)\log_2\frac{p(a\mid q,d)}{p(a\mid q)} = \sum_d p(d\mid q)\,\mathrm{KL}(p(\cdot\mid q,d)\,\|\,p(\cdot\mid q))$ , the expected-gain form. Expanding the logarithm as $\log_2 p(a\mid q,d) - \log_2 p(a\mid q)$ and using the marginal once more turns the same expression into $-\sum_d p(d\mid q)\sum_a p(a\mid q,d)\log_2\frac{1}{p(a\mid q,d)} + \sum_a p(a\mid q)\log_2\frac{1}{p(a\mid q)} = H(A\mid q) - H(A\mid q,D)$ . Nonnegativity is Gibbs’ inequality: each $\mathrm{KL} \ge 0$ , so the $p(d\mid q)$ -weighted average is too. The notebook computes all three forms and asserts they agree — they coincide to machine precision (maximum pairwise disagreement $0$ to the printed precision), with $H(A\mid Q) = 2.06$ bits dropping to $H(A\mid Q,D) = 0.88$ bits, a gain of $I(A;D\mid Q) = 1.18$ bits.

∎

The three forms are not three approximations that happen to be close; they are one number written three ways, and the proof is what guarantees it. The entropy-reduction form is the slogan — retrieval removes uncertainty — the expected-gain form is the per-document decomposition the next sections lean on, and the joint sum is the definition of mutual information itself.

A relevant filing adds bits; a distractor costs them

The expected gain $\mathrm{KL}(\text{posterior}\,\|\,\text{prior})$ is always nonnegative — any document that moves the belief has positive gain in the Shannon sense. But the gain is an average over answers, and the pointwise pmi at a particular answer need not be positive. The answer that matters is the true one, $a^\star$ , and there the sign is diagnostic.

Proposition 1 (The sign of pmi at the true answer).

A relevant filing — the gold company’s own document — raises the posterior on $a^\star$ above the prior, so $\operatorname{pmi}(a^\star; d_{\mathrm{rel}} \mid q) > 0$ . A plausible distractor — a different company’s filing in the same sector — sharpens the posterior onto the wrong company, lowering the probability of $a^\star$ , so $\operatorname{pmi}(a^\star; d_{\mathrm{distr}} \mid q) < 0$ , even though the distractor’s information gain $\mathrm{KL}(\text{posterior}\,\|\,\text{prior}) > 0$ — it moves belief, just toward the wrong answer. A document that costs bits at the truth is a document that, in expectation, makes the generator more likely to be wrong.

This is a proposition about the toy, not a theorem about relevance labels, and it is one we run rather than assume: across the query set the mean pmi at the true answer is $+0.98$ bits for relevant filings and $-1.11$ bits for distractors, and $96.9\%$ of distractors land on the negative side (Panel B). The point of pmi as a word-association score was always that the sign carries meaning — co-occurrence above chance is positive, below chance negative — and the same sign here separates evidence that helps the generator from evidence that misleads it. It is also the precise sense in which a high-recall retriever can hurt: returning a fluent, on-topic, wrong passage is returning negative bits.

Diminishing returns: a redundant document adds almost nothing

If one document adds bits, do two add twice as many? Only if the second is new. The bits a sequence of documents adds obey the chain rule of mutual information, and redundancy makes the later terms vanish.

Proposition 2 (Saturation is the chain rule of mutual information).

The information two documents add about the answer decomposes as

I(A; D_1, D_2 \mid Q) = I(A; D_1 \mid Q) + I(A; D_2 \mid Q, D_1).

When $D_2$ is redundant with $D_1$ — the same filing again — the conditional term $I(A; D_2 \mid Q, D_1)$ is near zero: the posterior is already where $D_1$ put it, and a second copy barely moves it. A novel $D_2$ — a different filing — leaves the conditional term large.

Measured as belief movement $\mathrm{KL}(\text{new}\,\|\,\text{old})$ , the gold filing moves the posterior $0.67$ bits from the prior; a second identical copy then adds only $0.09$ bits, while a different filing adds $0.44$ (Panel C). The marginal value of evidence diminishes with redundancy — which is exactly why a context window should be filled with diverse passages rather than the top- $k$ near-duplicates a similarity search returns. Quantifying that diversity gain is a submodular-coverage problem, the subject of context selection, the topic this one points forward to.

The encoder maximized a lower bound on these bits

The dense encoder did not learn its geometry by accident. It was trained by contrastive learning, and the InfoNCE objective it minimized is, up to sign, a lower bound on the mutual information between a query and its positive document — the same $I(Q;D)$ whose answer-side payoff this topic measures.

Corollary 1 (InfoNCE is a lower bound on I(Q;D), ceilinged at log(N+1)).

For an in-batch problem with one positive and $N$ negatives, the InfoNCE loss $\mathcal{L}$ satisfies

I(Q;D) \;\ge\; \log(N+1) - \mathcal{L}.

The bound can never exceed its ceiling $\log(N+1)$ — even a perfect critic ( $\mathcal{L}\to 0$ ) certifies at most $\log(N+1)$ bits — so it saturates: certifying more information requires more in-batch negatives, not just a better encoder.

The ceiling is the load-bearing caveat, and Panel D shows it directly: as the batch grows from two candidates to eight, the bound climbs but stays strictly under the rising $\log_2(N{+}1)$ line. So the encoder’s training and the retriever’s value are two views of one quantity — training pushed $\mathcal{L}$ down to maximize a lower bound on $I(Q;D)$ , and the bits a retrieved document adds to the answer are that information cashed out at generation time. (The bound is loose, and the empirical $\log(N{+}1) - \mathcal{L}$ from a finite batch is a notoriously unreliable point estimate of the true MI — the variational-bounds literature is a catalog of the bias; the robust statement is the saturation, not a number.)

Bits added is a different axis from recall

Return finally to the set-metric layer. On this corpus retrieval is easy: the gold filing is the query’s nearest company prototype, so it is always rank one, and recall@k and average precision are saturated at $1.0$ for every query. By the set-metric reading, every query is equally, perfectly served.

Yet the bits retrieval adds range from $0.68$ to $1.60$ across those same queries (Panel D, right). The variation is real — an ambiguous query whose prior was spread wide gains more than one whose prior already leaned toward the answer — and recall is blind to it. Recall measures presence: did the document come back. Mutual information measures contribution: how much did it change the answer. They are different axes, and an evaluation that reports only the first cannot see what retrieval actually did for generation.

⚠ Rigor flag

Every “bit” here is measured against a synthetic answer model, not a real LLM generator. The answer distributions $p(a\mid q)$ and $p(a\mid q,d)$ are a softmax over a small fixed answer set induced from the same von Mises–Fisher finance geometry the dense-retrieval topic used (four sectors, two companies each, one filing per company) — an exponential-family stand-in chosen so that pmi, the per-document KL information gain, and the conditional mutual information all have closed forms the notebook owns to the decimal. A transformer’s $p(a\mid q,\text{context})$ is intractable and would have to be estimated, with its own variance. Four further joints. MI estimation from samples is hard and biased: the variational bounds we name — InfoNCE among them — are the practical tools, and InfoNCE is a lower bound ceilinged at $\log(N{+}1)$ , so a measured “bits added” from a finite batch saturates exactly as the dense-encoder topic’s bound did; the identity $I = H(A\mid Q) - H(A\mid Q,D)$ holds for the true distributions, not their finite-sample estimates. The “distractor costs bits” claim is about $\operatorname{pmi}(a^\star; d\mid q) < 0$ for the true answer under this model — a property of (answer, document) pairs, not of relevance labels; pmi can be negative for a relevant document too when the answer set is coarse. Saturation is chain-rule arithmetic, exact for the model, but real second documents are neither perfectly redundant nor independent. And bits-added is a different axis from recall by construction here: recall is saturated because retrieval is easy, which is the cleanest way to show the dissociation, not a defect. Queries are sector-ambiguous synthetic vMF draws, not natural language.

§ How many bits does each retrieved filing add to the answer? text pdf

An analyst asks which company in the energy sector took a specific impairment charge. Before retrieval the generator’s prior spreads over the sector’s companies — it knows the sector from the question but not the name, an answer entropy of roughly a bit. A relevant 10-K passage that names the charge sharpens the posterior onto the right company and adds positive bits; a same-sector 10-Q from a different company that discusses a similar charge is a distractor — it sharpens the posterior onto the wrong name and costs bits at the truth, the information-theoretic signature of a confident hallucination. A second copy of the boilerplate risk-factors section adds essentially nothing: the posterior is already placed. And the passage the similarity search ranked first — the one with the highest cosine to the query — may add fewer bits than a lower-ranked disclosure that actually resolves the ambiguity, because rank measures lexical and semantic proximity, not the reduction in answer uncertainty. The numbers here come from a synthetic vMF stand-in for that pipeline, not a benchmark; the mechanism is what the case study illustrates.

What retrieval adds, in one line

The evaluation layer asked whether the right documents came back; this topic asks how many bits they add to the answer. The answer is the conditional mutual information $I(A;D\mid Q) = H(A\mid Q) - H(A\mid Q,D)$ , assembled from a prior $p(a\mid q)$ , a posterior $p(a\mid q,d)$ , and the pointwise log-ratio $\operatorname{pmi}(a;d\mid q)$ between them. The pieces carry the lessons forward: a distractor with negative pmi is the seed of the faithfulness question — when does retrieved evidence ground an answer versus mislead it; the chain rule and diminishing returns are the relevance term in context selection; and the prior-to-posterior sharpening, read as a channel, is the retriever as a noisy channel whose recall and precision bound the bits it can deliver — Fano’s inequality lower-bounds the answer error by the residual entropy $H(A\mid Q,D)$ . The dense encoder maximized a lower bound on exactly these bits. We now have the unit the rest of the information-theory layer is measured in.

Connections

the direct prerequisite supplies both the encoder and the corpus this topic measures: the same von Mises–Fisher finance geometry (four sectors, two companies each, one filing per company) induces the answer model here, and the dual encoder whose rank-d ceiling that topic proved is exactly the retriever whose per-document bits we now quantify — the score it ranks by and the information it adds are two readings of one inner product dense-retrieval-dual-encoders
that topic proved InfoNCE is a lower bound on the mutual information I(q;d⁺) ceilinged at log(N+1); this topic measures that same mutual information directly as the bits retrieval adds, so the contrastive loss the encoder minimized and the retrieval value we compute are the same quantity approached from opposite sides — training maximized a lower bound on exactly the I(Q;D) this topic estimates, and the log(N+1) saturation reappears as the ceiling on certifiable bits infonce-contrastive-objective
that topic scored retrieval by whether the relevant set came back — recall, AP, MAP over the same legs and corpus; this topic shows bits-added is a DIFFERENT axis: on an easy corpus where the gold filing is always retrieved, recall reads 1.0 for every query while the information actually delivered varies, so presence is not contribution set-metrics-precision-recall-map-mrr
the probability ranking principle ranks documents by probability of relevance to minimize expected loss; pointwise mutual information gives the information-theoretic relevance score behind that ordering — ranking by how much each document sharpens the answer posterior is ranking by bits, so this topic supplies the bits-based relevance signal the ranking principle assumes exists and orders by probability-ranking-principle
the query-likelihood model scores a document by the probability it generates the query, P(q|d); pointwise mutual information is the symmetric log-ratio relating that generative score to relevance, since pmi(q;d) = log P(q|d)/P(q) is the query-likelihood log-odds, so this topic reframes the language-model relevance score that topic built as the bits a document carries about the query query-likelihood-language-models
the capstone wired a full filings-and-transcripts RAG pipeline and measured it with recall@k; this topic adds the information-theoretic axis that pipeline never reported — how many bits each retrieved passage adds to the generated answer — and exhibits the distractor that costs bits, the redundant passage that adds none, and the gap between a passage's rank and its information value capstone-multimodal-financial-rag

References & Further Reading

paper A Mathematical Theory of Communication — Shannon (1948) The founding paper: entropy H as the bits of uncertainty in a source, mutual information as the entropy a channel removes — the H(A|Q) − H(A|Q,D) = I(A;D|Q) identity this topic instantiates is Shannon's; Bell System Technical Journal 27 (1948)
book Elements of Information Theory — Cover & Thomas (2006) The standard reference for entropy, KL divergence, mutual information, the chain rule I(A;D₁,D₂|Q) = I(A;D₁|Q) + I(A;D₂|Q,D₁) behind the saturation panel, and pmi as the log density-ratio whose expectation is MI; 2nd edition, Wiley
paper Word Association Norms, Mutual Information, and Lexicography — Church & Hanks (1990) The origin of pointwise mutual information as an NLP association score: log P(x,y)/(P(x)P(y)) for word co-occurrence — the same pmi we apply to (answer, document) pairs; Computational Linguistics 16(1)
paper Representation Learning with Contrastive Predictive Coding — van den Oord, Li & Vinyals (2018) InfoNCE / CPC: minimizing the loss maximizes a lower bound on the mutual information between query and positive, I(q;d⁺) ≥ log(N+1) − L — the bound this topic measures against, and the source of the log(N+1) saturation ceiling on certifiable bits
paper On Variational Bounds of Mutual Information — Poole, Ozair, van den Oord, Alemi & Tucker (2019) The unifying treatment of variational MI bounds — the bias–variance tradeoff of MI estimators, InfoNCE as a low-variance lower bound, and why estimating MI from samples is hard — the methodological caveat behind this topic's MI-estimation rigor flag; ICML 2019
paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel, Riedel & Kiela (2020) The RAG architecture: a retriever marginalizes a generator p(a|q) = Σ_d p(d|q) p(a|q,d) over retrieved documents — the prior-to-posterior sharpening this topic measures the bits of, with p(a|q,d) the per-document posterior; NeurIPS 2020
book Information Theory, Inference, and Learning Algorithms — MacKay (2003) Entropy and mutual information from a Bayesian-inference viewpoint, and Fano's inequality lower-bounding answer error by the conditional entropy H(A|Q,D) — the bridge to the retriever-as-noisy-channel topic; Cambridge University Press, freely available