The Retriever as a Noisy Channel: Recall, Precision, and Information Limits

From “how many bits did it add” to “how many bits survive the channel”

The previous topic measured what a retrieved document is worth to the generator in bits: the conditional mutual information $I(A;D\mid Q) = H(A\mid Q) - H(A\mid Q,D)$, the entropy retrieval removes from the answer. That measurement assumed the document arrived. A deployed retriever is not so obliging — it misses relevant filings and returns plausible wrong ones — and the question this topic asks is what those failures do to the bits, and what the bits do to the answer.

The reframing is a single move: read the retriever as a communication channel. A query $q$ enters; the channel emits a context — one retrieved document $d \sim p(d\mid q)$; the generator decodes an answer $\hat a$. This is the noisy-channel reading of retrieval that goes back to Berger and Lafferty, made information-theoretic: a channel has a capacity — the most information it can carry — and a converse — Fano’s inequality, which turns whatever uncertainty the channel leaves into a floor on the decoder’s error. The two failure modes of a retriever are the two classical ways a channel corrupts a message, and they map onto the two words of this topic’s title. A recall failure is an erasure; a precision failure is a substitution. They are co-equal headlines, and they tell opposite stories about what an information bound can see.

We will treat the two channels in turn. The corpus is the one the previous topic built and this one never rebuilds: four sectors, two companies each, one filing per company, so $K=8$ answers that are also the $8$ documents, with $32$ sector-ambiguous queries and temperatures $\tau = \tau_{\mathrm{doc}} = 0.2$. On the clean channel the answer model carries $I(A;D\mid Q) = 1.18$ bits, dropping $H(A\mid Q) = 2.06$ bits of prior uncertainty to $H(A\mid Q,D) = 0.88$ bits — the numbers the notebook owns and asserts.

Recall is an erasure channel; its capacity is the recall

The cleanest channel in information theory is the binary erasure channel: each symbol arrives intact with probability $1-\varepsilon$ or is replaced by an erasure mark $?$ with probability $\varepsilon$, and the receiver always knows which. Its capacity is $1-\varepsilon$: the fraction of symbols that survive. A retriever that misses the relevant filing with probability $1-\mathrm{recall}$ is exactly this channel for the answer, and the consequence for the delivered bits is an identity, not an approximation.

So recall is literally the surviving fraction of the relevant bits — the erasure-channel capacity, read off the retriever. Panel A of the laboratory makes the picture concrete: as you lower the recall, the delivered-bits meter empties in proportion and the answer belief flattens from its peaked clean shape toward the uniform distribution. Nothing about the answers changed; the channel simply passed fewer of the bits that distinguish them.

A · the channelB · recall floorC · precision gapD · rate–distortion

retriever recall (1 − erasure rate) = 0.60

bits delivered I = recall · I₀

BEC capacity = recall

0.60

bits delivered I = recall·I₀

0.710 bits

answer-belief entropy

2.695 bits

A retrieval miss erases the relevant filing (probability 1 − recall), and the generator falls back to a non-informative belief. Because the erased symbol is independent of the answer, the retriever is a Binary Erasure Channel: the surviving fraction of the relevant bits is exactly the recall, so the delivered information is I = recall · I₀ and the belief flattens toward uniform.

Finance vMF corpus reused from the PMI topic (8 companies across 4 sectors, one filing per company), 32 sector-ambiguous queries; the answer model is a synthetic softmax stand-in, not an LLM. Clean channel: I₀ = 1.183 bits, H(A|Q) = 2.059 → H(A|Q,D) = 0.876. Numbers mirror retriever_as_noisy_channel.py; the lab recomputes entropy, the Fano floors, and the channel capacities in closed form.

Fano’s inequality: the residual entropy is a floor on the answer error

What recall fails to deliver does not vanish; it stays behind as residual uncertainty. Under the erasure channel the residual entropy is a convex blend,

$$H_{\mathrm{erasure}}(A\mid Q,D) = \mathrm{recall}\cdot H(A\mid Q,D) + (1-\mathrm{recall})\cdot \log_2 K,$$

rising from $0.88$ bits on the clean channel toward the full $\log_2 8 = 3$ bits of a uniform guess. The reason this residual matters is Fano’s inequality, the converse half of channel coding: it lower-bounds the error of any decoder by the uncertainty the channel leaves.

The loose form on the right makes the structure visible. The floor is vacuous while $H(A\mid Q,D) < 1$ bit: a channel that leaves less than one bit of uncertainty permits, in principle, a perfect decoder. As recall falls and the residual climbs past one bit, the floor lifts off zero and rises — on this corpus the crossover is at $\mathrm{recall} \approx 0.94$, after which the floor climbs to $0.67$ at full erasure, where the generator can do no better than guess. Panel B draws all of it: the residual entropy rising, the Fano floor (loose and tight) lifting off at the one-bit line, and the model’s Bayes error sitting above the floor at every operating point, exactly as the theorem requires (test_fano_is_a_theorem). This is the sense in which recall sets the floor: the bits the retriever fails to deliver become a hard lower bound on the generator’s error that no amount of generation skill can beat.

Precision is a substitution channel — and Fano goes blind

The recall story is the optimistic one: the channel is honest about what it does not know, the residual entropy rises, and Fano’s floor faithfully tracks the growing difficulty. The precision story is the treacherous one. A false positive does not erase the relevant filing — it substitutes a plausible distractor, and the generator reads it with full confidence.

This is a proposition we run, not assume. Across the query set, reading the gold filing is correct for every query and reading the same-sector distractor is wrong for every query, so the realized error is exactly $\varepsilon$; meanwhile the residual entropy creeps only from $0.60$ to $0.87$ bits — never reaching the one bit Fano needs to react — so the floor reads $0$ at every $\varepsilon$ and the confident-wrong gap grows from $-0.12$ to $+0.76$ (Panel C, test_confident_wrong_gap_widens). The asymmetry with the recall case is the whole point. Recall failures are honest: they raise the entropy, and Fano accounts for them. Precision failures are confident: they keep the entropy low while making the answer wrong, and no entropy bound accounts for them at all.

The operating point is a rate–distortion choice

The two failures meet in a single design decision. Reading more documents raises recall (more chances to catch the relevant filing) but lowers precision (more false positives in the context), and the generator’s belief blends them. On this corpus the gold filing is always rank one, so recall@k is pinned at $1$ and contamination cannot show up as a wrong answer — it shows up as bits of uncertainty added: as $k$ grows from $1$ to $8$, precision falls as $1/k$ while the blended belief’s entropy rises from $0.60$ toward the full marginal $H(A\mid Q) = 2.06$ bits (Panel D). The retriever is choosing a point on a rate–distortion curve — spending rate (bits read) to buy down distortion (answer error) — which is also the information bottleneck the generator imposes on the corpus: keep the bits that predict the answer, discard the rest. Erasure drops the bits the answer needs; substitution injects bits it must not trust.

What the channel can and cannot deliver, in one line

A retriever is a channel, and its two metrics are the channel’s two error mechanisms. Recall is the erasure capacity — the surviving fraction of the relevant bits — and what it fails to deliver becomes, by Fano’s inequality, a floor on the answer error that no generator beats: $P_e \ge (H(A\mid Q,D) - 1)/\log_2 K$. Precision is the substitution rate, and its failures are confident: they keep the residual entropy low while driving the realized answer wrong, so the information floor sees nothing and the gap it misses is the calibration problem. The bits a retriever delivers bound what the generator can do; whether the generator does it is a question of trusting the right bits — the relevance term in context selection, the faithfulness of the grounded answer, and the calibration of its confidence, each a layer above this one.