advanced generation-grounding 30 min read

Faithfulness and Groundedness as Measurable Quantities

HyDE modeled the quality of a generated answer with a single hallucination rate p. Here we stop modeling it with a knob and measure it on the text itself — as two numbers, not one. Faithfulness is the precision of an answer's atomic claims against the retrieved context (what fraction of what you said is supported); groundedness is the recall (what fraction of the supportable facts you used). They diverge, a noisy judge measures both with a bias we debias and a confidence we calibrate, and trading coverage for guaranteed faithfulness is the abstention frontier.

Part of the Generation & Grounding track · View full curriculum →

Prerequisites: Query Transformation and HyDE: Correcting Distribution Shift in Embedding Space Pointwise Mutual Information: What Retrieval Adds to Generation, in Bits

From a hallucination rate to a measured quantity

HyDE left us with a knob. It wrote a hypothetical answer, embedded it, and retrieved real documents near it — and its whole risk lived in a single number, the rate $p$ at which the generator hallucinated about the entity in the question. We could turn that knob and watch recall fall, but we never measured it. The closing question was explicit: turning $p$ into a measured quantity, on real generated text rather than a synthetic von Mises–Fisher knob, is the next step.

That is the step we take here. The object we measure is not a knob but a generated answer, and the first thing to notice is that “is it faithful?” is the wrong question — it has two answers, not one. An answer is a set of atomic claims; the context the system retrieved is a set of supportable facts. The two sets can fail to coincide in two independent ways. The answer can say things the context does not support — it can hallucinate — and it can omit things the context does support — it can be incomplete. Faithfulness is the first failure; groundedness is the second. A single “factuality score” collapses them, and in doing so hides the trade between them that every generation system actually makes.

Faithfulness is precision, groundedness is recall

Write the answer’s atomic claims as $C = \{c_1, \dots, c_n\}$ , and let $S \subseteq C$ be the subset of claims the context supports. Let $F$ be the set of supportable facts in the context. Each supported claim grounds one fact in $F$ — it is an instance of that fact in the answer.

Definition 1 (Faithfulness and groundedness).

The faithfulness of an answer is the precision of its claims against the context,

\text{faithfulness}(C) \;=\; \frac{|S|}{|C|} \;=\; \frac{\#\{\text{supported claims}\}}{\#\{\text{claims made}\}},

and the groundedness (or coverage) is the recall of the supportable facts,

\text{coverage}(C) \;=\; \frac{|\{\, f \in F : \text{some claim in } S \text{ grounds } f \,\}|}{|F|} \;=\; \frac{\#\{\text{facts the answer used}\}}{\#\{\text{facts available}\}}.

These are not new metrics. They are precision and recall — the same pair the evaluation layer built for retrieval — read over an answer’s claims instead of a ranking’s documents. Faithfulness divides by the number of claims the answer makes; coverage divides by the number of facts the context offers. The two denominators are different objects, and that difference is the whole story: a precision over claims and a recall over facts can move independently because a longer answer adds to the precision denominator without touching the recall denominator, and a fact can be grounded by many claims or none.

The collapse anchor pins the definition to the imported machinery. A perfect judge — one with sensitivity and specificity equal to one — labels each claim’s support exactly, so its measured faithfulness is the mean of the true support labels, which is precisely precision_at_k over the claim set. The notebook asserts this equality to machine precision; faithfulness is not like precision, it is precision, over the right index set.

confidence cut τ (retain claims with conf ≥ τ) = 0.50

faithfulness = precision

1.00

coverage = recall

1.00

claims retained

6 / 9

hallucinations kept

16 synthetic finance filings (one vMF company prototype each), context = top-3retrieved; a claim is supported iff its cosine to a context fact clears 0.78. The lenient judge (AUC 0.83) over-endorses, so the naive faithfulness 0.70 is biased above the oracle 0.62. Numbers mirror faithfulness_groundedness.py; the lab recomputes Panel A's precision/coverage at the live τ and Panel C's ECE from the baked reliability bins, and bakes the corpus-derived confidences, frontier, calibration, and per-claim bits.

Terse versus verbose: why the two diverge

Because faithfulness counts over claims and coverage over facts, the two pull apart as the answer’s length changes.

Proposition 2 (Precision and recall diverge with verbosity (demonstrated)).

On the worked corpus, a terse answer of a single supported claim has faithfulness $1.00$ and coverage $0.33$ ; a verbose answer of a dozen claims, half of them invented, has faithfulness $0.53$ and coverage $1.00$ . The two move in opposite directions, and their harmonic mean — the F1 of generation — peaks at an interior answer length (five claims here), neither a single claim nor an exhaustive one.

The intuition is the one any careful writer knows. A terse answer states only what it is sure of: it is faithful but thin. A verbose answer reaches for completeness, and somewhere past the facts it actually has, it begins to invent — coverage saturates while precision erodes. Reporting one number forces a choice the data does not support. The honest summary is the pair, and the geometry that produces it is on the frontier panel above: the terse and verbose answers are the two extremes of one precision–recall curve.

Measuring with a noisy judge

On synthetic text we have an oracle for support. On real text we do not — we have an LLM judge, and a judge is an instrument with error. Its sensitivity (the rate at which it endorses a genuinely supported claim) and its specificity (the rate at which it rejects an unsupported one) are both below one, and a lenient judge that over-endorses will report a faithfulness number that is systematically wrong.

Proposition 3 (The judge biases the measurement).

The naive faithfulness — the mean of the judge’s per-claim verdicts — estimates not the latent supported fraction $\pi$ but $p_{\text{obs}} = \pi\,\mathrm{se} + (1-\pi)(1 - \mathrm{sp})$ , where $\mathrm{se}, \mathrm{sp}$ are the judge’s sensitivity and specificity. For an over-endorsing judge this overstates $\pi$ : on the worked panel the judge reports faithfulness $0.70$ where the oracle is $0.62$ .

The fix is the prevalence correction from epidemiology, where the same problem — estimating how many people have a disease from an imperfect test — is a century old.

Theorem 4 (Rogan–Gladen debiasing).

Inverting the bias relation gives the unbiased estimate

\hat\pi \;=\; \frac{p_{\text{obs}} + \mathrm{sp} - 1}{\mathrm{se} + \mathrm{sp} - 1},

whose variance scales as $1/J^2$ in the Youden index $J = \mathrm{se} + \mathrm{sp} - 1$ and diverges as the judge degrades toward a coin flip. With the audited rates, the correction recovers the oracle faithfulness exactly; the notebook pins $\hat\pi$ to $0.62$ .

The correction is only as good as the audit, and the judge’s confidence carries a second problem: it is not a probability. A judge that says ” $0.9$ supported” should be right nine times in ten, and an over-confident one is not. We measure that gap with the expected calibration error and close it with the recalibrators the evaluation layer built — Platt scaling and isotonic regression — fit on held-out claims. The raw judge here has ECE $0.125$ ; Platt lowers it to $0.120$ and isotonic to near zero, and Platt does so while preserving the ranking exactly, so a recalibrated confidence is a probability we can threshold without disturbing which claims the judge prefers. The calibration panel above shows the reliability diagram move toward the diagonal.

The frontier and the back-off guarantee

A calibrated confidence is a cut we can turn. Retain only the claims the judge is confident enough about, and the answer’s precision rises as its recall falls — the back-off frontier, swept by the confidence cut $\tau$ .

Algorithm 5 (Confidence-cut back-off).

Given calibrated per-claim confidences $\tilde c_i$ and a cut $\tau$ , the retained answer is $\hat C(\tau) = \{\, i : \tilde c_i \ge \tau \,\}$ . Raising $\tau$ removes the least-confident claims; faithfulness rises and coverage falls along the frontier. An empty retained set is an abstention.

A heuristic cut gives a frontier but no promise. The promise comes from conformal risk control, which fixes the cut so that the expected false-claim rate is bounded — a distribution-free, finite-sample guarantee on a monotone loss.

Theorem 6 (Conformal risk control bounds the false-claim rate).

Let the per-answer loss be the fixed-denominator false-claim rate $L(\lambda) = \tfrac{1}{k}\#\{\text{retained and unfaithful at cut } \lambda\}$ , which is non-increasing in $\lambda$ and bounded above by $B = 1$ . Writing $\hat R_n(\lambda)$ for the empirical mean of $L(\lambda)$ over the $n$ calibration answers, the threshold

\hat\lambda \;=\; \inf\Big\{ \lambda : \tfrac{n}{n+1}\hat R_n(\lambda) + \tfrac{B}{n+1} \le \alpha \Big\}

controls the expected test loss, $\mathbb{E}[\,1 - \text{faithfulness}(\hat\lambda)\,] \le \alpha$ . On the worked panel at $\alpha = 0.10$ the realized false-claim rate is $0.03$ , retaining about a third of the claims.

The fixed denominator is load-bearing: dividing by the shrinking retained count gives a fraction-of-retained loss that is not monotone, and conformal risk control silently fails on it — the notebook reproduces the counterexample so the prose does not claim the naive loss works. And the guarantee is in expectation over the calibration draw, not a single realization: a deployment can overshoot, the honest caveat the rigor note records. Trading coverage for a guaranteed faithfulness this way is the entry to selective generation — when the certified-faithful answer is too thin to be useful, the system should abstain rather than emit. Choosing when to abstain is the next topic.

Bits of grounding

There is a second reading of “grounded,” and it is information-theoretic. A claim is grounded when the context raises its probability — when knowing the retrieved filings makes the claim more likely than it was a priori. That is exactly a positive pointwise mutual information.

Proposition 7 (A supported claim has positive PMI with the context (demonstrated)).

For a claim whose nearest answer prototype is $a$ , the bits the context contributes are

\mathrm{pmi}(a\,;\,\text{ctx} \mid q) \;=\; \log_2 \frac{p(a \mid \text{ctx}, q)}{p(a \mid q)},

positive when the context concentrates the answer onto $a$ and non-positive when it argues elsewhere. On the panel, supported claims average $+0.49$ bits and hallucinated claims $-0.73$ bits, with $94\%$ of hallucinations carrying non-positive bits.

This closes the loop with the prerequisite that supplied the answer model. Faithfulness, the judge’s verdict, and positive bits-of-grounding are three views of one event: a claim the retrieved context supports. The bits panel above shows the sign split — the supported claims piled on the positive side of zero, the hallucinations on the negative.

§ A filings answer, claim by claim text pdf news

A portfolio manager asks how a company’s most recent quarter looked. The system retrieves the 10-K management discussion and a recent earnings item, and the generator writes a paragraph that decomposes into nine atomic claims. Five are supported by the filings — the revenue figure, the year-over-year growth, the consensus beat, the margin, the raised guidance — and a sixth, the foreign-exchange headwind, is supported by the risk factors. Three are not: a specific cloud-segment growth rate the filings never give, a buyback the company did not announce, and a fabricated data-center location. Faithfulness is six-of-nine; coverage is complete, since every supportable fact appears. The three inventions are exactly the claims a lenient judge endorses at moderate confidence and a careful back-off removes — and exactly the claims whose pointwise mutual information with the retrieved filings is negative. Raising the confidence cut past the inventions recovers a faithful answer without dropping a single supported fact.

Where this sits

We began with a knob and end with a measurement. Faithfulness and groundedness are precision and recall over an answer’s claims; they diverge, so we report both. A noisy judge measures them with a bias we remove by Rogan–Gladen and an over-confidence we remove by calibration. And the back-off frontier — coverage traded for a conformally guaranteed faithfulness — is the decision surface of the next topic, selective generation, where the system that cannot certify enough faithful claims should abstain rather than guess. The bits-of-grounding reading ties the whole construction back to the information the context carries, the thread the retrieval-information-theory layer began.

⚠ Rigor flag

The judge here is a synthetic Bernoulli rater with known, audited sensitivity and specificity, not a real language-model judge whose error rates drift across domains and prompts; Rogan–Gladen debiasing is exact only for a homogeneous judge and before the clip, and a real judge’s heterogeneous, drifting bias is corrected only imperfectly by one audited rate pair. The conformal back-off bounds the expected false-claim rate over the calibration draw, not a single realization, and assumes exchangeability of calibration and test claims, which intra-answer dependence stresses. The construction rests on a claim-decomposition assumption — that an answer cleanly factors into independent, separately-checkable atomic claims — which natural language only approximately satisfies, and “supported” means supported under the synthetic geometric grounding oracle, not human judgment. The numbers are exact for the model and illustrative of a deployed generator; a tested notebook runs every claim and pins the headline numbers to the observed seed.

Connections

The direct prerequisite and the handoff: HyDE modeled the faithfulness of its generated hypothetical with a single hallucination rate p that no amount of Monte-Carlo averaging could remove. This topic turns that rate into a measured, two-sided quantity on real generated claims rather than a synthetic von Mises–Fisher knob, picking up exactly where HyDE's open question left off. query-transformation-hyde
Supplies the bits-of-grounding view: a supported claim is exactly one whose pointwise mutual information with the retrieved context is positive, and a hallucination one whose PMI is non-positive. We import its answer-posterior and entropy machinery to score each atomic claim's grounding in bits, the information-theoretic reading of faithfulness. pmi-retrieval-value
The judge that measures faithfulness is the noisy Bernoulli instrument that topic built; faithfulness is the mean of its per-claim verdicts, a biased estimate of the latent supported fraction that Rogan–Gladen debiases. We import its judge form, audited sensitivity and specificity, and Rogan–Gladen correction to read the raw faithfulness number honestly. llm-as-judge-ragas
The distribution-free back-off guarantee that turns a confidence cut into a controlled false-claim rate. We import its conformal-risk-control machinery so that trading coverage for guaranteed faithfulness is a finite-sample guarantee on a monotone loss, not just a heuristic threshold. conformal-factuality
Supplies the calibration suite that makes the judge's confidence a probability before it can drive the back-off cut: the reliability diagram, the expected calibration error, and the Platt and isotonic recalibrators we import to recalibrate the judge in the calibration panel. significance-testing-calibration
The root of the evaluation layer and the home of precision and recall, the two metrics this topic recognizes as faithfulness and groundedness over atomic claims rather than retrieved documents. The collapse anchor is exact: a perfect judge's faithfulness equals the imported precision at k to machine precision. set-metrics-precision-recall-map-mrr
Provides the finance document manifold the whole construction sits on: one von Mises–Fisher company prototype per filing, so a supported claim aligns with its company's context direction and a hallucinated claim points off it. We reuse its geometry and separable score to build the claim-grounding model. dense-retrieval-dual-encoders

References & Further Reading

paper Precise Zero-Shot Dense Retrieval without Relevance Labels — Gao, Ma, Lin & Callan (2023) HyDE: generate a hypothetical answer, embed it, retrieve real documents near it. The prerequisite whose open question — is the hypothetical faithful? — this topic measures.
paper FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation — Min, Krishna, Lyu, Lewis, Yih, Koh, Iyyer, Zettlemoyer & Hajishirzi (2023) Decomposes a generated answer into atomic facts and scores the fraction supported by a source — factual precision as the mean of per-claim support, the operational definition of faithfulness here.
paper RAGAs: Automated Evaluation of Retrieval Augmented Generation — Es, James, Espinosa-Anke & Schockaert (2024) Defines faithfulness (claims entailed by the context) and context recall (ground-truth claims covered) as the precision/recall pair this topic formalizes and measures with a judge.
paper SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models — Manakul, Liusie & Gales (2023) A sampling-based hallucination detector — evidence that the per-claim support signal a judge produces is a measurable quantity, not a binary verdict.
paper Estimating Prevalence from the Results of a Screening Test — Rogan & Gladen (1978) The prevalence correction that debiases a measurement made with an imperfect instrument: it inverts the judge's sensitivity and specificity to recover the latent supported fraction.
paper Conformal Risk Control — Angelopoulos, Bates, Fisch, Lei & Schuster (2024) Generalizes split conformal from a coverage guarantee to the control of any monotone, bounded loss in expectation — the machinery that turns the faithfulness back-off into a finite-sample guarantee.
paper Language Models with Conformal Factuality Guarantees — Mohri & Hashimoto (2024) Per-claim back-off with a conformal guarantee on the false-claim rate — the generation-layer application this topic frames as the abstention frontier.
paper Predicting Good Probabilities with Supervised Learning — Niculescu-Mizil & Caruana (2005) The empirical study of Platt scaling and isotonic regression as recalibrators — the recalibration this topic applies to the judge's over-confident scores.
paper Word Association Norms, Mutual Information, and Lexicography — Church & Hanks (1990) The origin of pointwise mutual information in language: log of the ratio of joint to independent probability — here, the bits a context adds to a claim.
paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel, Riedel & Kiela (2020) The RAG framework whose generated answer this topic measures against its retrieved context.