advanced generation-grounding 28 min read

Selective Generation: When a RAG System Should Abstain

The faithfulness topic ended at a back-off frontier — coverage traded for a conformally guaranteed faithfulness — and left a question: what should the system do when the certified-faithful answer is too thin to be worth emitting? It should abstain. The two prior topics decided which claims to keep inside an answer; this one decides whether to answer at all. That answer-level decision has its own mathematics: Chow's cost-optimal reject rule, the risk–coverage curve and its area, the gap a real confidence signal pays against the oracle, a distribution-free guarantee on the wrong-emission rate, and a cost model for abstaining versus erring.

Part of the Generation & Grounding track · View full curriculum →

Prerequisites: Faithfulness and Groundedness as Measurable Quantities Score Calibration, Drift Detection, and Significance Testing for Retrieval

From a back-off frontier to a decision

The faithfulness topic ended at a frontier. It decomposed a generated answer into atomic claims, measured faithfulness as their precision and groundedness as their recall, and then turned a confidence cut: raise it, drop the least-supported claims, and the retained answer becomes more faithful but thinner. Conformal risk control pinned that cut so the expected false-claim rate was bounded. The closing line was a question, not an answer: when the certified-faithful answer is too thin to be useful, the system should abstain rather than emit — and choosing when to abstain was deferred to here.

That deferral marks a change of object. The two prior factuality topics — faithfulness and conformal factuality — both work inside an answer: given the claims the generator produced, they decide which claims to keep. Their units are claims, their back-off removes the weak ones, and an answer that loses every claim abstains only as a side effect. This topic asks the question one level up: given a query, should the system answer at all? The unit is the query; the decision is binary, emit or abstain; and the object that governs it — the risk you accept among the answers you emit, traded against how many queries you are willing to leave unanswered — is genuinely new. It is the difference between editing a draft and deciding whether to send one.

Chow’s rule: the cost-optimal reject

Fix a query and let the system produce a candidate answer together with a score $s \in [0,1]$ meant to be its probability of being correct. A selective policy emits the answer when $s \ge t$ for a threshold $t$ , and abstains otherwise. Two numbers describe such a policy across a set of queries.

Definition 1 (Selective risk and coverage).

For a selective policy that emits on a subset of queries, define

\text{coverage} = \Pr(\text{emit}), \qquad \text{selective risk} = \Pr(\text{wrong} \mid \text{emit}).

Coverage is the fraction of queries answered; selective risk is the error rate among the answered. Both are read over queries, not over an answer’s claims — the answer-level counterpart of the claim-level retention and false-claim rate the back-off frontier swept.

Why abstain at all? Because the costs are asymmetric. Emitting a wrong answer is expensive; abstaining — returning “I cannot answer this from the retrieved filings,” deferring to a human — is cheaper, but not free. Write $c_{\mathrm{err}}$ for the cost of emitting a wrong answer, $c_{\mathrm{abs}}$ for the cost of abstaining, and $0$ for a correct answer. The optimal policy compares the expected cost of the two actions.

Theorem 2 (Chow's optimal reject rule).

The expected cost of emitting an answer with probability of error $\Pr(\text{wrong}) = 1 - s$ is $c_{\mathrm{err}}(1-s)$ ; the cost of abstaining is $c_{\mathrm{abs}}$ . Emitting is preferable exactly when $c_{\mathrm{err}}(1-s) \le c_{\mathrm{abs}}$ , i.e.

\text{emit} \iff s \;\ge\; 1 - \frac{c_{\mathrm{abs}}}{c_{\mathrm{err}}} \;=\; t^\star.

This is Chow’s rule (1970): the optimal reject threshold on the probability of being correct is one minus the cost ratio. The degenerate limits are sane — free abstention ( $c_{\mathrm{abs}} = 0$ ) gives $t^\star = 1$ , emit only when certain, effectively always abstain; abstaining at least as costly as erring ( $c_{\mathrm{abs}} \ge c_{\mathrm{err}}$ ) gives $t^\star = 0$ , always emit.

With a wrong financial answer costing five times an abstention ( $c_{\mathrm{err}} = 5$ , $c_{\mathrm{abs}} = 1$ ), the cutoff is $t^\star = 0.8$ : emit only the answers the system is at least $80\%$ confident are correct. The rule is exact if $s$ is the true probability of correctness — a caveat we return to, because the score here is a calibrated but imperfect judge, and the gap is the whole reason the next sections exist.

The risk–coverage curve and its area

Chow’s rule picks one threshold; sweeping the threshold traces the whole trade. Order the answers by score and emit the most-confident first: as the threshold falls, coverage rises and the system is forced to emit answers it is less sure of, so selective risk rises too. The plot of selective risk against coverage is the risk–coverage curve, and the panel below lets you walk it.

coverage (emit the k most-confident answers) = 31%

selective risk (achievable)

0.000

selective risk (oracle)

0.000

AURC (achievable)

0.185

excess AURC (gap)

0.088

signal AUC

0.773

32 synthetic finance answers (one per query, base error 0.406); the lenient judge's answer score has AUC 0.77 — informative but imperfect, so the achievable risk–coverage curve sits above the oracle (excess AURC 0.088). At the default cost ratio selective generation costs 0.66, beating always-emit (2.03) and always-abstain (1.00). Numbers mirror selective_generation_abstention.py; the lab recomputes the cost curve, the two gates, and the conformal threshold live from the baked (score, correct) cloud. The Chow rule assumes a calibrated score (ECE 0.220 here) — the calibration gap is why the realized optimum can drift from t⋆.

The curve has an area, and the area is the summary number.

Definition 3 (AURC and the excess over the oracle).

The area under the risk–coverage curve is

\mathrm{AURC} = \int_0^1 \mathrm{risk}(c)\, dc,

the answer-level mirror of average precision = area under the precision–recall curve. Lower is better: a perfect confidence signal keeps selective risk at zero until coverage is forced past the base accuracy. The oracle curve orders answers by truth instead of score — every correct answer emitted before any wrong one — and is the lowest curve any signal can achieve. The excess AURC, achievable minus oracle, is non-negative and vanishes exactly when the score orders correctness perfectly.

On the worked panel of $32$ synthetic finance answers, the base error rate is $0.41$ , so always-answering is wrong two times in five. The judge’s answer score discriminates correct from incorrect with an AUC of $0.77$ — informative but far from perfect — so the achievable AURC is $0.185$ against an oracle of $0.097$ , an excess of $0.088$ . That excess is the price of an imperfect signal, and it is the quantity every confidence method is trying to shrink: the panel’s perfect-judge collapse shows that when the score equals the truth, the achievable curve lies exactly on the oracle and the excess is zero. The gap is the signal’s quality — the same fact, read as an area, that the AUC reads as a ranking.

A guarantee on the wrong-emission rate

The risk–coverage curve is descriptive. The guarantee comes from lifting conformal risk control — the machinery the faithfulness topic used on claims — up to the answer level. The subtlety is choosing the right loss, and it is exactly the subtlety the claim-level topic flagged.

Theorem 4 (Answer-level conformal selective-risk control).

Let the per-answer loss at threshold $t$ be the wrong-emission loss with a fixed denominator,

L(t) = \frac{1}{N}\#\{\text{emitted and wrong at } t\},

which is non-increasing in $t$ — raising the bar can only stop an answer being emitted — and bounded in $[0,1]$ . The conformal-risk-control threshold

\hat t = \inf\Big\{ t : \tfrac{n}{n+1}\hat R_n(t) + \tfrac{1}{n+1} \le \alpha \Big\}

computed on $n$ calibration answers controls the expected wrong-emission rate, $\mathbb{E}[L(\hat t)] \le \alpha$ .

The fixed denominator is what makes the loss monotone, and the choice is load-bearing in exactly the way the claim-level topic warned. The unconditional wrong-emission rate — confident wrong answers as a fraction of all queries — is monotone in the threshold, so conformal risk control governs it. The conditional selective risk — wrong answers as a fraction of the emitted ones, the risk–coverage y-axis — divides by a shrinking count and is not monotone, so the same machinery silently fails on it. This is the answer-level echo of the false-claim-loss versus fraction-loss distinction, one level up; the notebook reproduces the non-monotonicity rather than asserting it. On the panel, at $\alpha = 0.10$ the threshold $\hat t = 0.78$ holds the test wrong-emission rate to zero while answering about a third of the held-out queries — and because the guarantee is conformal, it is calibration-agnostic: recalibrating the score changes how many answers clear the bar, not whether the rate is controlled.

The two-stage gate: too risky, or too thin

Selective generation does not replace the claim-level back-off; it sits on top of it. The faithfulness hand-off named both failure modes: an answer can be too risky to emit, or — after the unfaithful claims are stripped away — too thin to be worth emitting. The two stages compose.

Algorithm 5 (The two-stage decision).

Given a candidate answer, first run the claim-level conformal back-off to certify the faithful claims, then decide:

\text{emit} \iff \underbrace{s \ge t^\star}_{\text{not too risky}} \;\wedge\; \underbrace{|\hat C_{\hat\lambda}| \ge m}_{\text{not too thin}},

where $\hat C_{\hat\lambda}$ is the certified-claim set and $m$ a minimum-substance floor. An answer that fails either gate abstains: the system would rather say nothing than emit a likely-wrong answer or a certified-but-empty one.

On the worked panel the risk gate at $t^\star = 0.8$ emits $10$ of the $32$ answers, all of them correct — a residual error of zero among the emitted — and defers the other $22$ . Here the two gates are correlated: the answers that fall below the confidence cut are also the ones the back-off leaves thin, so the risk gate binds and the thinness floor removes nothing further at $m = 3$ . That alignment is itself the honest observation — a low-confidence answer is usually a poorly-grounded one — and the minimum-claims slider in the lab shows the thinness gate begin to bite as you demand more substance, abstaining confident-but-sparse answers at a cost. The thinness floor is the editorial criterion the risk model alone does not capture: a single supported sentence can be perfectly faithful and still not an answer.

The cost of being wrong

The point of the cost model is that selective generation must beat both trivial policies, and on a corpus where the signal is good enough, it does.

The expected cost of always emitting is $c_{\mathrm{err}}$ times the base error rate — here $5 \times 0.41 = 2.03$ . The cost of always abstaining is $c_{\mathrm{abs}} = 1.0$ . The selective policy at the Chow cutoff costs $0.66$ : it emits the answers it can stand behind and defers the rest, beating both baselines. But the cost curve in the lab is recomputed live from the answer scores, and its empirical minimum need not fall exactly at the closed-form $t^\star$ — because the score is calibrated only approximately. Its expected calibration error is $0.22$ : a score the system reports as $0.8$ is not correct four times in five. Chow’s rule treats the score as that probability; to the extent it is not, the realized cost-minimizing threshold drifts from $t^\star$ , and the drift is precisely the calibration gap. The rule is no better than the calibration of the number it cuts — which is why the prior topic’s calibration suite is a prerequisite, not a footnote.

§ When the analyst should be paged text pdf news

A portfolio manager asks a RAG system about a company’s latest quarter. For some queries the retrieved filings contain the answer cleanly, the generator’s claims are well-grounded, and the answer score is high — these the system emits. For others the filings are ambiguous or the relevant figure is simply not in the retrieved set; the generator pads with plausible inventions, the certified-claim set after back-off is thin, and the score is low — these the system should not emit. A wrong figure in a portfolio memo is expensive; an “I could not confirm this from the filings; routing to an analyst” is cheap. With the cost of an error five times the cost of a deferral, the optimal policy answers the $31\%$ of queries it is confident about, makes no confident errors on the held-out set, and defers the rest. The operating point is the one an auditor signs off on: the system answers when it can certify enough faithful, substantial claims, and pages a human when it cannot.

Where this sits

This is the terminal node of the generation-grounding layer, and it closes a through-line. The evaluation layer taught that a retrieval metric is an estimator with a standard error; the grounding layer taught that a generated answer is measured by a noisy, calibrated judge; conformal factuality turned that measurement into a guarantee on which claims to keep; and selective generation turns the guarantee into an action — whether to answer at all. The new objects are Chow’s cost-optimal reject rule, the risk–coverage curve and its area, the excess over the oracle that any imperfect signal pays, and a distribution-free bound on the rate of confident wrong answers. The recurring honesty is that none of it removes the calibration gap: a guarantee on the wrong-emission rate and a Bayes-optimal threshold both rest on the score being what it claims to be, and a real judge’s score only approximately is. The next layers — distribution-free factuality and the information-theoretic limits of what retrieval can ground — develop exactly that residual.

⚠ Rigor flag

Chow’s rule is Bayes-optimal only when the answer score is the true probability of being correct; the lenient judge’s score is not, so the realized selective risk differs from the Bayes selective risk and the gap is a calibration gap — quantified here by the score’s expected calibration error and Brier score but not removed. The conformal selective-risk guarantee is marginal: it controls the expected wrong-emission rate over the calibration draw, not a single realization, and assumes exchangeability of calibration and test answers; it bounds the unconditional wrong-emission rate, while the conditional selective risk is non-monotone. The AURC and its excess over the oracle are single-corpus point estimates with no standard error reported here, and “correct” means correct under the synthetic geometric grounding oracle and the synthetic judge, not human judgment. The two-stage thinness floor is an editorial criterion, not a guarantee. A tested notebook runs every claim and pins the headline numbers to the observed seed — and the build-and-run discipline is load-bearing, since a too-weak confidence signal makes selective generation no better than always abstaining.

Connections

The direct prerequisite and the handoff: that topic measured faithfulness and groundedness and traded coverage for a conformally guaranteed faithfulness along a claim-level back-off frontier, ending on the question this topic answers — when the certified-faithful answer is too thin to be useful, the system should abstain. We import its answer-generation model, its faithfulness measurement, and its conformal back-off to build the answer-level decision on top of the claim-level one. faithfulness-groundedness
Supplies the calibration suite that makes the judge's answer-level confidence a probability before Chow's rule can threshold it: the expected calibration error, the Brier score, and the AUC we import to quantify the calibration gap between the score the rule treats as the probability of being correct and the truth. The gap is exactly why the realized cost-minimizing threshold can drift from the theoretical Chow cutoff. significance-testing-calibration
The claim-level conformal back-off whose monotone-loss machinery we reuse at the answer level. Its lesson — that the false-claim loss must use a fixed denominator to stay monotone, while the fraction-of-retained loss is non-monotone and silently voids conformal risk control — recurs here as the distinction between the unconditional wrong-emission rate (monotone, controllable) and the conditional selective risk (non-monotone, the risk–coverage y-axis). conformal-factuality
The noisy, lenient judge whose per-claim confidence we aggregate into the answer-level score the abstention gate keys on. Because the judge over-endorses, that score orders correctness imperfectly — the reason the achievable risk–coverage curve sits strictly above the oracle, and the reason Chow's rule, which assumes a calibrated posterior, is only as good as the score's calibration. llm-as-judge-ragas
Through the imported answer model it supplied, this topic inherits the synthetic finance answer posterior that makes a generated answer correct-or-not under a geometric oracle. The information-theoretic reading carries over: abstaining when the context cannot ground enough bits is the operational form of the bits-of-grounding view. pmi-retrieval-value
The root of the evaluation layer, where the area-under-a-curve summary first appeared as average precision = area under the precision–recall curve. The AURC defined here is its answer-level mirror: the area under the risk–coverage curve, a Riemann sum over coverage rather than over a retrieval ranking. set-metrics-precision-recall-map-mrr
Provides the finance document manifold the whole construction sits on — one von Mises–Fisher company prototype per filing — through the imported faithfulness panel. The answers whose emit-or-abstain decision we study are generated against this geometry, so 'a wrong answer' is one whose claims point off the retrieved filings' directions. dense-retrieval-dual-encoders

References & Further Reading

paper On Optimum Recognition Error and Reject Tradeoff — Chow (1970) The foundational reject-option result: under a cost for errors and a cost for rejecting, the Bayes-optimal classifier abstains exactly when the maximum posterior falls below a threshold set by the cost ratio — the rule this topic applies to the answer-level emit decision.
paper On the Foundations of Noise-Free Selective Classification — El-Yaniv & Wiener (2010) Formalizes the risk–coverage trade-off for a classifier with a reject option — coverage as the fraction predicted on, selective risk as the error among them — the curve this topic reads at the answer level.
paper Selective Classification for Deep Neural Networks — Geifman & El-Yaniv (2017) Brings selective prediction to modern networks, using a model's softmax confidence as the rejection score and choosing a threshold to bound selective risk — the deep-learning ancestor of the answer-level gate, with the confidence signal here a calibrated judge.
paper Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers — Geifman, Uziel & El-Yaniv (2019) Introduces the AURC and excess-AURC metrics — the area under the risk–coverage curve and its gap above the optimal ordering — the summary numbers this topic bakes and reads.
paper SelectiveNet: A Deep Neural Network with an Integrated Reject Option — Geifman & El-Yaniv (2019) Trains the predictor and the reject mechanism jointly at a target coverage — evidence that the emit decision is a learnable object, here decoupled into a calibrated score plus a cost-set threshold.
paper A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks — Hendrycks & Gimpel (2017) Establishes the maximum-softmax-probability confidence baseline whose risk–coverage behavior selective prediction improves on — the simplest version of the score the abstention gate thresholds.
paper Selective Question Answering under Domain Shift — Kamath, Jia & Liang (2020) Trains a calibrator to decide when a QA system should answer versus abstain, especially under distribution shift — the question-answering instance of selective generation this topic formalizes for RAG.
paper Conformal Risk Control — Angelopoulos, Bates, Fisch, Lei & Schuster (2024) Generalizes split conformal to control any monotone, bounded loss in expectation — the machinery this topic lifts from the claim level to bound the answer-level wrong-emission rate at a chosen level.