LLM-as-Judge and Faithfulness: RAGAS as a Family of Estimators

The label we no longer have

The evaluation layer was built on a fixed answer key. Set metrics, NDCG, and significance testing all measured a retrieval system against a qrel — a relevance label fixed in advance — and treated the resulting number as the sample mean of a per-query score, with a standard error, a confidence interval, and a paired test. Generation breaks that contract. A generated answer has no answer key: there is no fixed label that says whether “Q3 revenue rose 12% to $4.2B” is grounded in the retrieved filing. The production response, formalized by RAGAS, is to hire a large language model as a judge and let its verdicts stand in for the missing labels.

The judge, however, is not an oracle. It is a measurement instrument: it answers “is this claim supported?” with imperfect sensitivity and specificity, it carries systematic biases toward longer or earlier-shown candidates, and it adds its own noise on top of the noise we already had from sampling queries. Reading a RAGAS number honestly means treating it as an estimate produced by that instrument, and correcting for the instrument before believing it.

We introduce the notation once. A generated answer is decomposed into $n$ atomic claims $c_1, \dots, c_n$; each claim has a latent truth $Y_i \in \{0, 1\}$ (is it genuinely supported by the context?) with prevalence $\pi = \Pr(Y_i = 1)$, and the judge returns a verdict $\hat Y_i \in \{0, 1\}$. The judge’s sensitivity is $\mathrm{se} = \Pr(\hat Y_i = 1 \mid Y_i = 1)$ and its specificity is $\mathrm{sp} = \Pr(\hat Y_i = 0 \mid Y_i = 0)$. RAGAS faithfulness is the sample mean of the verdicts, $\hat F = \frac{1}{n}\sum_i \hat Y_i$, with observed positive rate $p_{\mathrm{obs}} = \Pr(\hat Y_i = 1)$.

An LLM judge is a measurement instrument with its own bias and variance; every RAGAS metric is an estimator built from its verdicts, and reading it correctly means correcting for the instrument — debiasing the verdict, agreeing on the protocol, calibrating the confidence, and pricing the noise floor.

The lab above is the topic in five panels. The arc: watch a biased verdict get debiased and a ranking get reversed; ask whether two judges agree once chance is removed; calibrate the judge’s stated confidence; price the irreducible judge noise; and finally recover the judge’s error rates with no answer key at all.

The judge as a noisy instrument

Faithfulness inherits everything the prerequisite said about a metric — it is a mean of per-item scores — with one change: the scores are now noisy verdicts rather than fixed labels.

Because the judge is imperfect, that mean does not estimate $\pi$; it estimates something shifted away from it.

A lenient judge — one that over-endorses, with high sensitivity but mediocre specificity — inflates faithfulness. On the corpus, such a judge reports faithfulness $p_{\mathrm{obs}} = 0.703$ for the dense leg whose true grounded fraction is $\pi = 0.648$. The fix is to invert the error model.

On the dense leg, the audited correction recovers $\hat\pi_{\mathrm{RG}} = 0.648$ — exactly the oracle prevalence — at the cost of a variance inflated $2.6\times$. Panel A drives this with sensitivity and specificity sliders: the truth $\pi$, the inflated observation $p_{\mathrm{obs}}$, and the corrected $\hat\pi_{\mathrm{RG}}$ sit on one axis, and the $1/J^2$ curve climbs without bound as the sliders approach a coin flip.

The consequence is a decision, not just a number. Two answer-generation systems judged by the same lenient judge can be ranked in the wrong order by raw faithfulness: a system whose documents the judge over-endorses (low effective specificity) is inflated above a genuinely better one. Panel A’s lower chart shows the constructed case — raw faithfulness favors system A ($0.80$ over $0.74$), but Rogan–Gladen with each system’s audited rates restores the truth, favoring system B ($0.79$ over $0.70$), matching the oracle. The instrument was masking the ranking; correcting for it flips the verdict back.

Agreement is not accuracy: chance-corrected reliability and the variance floor

Before trusting a judge against the truth, we ask a weaker question we can actually check in production: does the judge agree with a second rater — another model, or a human auditor on a sample? Raw agreement is the wrong summary, because two raters agree by chance even when they share no signal.

The trouble is that $p_e$ depends on the marginals, and in a setting where almost every claim is supported, the marginals are extreme and $p_e$ swells toward $1$.

The remedy is a chance term that does not balloon with skew.

On the same two tables, Gwet’s AC1 reads $0.70$ and $0.81$ — it does not collapse where kappa does. Panel B sweeps the prevalence at fixed agreement and overlays the two curves: kappa plunges toward zero at the extremes while AC1 stays high. The lesson is not that AC1 is “correct” — none of the three is canonical — but that kappa must never be read without its marginals.

Agreement also exposes a second, structural fact: the judge is a source of variance that more queries cannot average away. Model each per-query faithfulness as $X_{q,j} = \mu + a_q + b_j + \varepsilon_{q,j}$ with a query effect $a_q$, a judge effect $b_j$, and residual noise.

On the five-judge panel the components are $\sigma_q^2 = 0.0102$, $\sigma_j^2 = 0.0048$, $\sigma_e^2 = 0.0134$, giving $\mathrm{ICC}(2,1) = 0.36$. Panel D plots the total standard error descending toward the judge floor as queries grow, and prices the lever: at a fixed budget of $200$ judge-calls, five judges over forty queries ($\operatorname{SE} = 0.036$) beat one judge over two hundred queries ($\operatorname{SE} = 0.070$). You cannot buy confidence in a faithfulness score with queries alone — the judge noise is structural.

Judge bias, calibration, and learning the error rates

Two threads remain. First, the systematic biases the judge brings — and which of them we can remove. Second, the judge’s stated confidence: is it a probability?

On the corpus the lenient judge’s swap test returns a bias of $0.070$ at $p \approx 10^{-105}$ — a real, removable position effect — while a bias-free control judge returns bias $0$ at $p = 1.0$. The swap test detects and removes the slot effect; it cannot touch self-preference.

The judge does not only vote; it states a confidence. We feed those confidences straight into the prerequisite’s calibration suite.

The judge confidence ranks claims almost perfectly (AUC $\approx 0.998$ for the lenient judge) yet is wildly over-confident as a probability (ECE $0.154$): Panel C shows the raw dots bowed far off the diagonal. Platt scaling pulls the ECE to $0.003$ and isotonic to $0$, both leaving the AUC untouched at $0.998$ — calibration is orthogonal to ranking, the same reason rank fusion fuses ranks rather than scores. A pipeline that thresholds or averages a judge’s raw confidences inherits the miscalibration; one that ranks by them does not.

Learning the error rates with no gold set

The Rogan–Gladen correction needed the judge’s sensitivity and specificity. In production there is often no gold label to estimate them from — only the judges’ verdicts. The latent-class model recovers them anyway.

Panel E runs it: with no gold labels, EM recovers every planted per-judge sensitivity and specificity within $\approx 0.06$ and labels items more accurately than majority vote ($0.975$ versus $0.972$). This closes the loop — the error rates the instrument correction assumed are exactly what EM supplies. The result is identifiable only up to a label permutation (we anchor to the majority vote), and the likelihood is non-convex, so EM finds a local optimum from a sensible start; correlated judges, who violate the conditional-independence assumption, break the guarantee. It is a powerful tool, not a free lunch.

The thread that ran through the evaluation layer — a metric is an estimator — now reaches the generation layer with a second source of noise stacked on the first. Faithfulness and groundedness will give that support a semantic definition, and conformal factuality will turn a calibrated, debiased judge confidence into a distribution-free correctness guarantee. Both begin from the instrument this topic learned to read.