Score Calibration, Drift Detection, and Significance Testing for Retrieval

The question the variance only opened

Set metrics and NDCG made the same move and stopped at the same place. A retrieval metric is not a property of a system; it is the sample mean of a per-query score over a finite, fixed query set, and a different set of queries would give a different number. Both topics drew the 95% interval $\overline{m} \pm 1.96\,\widehat{\mathrm{SE}}$ around each leg’s mean and asked whether two legs could be told apart — and both ended on a cliffhanger. The closest pair of legs, lexical and dense, differs by only $0.05$ in mean NDCG, their intervals overlap across all forty queries, and the prerequisite estimated that gap would need roughly 185 queries to resolve.

That estimate is a crude read, and conservative in a specific way: it compares two intervals built independently, as if the two legs had been measured on different queries. They were not. Both legs ran on the same forty queries, and an easy query — one with an unambiguous answer near the top — is easy for every leg. The overlapping-intervals test throws that pairing away. The right instrument keeps it, and this topic is built around the single idea that keeping it unlocks.

A metric is an estimator; every question that remains is whether two distributions of per-query quantities differ. System versus system is the significance question. The score against the truth is the calibration question. Now against then is the drift question. Each is a paired comparison, and pairing — differencing query-by-query — is the geometric fact that runs through all three.

A · paired testB · distribution-freeC · calibrationD · drift detection

MAPNDCG

query count n = 40

paired t · p

-1.65 · 0.107

paired separates at n

57

unpaired needs n

185

80%-power n

116

var(d)/(varA+varB)

0.59

Pairing cancels the shared per-query difficulty (the legs correlate 0.45 across queries), so var(d) is only 0.59× the unpaired variance and the paired interval excludes 0 at n = 57 where the unpaired two-CI read needs 185. This closes the NDCG cliffhanger: 185 → 116 (rigorous 80%-power) versus 57 (a single interval clearing 0); at all 40 queries the gap is not yet significant (p = 0.107).

120 synthetic finance documents, 40 queries, top-10 relevant; the three legs and the MaxSim oracle are the prerequisites' shared corpus. Numbers mirror significance_testing_calibration.py; the lab recomputes the CI half-widths, ECE/MCE, histograms, CDFs, and the KS gap in closed form, and bakes the corpus-derived means, p-values, reliability bins, and PSI.

The lab above is the topic in four panels; we take each in turn. The arc: build the paired test and watch it resolve the cliffhanger, make it distribution-free and correct it for multiple comparisons, then turn the same comparison sideways to ask whether a score is calibrated and whether the distribution has drifted.

The paired test: pairing cancels difficulty

Fix two legs $A$ and $B$ and let $m_A(q), m_B(q)$ be their per-query metric (AP or NDCG) on query $q$. The prerequisite compared the two means. Instead, form the per-query difference and treat it as the sample.

The whole advantage is in the variance of $d$. Because $A$ and $B$ are evaluated on the same queries, their per-query scores are positively correlated — a hard query drags both down — and the variance of the difference subtracts that shared component.

On this corpus the legs correlate $0.32$ (MAP) and $0.45$ (NDCG) across queries, so the paired variance is only $0.69$ and $0.59$ of the unpaired sum. Panel A shows the per-query difference strip $d_q$ — mostly negative, lexical below dense, with a few queries where lexical wins — and the two intervals on $\overline{d}$ shrinking under the query-count slider. The payoff is the resolution of the cliffhanger.

For the closest NDCG pair, $d_z = 0.26$, and the three query counts form a clean ladder. The prerequisite’s overlapping-intervals read asked for 185. The rigorous 80%-power count is 116 — fewer, because the paired test is the sharper instrument. The single-realization “interval clears zero” count is just 57, but that is a one-time event with about 50% power, which is why it sits below the power-based number rather than above it. The honest report is 185 → 116: the proper required query count is well under the crude one.

A second, equally honest fact: pairing tightens the test, it does not manufacture significance. On MAP the same lexical–dense gap is significant at the forty queries we have ($t = -3.87$, $p = 4.0\times10^{-4}$, separating by $n = 11$). On NDCG it is not yet ($t = -1.65$, $p = 0.107$): the gap is real-looking but inside sampling error at $Q = 40$. The paired test answers “is the 0.05 NDCG gap real?” with not on this evidence — which is the correct answer, not a louder one.

Distribution-free tests and the cost of asking three questions

The paired t-test assumes the differences are approximately normal. They are not: per-query AP and NDCG are bounded in $[0,1]$ and skewed. Two instruments make no such assumption.

On the worked MAP pair all three agree — $t$-test $p = 4.0\times10^{-4}$, permutation $p = 7.0\times10^{-4}$, bootstrap $p = 1.0\times10^{-4}$ — which is the cross-check the prerequisites’ rigor flag promised. The permutation test is the one the IR literature settled on (Smucker, Allan & Carterette, 2007); Panel B draws its sign-flip null with the observed statistic marked and the tail shaded.

Comparing three legs is not one test but $\binom{3}{2} = 3$, and running three tests at $\alpha = 0.05$ inflates the chance of a false “significant” somewhere. The multiple-comparisons corrections adjust for it: Bonferroni multiplies each p-value by the number of tests, Holm does so in a uniformly more powerful step-down, and Benjamini–Hochberg controls the false-discovery rate instead of the family-wise error. On the NDCG family the two genuine separations survive every correction, while the marginal lexical–dense pair is pushed from $0.107$ to $0.32$ under Bonferroni — cemented as not distinguishable. Correcting for the questions you asked is what keeps a three-way leaderboard honest.

Calibration: is a score a probability?

Turn the comparison sideways. So far the two distributions were two systems’ metrics; now they are a score and the truth it predicts. A retrieval score $s(q,d)$ is calibrated as a relevance probability if, among all query–document pairs scoring near $s$, a fraction near $s$ are actually relevant. The geometric object is the reliability diagram.

Calibration is forced to be a pooled object — with ten relevant documents per query, a per-query diagram would have one document per bin — and that is the point: pooling exposes whether a score means the same thing across queries. It does not. The smooth cosine (dense) and MaxSim (late-interaction) scores are massively over-confident: across the $4{,}800$ pooled pairs, with a base relevance rate of $0.083$, the raw ECE is $0.43$ and $0.39$. A cosine of $0.9$ does not mean a $90\%$ chance of relevance; it means whatever the query’s own scale makes it mean, and the concentration of cosine similarities in high dimensions is why a fixed threshold is incomparable across queries. Lexical BM25 is the interesting exception: its huge mass of exact-zero scores sits at confidence near zero and is well-calibrated in bulk (ECE $0.05$), but its tail is badly off (MCE $0.19$).

This is precisely the property rank fusion routed around. Reciprocal-rank fusion combines $1/(c + \mathrm{rank})$ — it uses only ranks, never scores — because raw scores are incommensurable across queries and legs; fusing them directly would weigh one query’s confidence against another’s on a scale that does not transfer. Rank fusion sidesteps calibration by construction; this section is the reason it had to.

The fix is to learn the map from score to probability. Platt scaling fits a logistic $\sigma(a\,s + b)$; isotonic regression fits the monotone step function minimizing squared error. Both crush the ECE — dense from $0.43$ to under $0.01$ — and both do something that looks surprising until you see why.

The pooled AUC is identical to six decimals before and after Platt scaling on every leg — the badge in Panel C — while the ECE collapses. Calibration is orthogonal to ranking: you can fix what the scores mean as probabilities without moving a single document, because calibration reshapes the score axis and the metrics read only the order. The Brier score — the mean squared error of the probability against the label — decomposes (Murphy) into reliability, resolution, and uncertainty, and recalibration drains the reliability term ($0.0002$ for recalibrated dense) while sparing the resolution that the ranking depends on.

Drift: has the distribution moved?

The last turn of the comparison is over time. In production the data shifts: the corpus grows, the query mix changes, an embedding model is swapped, and retrieval quality decays silently. Drift detection is the two-sample test of a reference window against a current one.

The middle equality is exact — $\sum_b(p_b-q_b)\ln(p_b/q_b) = \sum_b p_b\ln(p_b/q_b) - \sum_b q_b\ln(p_b/q_b)$ — so PSI is a symmetrized KL divergence, and the continuous divergence it approximates is the integral of the log Radon–Nikodym derivative, $\int \log(dP/dQ)\,dP$; the binned PSI is its Darboux sum, exactly as Average Precision was the area under a curve. Panel D drives an embedding-degradation knob: as the dense leg’s vectors are perturbed, the per-query NDCG distribution shifts left, the KS sup-gap grows from $0$ to $0.93$, and PSI crosses into the red. Under the matched-n null (the same distribution resampled) the detector is silent — mean PSI $0.06$, KS $p = 0.92$.

Two subtler facts are the real lessons. First, a drift the aggregate mean cannot see is still catchable — and the catcher is, once again, the paired test.

At a degradation that moves the mean by under $2$ standard errors — overlapping intervals, a dashboard staying green, and even the marginal KS test missing it ($p = 0.17$) — the paired test fires at $p = 0.002$. The same pairing that resolved the significance cliffhanger is the right drift monitor.

Second, input drift and quality decay are different things. A pure covariate shift — the query mix moves from easy to hard, the retriever unchanged — fires the input-distribution alarm (the top-1 score PSI is $0.43$, observable without any labels) while the model’s quality on a fixed query set is exactly unchanged (the paired outcome difference is zero). A genuine model decay, by contrast, drops the paired outcome ($-0.60$, $p \approx 10^{-21}$). So an input-drift alarm cannot diagnose decay on its own; you also need a labeled, paired outcome test. Monitor both, and read them together: input moved with outcome flat is data drift; outcome moved is decay.