Set Metrics: Precision, Recall, MAP, and MRR as Estimators

Overview & motivation

The retrieval problem ended with a ranking: a query $q$ , a score $\mathrm{rel}(q, d)$ on every document $d$ , and the top- $k$ returned as a set. Everything the curriculum built afterward — BM25, dense dual encoders, IVF and product quantization, late interaction served by PLAID, reciprocal-rank fusion — was judged by a single recurring number, recall@k. The capstone reported it a dozen times. Not once did we say what it means.

That is the gap this topic closes. An evaluation metric is a functional that takes a ranking and a set of relevance judgments and returns a number. Write the judgments as a qrel set $R_q \subseteq \mathcal{D}$ : the documents a competent analyst would mark relevant to query $q$ . Because we want every number on this page to be a measurement rather than an illustration, we do not invent rankings — we score three real retrievers over a shared finance corpus. The lexical leg is BM25, the dense leg is a pooled dual encoder, and the strongest leg is late interaction, each producing a full ranking of $N = 120$ documents for each of $Q = 40$ queries. The relevance judgments are the top- $k$ documents under exact MaxSim, a neutral oracle: no single leg is the ground truth, so the three genuinely differ in quality, and “relevance” is a fixed target they all approximate.

rank cutoff k = 5

relevant in top-5

5 / 10

precision@k = TP/k

1.000

recall@k = TP/|R|

0.500

F1@k

0.667

Slide k right: recall never falls (each new relevant doc can only add coverage), while precision wobbles — it jumps up when slot k is relevant and decays when it is not. Recall reaches 1.0 once all 10 relevant docs are passed; precision at that point is only 10/(last relevant rank).

120 synthetic finance documents, 40 queries; relevance is the top-10 exact-MaxSim neighbor set (a neutral oracle). Numbers mirror set_metrics_precision_recall_map_mrr.py; the lab recomputes precision, recall, the PR/AP area, the mean, and the CI in closed form.

The lab above is the topic in miniature, and we will return to each of its four panels in turn. The arc is short: define the metrics on one ranking, integrate them into a curve, average them across queries — and then admit that every resulting number is a noisy estimate.

Set metrics at a cutoff: precision and recall

Fix a query and a cutoff $k$ . Of the top $k$ documents the system returned, write $\mathrm{TP}_k = |\,\mathrm{top}_k(q) \cap R_q\,|$ for the number that are relevant (true positives). The two foundational metrics divide this count two ways.

Definition 1 (Precision and recall at a cutoff).

For a ranking with qrel set $R$ (with $|R| > 0$ ) and a cutoff $k \ge 1$ ,

P@k \;=\; \frac{|\,\mathrm{top}_k \cap R\,|}{k}, \qquad R@k \;=\; \frac{|\,\mathrm{top}_k \cap R\,|}{|R|}.

Precision is the purity of the cutoff — what fraction of what you returned is relevant — and recall is the coverage of the relevant set — what fraction of what you should have returned you did. Their harmonic mean is the $F_1$ score, $F_1@k = 2\,P@k\,R@k / (P@k + R@k)$ .

The denominator of recall is $|R|$ , the size of the relevant set. This is the textbook convention, and it is worth a word because it differs from the one the capstone used: there, recall divided by $\min(k, |R|)$ , a cascade-retention reading (“of the $k$ slots, how many true neighbors survived”). The two agree exactly at $k = |R|$ , the capstone’s only operating point, so this topic defines precisely the recall@10 the capstone reported; the $|R|$ form is what generalizes the curve, because it makes the next proposition true.

Proposition 1 (Recall climbs, precision wobbles).

As the cutoff $k$ increases, $R@k$ is non-decreasing, and $R@N = 1$ . Precision $P@k$ is not monotone.

Proof. The set $\mathrm{top}_{k+1}$ contains $\mathrm{top}_k$ , so $\mathrm{TP}_k$ is non-decreasing in $k$ ; dividing by the constant $|R|$ preserves this. At $k = N$ every document is returned, so $\mathrm{TP}_N = |R|$ and $R@N = 1$ . For precision, $\mathrm{TP}_k$ rises by one exactly when document $k{+}1$ is relevant, while the denominator $k$ rises by one every step; so $P@(k{+}1)$ exceeds $P@k$ when the new document is relevant and falls when it is not. $\square$

Panel A makes the asymmetry tactile. Drag the cutoff across the worked query’s ranking: each slot is a true positive (relevant and retrieved), a false positive (retrieved, not relevant), or — beyond the cutoff — a false negative (relevant, missed). Recall only ever rises as the line moves right; precision jumps up on a relevant slot and decays on an irrelevant one. A single $(P@k, R@k)$ pair is a point on a plane; the whole topic is what happens when we let $k$ run.

The precision–recall curve and Average Precision

Sweeping $k$ from $1$ to $N$ traces a path in the precision–recall plane. Because recall only changes when a relevant document is hit, the informative points are exactly those hits. Let the relevant documents sit at ranks $\mathrm{pos}_1 < \mathrm{pos}_2 < \dots < \mathrm{pos}_m$ (with $m = |R|$ ); at the $i$ -th hit, recall is $R_i = i/m$ and precision is $P_i = i/\mathrm{pos}_i$ . Connecting these corners gives the familiar sawtooth — and the single most useful summary of a ranking is the area beneath it.

Theorem 1 (Average Precision is the area under the PR curve).

Define Average Precision as the mean of precision at the relevant ranks,

\mathrm{AP} \;=\; \frac{1}{|R|}\sum_{i=1}^{m} P_i \;=\; \frac{1}{|R|}\sum_{i=1}^{m} \frac{i}{\mathrm{pos}_i}.

Then $\mathrm{AP}$ equals the area under the (raw) precision–recall curve,

\mathrm{AP} \;=\; \sum_{i=1}^{m} (R_i - R_{i-1})\,P_i, \qquad R_0 = 0.

Proof. At rank $\mathrm{pos}_i$ exactly $i$ relevant documents have been retrieved, so recall steps from $(i{-}1)/m$ to $i/m$ — an increment $R_i - R_{i-1} = 1/m$ — and precision there is $P_i$ . Between consecutive relevant ranks recall is constant, so those stretches add nothing to a recall-integrated sum. Hence $\sum_i (R_i - R_{i-1}) P_i = \sum_i \tfrac{1}{m} P_i = \tfrac{1}{m}\sum_i P_i = \mathrm{AP}$ . $\square$

This is a Riemann sum over the recall axis: the relevant documents partition $[0,1]$ into $m$ equal increments of width $1/m$ , and AP is the sum of precision times those widths — the area-as-a-limit-of-sums construction applied to a step function. The proof also explains the denominator $|R|$ rather than “number of documents retrieved”: a relevant document that never surfaces contributes a missing $1/m$ recall step, i.e. a term of zero, which correctly drags AP down. Dividing by the count actually found would give a ranking that retrieves one of ten relevant documents (at rank 1) an AP of $1.0$ — exactly wrong.

Panel B draws this. The shaded bars are the recall steps; their total area is AP. Toggle the interpolated envelope and a second, larger area appears.

⚠ Rigor flag

“Average Precision” is ambiguous until you name a convention. The raw AP above — the area under the sawtooth, equal to the mean of precision at the relevant ranks — is the form MAP averages and the one we report. The interpolated AP replaces each precision with the best precision attainable at any equal-or-higher recall, $p_{\text{interp}}(r) = \max\{P_j : R_j \ge r\}$ , a monotone non-increasing envelope. Its area is always at least the raw AP, because the envelope dominates the raw precision pointwise — interpolation inflates the number, it does not correct it. The classic eleven-point interpolated average precision (TREC) samples that envelope at recall $0, 0.1, \dots, 1$ . None of these is “the” AP; they are conventions, and a reported AP that does not state which one was used is underspecified.

Mean Average Precision

A single query’s AP is a property of one ranking. To score a system we average over a query set.

Definition 2 (Mean Average Precision).

For a set of $Q$ queries, $\displaystyle \mathrm{MAP} = \frac{1}{Q}\sum_{q=1}^{Q} \mathrm{AP}_q$ , the sample mean of per-query Average Precision.

Over our three legs the MAP values are a clean quality ladder: lexical $0.619$ , dense $0.735$ , late interaction $0.874$ . Panel C shows why an average is the right summary — the per-query AP scatter widely (the late-interaction leg ranges from below $0.6$ to a perfect $1.0$ ), and only their mean is stable. That scatter is not noise to be ignored; it is the metric’s variance, and we return to it.

Reciprocal rank and known-item retrieval

Some queries have one right answer — the filing that announced a merger, the slide that stated a guidance number. There, what matters is not coverage of a set but how high the one answer ranks.

Definition 3 (Reciprocal rank and MRR).

The reciprocal rank of a query is $\mathrm{RR} = 1/\mathrm{rank}_1$ , where $\mathrm{rank}_1$ is the rank of the first relevant document (and $\mathrm{RR} = 0$ if none appears). The Mean Reciprocal Rank is $\displaystyle \mathrm{MRR} = \frac{1}{Q}\sum_{q=1}^{Q}\mathrm{RR}_q$ .

MRR is built for the known-item regime, where each query has exactly one relevant document — and there it is not a different animal from AP at all.

Proposition 2 (MAP equals MRR in the known-item regime).

If every query has exactly one relevant document ( $|R_q| = 1$ for all $q$ ), then $\mathrm{AP}_q = \mathrm{RR}_q$ for every query, and hence $\mathrm{MAP} = \mathrm{MRR}$ .

Proof. With $m = 1$ , the single relevant document sits at some rank $r_q$ . Average Precision has one term: $\mathrm{AP}_q = \tfrac{1}{1}\cdot\tfrac{1}{r_q} = 1/r_q$ . The first (and only) relevant document is at rank $r_q$ , so $\mathrm{RR}_q = 1/r_q$ too. The per-query metrics coincide, and so do their means. $\square$

This is the topic’s collapse anchor: at $|R| = 1$ the whole apparatus of precision, recall, and the PR curve degenerates to a single reciprocal rank. Panel C makes it visible — switch to the known-item regime and the per-query AP strip lands exactly on the reciprocal-rank strip, mean line and all. Because MRR is degenerate when relevant sets are large (with ten relevant documents some leg almost always lands one near the top, so reciprocal rank says little), we score it in the regime it was built for: relevance is the single top-1 oracle document.

Holding both metrics in view exposes something the quality ladder hid. The MAP ranking is lexical $<$ dense $<$ late interaction; the MRR ranking is dense $<$ lexical $<$ late interaction.

Proposition 3 (Metric choice can reverse the verdict (demonstrated)).

On this corpus, dense beats lexical on MAP ( $0.735$ vs $0.619$ ) but lexical beats dense on MRR ( $0.522$ vs $0.472$ ). The verdict between two systems reverses depending on the metric.

The mechanism is structural: MAP rewards ranking the whole relevant set high, and the dense leg does that better on average; MRR rewards getting one relevant document to the very top, and the lexical leg’s exact-match behavior more often plants a relevant document at rank 1. This is a demonstrated proposition on one corpus, not a theorem — but it is built and run before it is stated, and it makes the lesson concrete: “system A is better” is meaningless without naming the metric.

Metrics as estimators

Here is the move the rest of the evaluation layer rests on. A MAP of $0.735$ is not a property of the dense retriever; it is the sample mean of forty per-query AP values, and a different forty queries would give a different number. Every metric on this page is an estimator of an unknown population quantity, and estimators have variance.

Definition 4 (MAP as an estimator with standard error).

Treat the per-query Average Precisions $\mathrm{AP}_1, \dots, \mathrm{AP}_Q$ as a sample with mean $\widehat{\mathrm{MAP}}$ and sample standard deviation $\hat\sigma$ . The standard error of the MAP estimate is

\widehat{\mathrm{SE}} = \frac{\hat\sigma}{\sqrt{Q}}, \qquad \text{95\% CI} = \widehat{\mathrm{MAP}} \pm 1.96\,\widehat{\mathrm{SE}},

and the standard error shrinks like $1/\sqrt{Q}$ : quadrupling the query count halves the interval.

That $1/\sqrt{n}$ rate is not a heuristic; it is a concentration inequality. The per-query AP are bounded in $[0,1]$ , so Chebyshev’s inequality already gives a confidence interval from the variance alone, and the sub-Gaussian (Hoeffding) bound sharpens it — the sample mean concentrates around its expectation at a rate controlled by the query count. The companion notebook verifies the rate empirically: resampling the per-query AP at growing $n$ , the measured standard error tracks $\hat\sigma/\sqrt{n}$ , and $\widehat{\mathrm{SE}}\cdot\sqrt{n}$ holds essentially constant.

Panel D is the payoff. It plots two legs — dense and late interaction — with their 95% intervals, and a slider for the query count $n$ . The gap in MAP is real ( $0.139$ ), but at few queries the intervals overlap: you cannot tell the systems apart. They separate only at $n = 12$ . Below that, the honest report is “indistinguishable on this query set,” not “late interaction wins.”

§ Choosing a metric for a filings-and-transcripts retriever text audio

The production system this series tracks retrieves over 10-K and 10-Q filing text and earnings-call transcripts. Suppose an analyst tool needs the one passage stating next quarter’s guidance — a known-item task — and we are choosing between the dense and lexical legs. By MAP, dense is the clear winner ( $0.735$ vs $0.619$ ); by MRR, the metric that actually matches the task, lexical wins ( $0.522$ vs $0.472$ ). Picking the system on the wrong metric ships the worse retriever for the job. And even the right comparison is provisional: with only a few dozen evaluation queries, a $0.139$ MAP gap between two systems can sit inside the sampling error, so the disciplined report is a confidence interval, not a bare number — and a decision to switch retrievers waits on a significance test, not a leaderboard delta. The corpus here is a synthetic vMF token cloud standing in for those documents; every value illustrates the mechanism, not a benchmark.