The Architecture Pareto Frontier: What "Which One Should We Use?" Actually Depends On

The question that has no answer

The predecessor to this topic ran six named RAG architectures as arms over two regimes and found that no row won both. It also did something quieter that mattered more: it reported cost in two currencies — similarity operations and generator calls — and refused to collapse them into one, on the grounds that the exchange rate between them is precisely what a cost model should not invent.

That refusal is what leaves us here, with a Pareto set rather than a winner, and the obvious next question. Which one should we deploy?

The answer is that the question, as asked, has none. It acquires one only after three further things are fixed, and none of the three is a property of any architecture:

• the workload mixture $w$ — what fraction of your traffic is each kind of question;
• the price ratio $\rho$ — what one generator call costs in units of a similarity operation;
• the amortization horizon $N$ — how many queries an offline index build is spread over.

Move any one and the winner changes. That is not a disclaimer, it is the subject. This topic is about the shape of the dependence: a partition of a simplex, a hull the price slides along, and a crossover in deployment size.

Movement 1 — cost is a pair, and it is not a constant

Write $\mathrm{ops}_{a,k}$ and $\mathrm{calls}_{a,k}$ for what arm $a$ spends on a question of kind $k$. A workload is a probability vector $w$ over the six kinds of question, and an arm’s cost on it is

$$\mathrm{cost}(a, w) = \Big( \textstyle\sum_k w_k\, \mathrm{ops}_{a,k},\ \ \sum_k w_k\, \mathrm{calls}_{a,k} \Big) \in \mathbb{R}^2 .$$

Two things about that expression are worth saying slowly.

The first is that it lands in $\mathbb{R}^2$, and $\mathbb{R}^2$ has no total order. “Cheaper” is no longer a comparison two architectures can always settle. An arm spending a hundred operations and no generation is not more or less expensive than one spending twenty-five and four generator calls; the two are simply incomparable until somebody supplies a price.

The second is that $\mathrm{ops}_{a,k}$ genuinely depends on $k$ for some arms, and this is where the familiar “cost” column of a comparison table stops being merely imprecise and becomes wrong. Adaptivity turns out to be a spectrum rather than a property, and the spread across question kinds measures where on it an arm sits. Four of these six are fixed pipelines that do the same work whatever they are asked, at exactly $1.00\times$. A fifth, the agentic arm, is nearly flat at $1.11\times$ — it stops hopping early when a filing it reads opens no new direction, so its cost is query-gated too, just barely. Only the corrective arm is adaptive enough for a single cost figure to be actively misleading rather than merely imprecise: its grader fires on some questions and not others, and its cost varies by a factor of 9.95 across the kinds of question it might meet.

So the cost of an adaptive architecture is a function of the workload it meets. There is no number to put in the column.

A · cost is a pairB · what domination depends onC · the winner mapD · what no price can buy

flat arms

5 of 6

corrective, across question kinds

9.95×

currencies

2, not 1

Operations per query, on a log color scale. Five of the six arms do the same work whatever they are asked — their cost is a property of the pipeline. The corrective arm's is not: its grader fires on some questions and not others, so its cost varies by a factor of 9.95 across the kinds of question it might meet. A single cost column cannot say that, and neither can a single cost column say that four of these arms also spend a second, incommensurable currency — generator calls — that no number of operations converts into.

Six architectures over 100 passages of 25 companies in 5 sectors, measured by the predecessor topic and re-run here not at all. Everything above is arithmetic on one 6 × 6 × 3 table of quality and both costs — 108 numbers, emitted from rag_architecture_pareto.py — and every derived object on this page, the profiles, the domination relation, the Pareto sets, the hulls, the winner map and the amortization, is recomputed live in the browser from them. The workload mixture, the price ratio and the amortization horizon are all free: none of them is a property of any architecture, and each one changes the answer.

Movement 2 — domination, and what it is a property of

On the mixture the predecessor implicitly reported at — its local regime, every class equally likely — the Pareto set is hybrid, hyde, corrective, graph. Two arms are outside it, and one of those two is outside it comprehensively. The agentic arm is beaten by the naive baseline on every axis at once: lower quality (0.280 against 0.290), more operations (288 against 100), and more generator calls (2.88 against 0). It is beaten by the graph arm as well.

A reader who stopped there would conclude, correctly by the evidence in front of them, that this architecture should never be deployed. And then:

This is the topic’s central claim and it is worth resisting the urge to soften it. The aggregate numbers were not wrong. They were computed correctly, on a mixture, and they support a conclusion that a different mixture reverses. Domination is a property of an architecture and a workload, and reporting it as a property of an architecture is a category error that a Pareto table makes very easy to commit.

Movement 3 — the winner map partitions the simplex

Fixing a price $\rho$ for generation and a price $\lambda$ for cost in units of quality gives a selection rule, the one every cost-aware evaluation computes:

$$\hat a(w) = \arg\max_a \Big\{ q_a(w) - \lambda\big(\mathrm{ops}_a(w) + \rho\, \mathrm{calls}_a(w)\big) \Big\}.$$

The map $w \mapsto \hat a(w)$ partitions the workload simplex into cells. What the partition looks like is the whole content of “which architecture should we use”, and on this arm set it is genuinely contested: at the headline price all five arms that ever win hold real territory — corrective 68%, graph 14%, hybrid 9%, agentic 6%, hyde 3% of randomly drawn workloads.

Raising $\lambda$ collapses the map onto the cheapest arm; lowering it collapses the map onto the most accurate one. Both limits are degenerate and neither is wrong, which is the uncomfortable part: a benchmark’s cost weighting decides its verdict before any measurement is taken. The weighting is usually chosen once, early, by whoever set up the harness, and is rarely reported at all.

Movement 4 — what no price can buy

There is a tempting response to all of this: sweep the prices. If the answer depends on $\lambda$ and $\rho$, vary them, and collect every architecture that wins somewhere. That set — the options a cost-weighted evaluation can ever recommend — is the honest output.

It is not, and the reason is geometry rather than retrieval.

The practical content is that a Pareto-optimal architecture can be unrecommendable by any weighting of the objectives — not because the weights were tuned badly, but because no weights exist. It is dominated by no single arm, and beaten at every price by a mixture of two others it sits between.

How often does that bite? Here, on 22.4% of randomly drawn workloads, and the arm that goes missing most often is agentic — the same arm the aggregate numbers already dominated. Its second and third most frequent companions are hyde and corrective, which is to say: the arms that spend the second currency.

That convergence is the topic’s sharpest point. Both of the standard ways of picking a winner — compute aggregate domination, or sweep a weighted objective — fail on the same architecture, and for the same underlying reason. It buys a capability that is worthless on most traffic and indispensable on a little of it, and neither an average nor a linear price can represent that.

Movement 5 — and the frontier moves under you anyway

One free parameter remains, and it is the one most often left out of a cost model entirely.

The graph arm holds an index nobody else builds: the entity–entity similarity matrix its community detection runs on, which is $K(K-1)/2$ similarity evaluations. Charged against a deployment of $N$ queries, its cost per query is $\mathrm{ops}_{\text{online}} + B/N$. Below the crossover the build has not paid for itself, and the domination reverses — at this corpus size the crossover at N* = 3.33 queries.

That number is small, and the honest reading is not that amortization does not matter but that it does not matter at this scale. The scaling is the result worth carrying:

entities $K$	25	100	500	2000
crossover $N^*$	3	13	64	253

The build is quadratic in the entity count while the saving is only linear, so the deployment must be bigger before the index pays for itself as the corpus grows — the opposite of the intuition that an index matters more the more there is to index.

What this leaves you with

Three free parameters, none of them a property of any architecture, and a decision that each of them moves. That is a worse situation than a ranking and a more accurate one, and it has a practical residue:

• A cost column with one number per architecture is wrong for any adaptive architecture, and wrong by a factor of ten for one of these.
• A Pareto verdict is void the moment the traffic mixture changes, which it does continuously in any deployed system.
• Sweeping the weights does not fix it: a quarter of workloads here contain a Pareto-optimal architecture that no weighting can return.
• And whether the arm with an index beats the arm without one is a question about how many queries you expect to serve.

The useful output of an architecture comparison is therefore not a name. It is the partition — which regions of workload space each architecture owns, and how far your traffic sits from a boundary.