The Workload You Do Not Know: Choosing an Architecture from an Estimate

A lookup table with a missing input

The previous topic ended with something that looks like an answer. Fix the price of a generator call and the size of the deployment, locate your workload mixture $w$ on the simplex, and read off the architecture that the cell you landed in recommends. The map $w \mapsto \hat a(w)$ partitions the simplex; you look yourself up.

Except that you do not know $w$ . You estimate it — from a sample of traffic, from a week of logs, from an intuition about what users will ask — and the estimate $\hat w$ carries error like any other. The obvious question is how much that error matters.

The obvious answer would be “a little, and less as you collect more data”. It is the right answer in the interior of a cell and badly wrong near a boundary, for a structural reason:

Remark (A partition is the worst possible consumer of an estimate).

The map from workload to architecture is piecewise constant. A piecewise constant function is maximally insensitive to its input almost everywhere and maximally sensitive on a measure-zero set, and there is no in-between. So estimation error is free in the interior of a cell — the recommendation does not move at all, however wrong you were — and at a boundary an arbitrarily small error flips it outright.

Neither half of that is what “the estimate has error” usually implies, and the second half is why a Pareto verdict cannot simply be annotated with an error bar.

Three rules, and one of them is not a rule

Write $\pi$ for a posterior over $w$ given $n$ observed queries: with a Dirichlet prior it is $\mathrm{Dir}(n\hat w + \alpha)$ , and $n$ is what concentrates it. Three ways to choose an architecture from it:

\hat a_{\text{plug}} = \arg\max_a U(a, \hat w), \qquad \hat a_{\text{Bayes}} = \arg\max_a \mathbb{E}_{\pi}[U(a, w)], \qquad \hat a_{\text{mm}} = \arg\min_a \max_{w} \big[ U(a^\ast(w), w) - U(a, w) \big].

The first pretends the estimate is the truth. The second averages over what it does not know. The third minimizes the worst regret it could come to feel — the amount by which the arm it picked trails whichever arm turns out to have been right.

That looks like three genuinely different ideas. It is two:

Proposition 1 (The Bayes rule is the plug-in rule at a shrunk estimate).

Utility is affine in the workload — each arm’s quality and both its costs are averages over question kinds, weighted by $w$ . Therefore, for any posterior $\pi$ whatsoever,

\mathbb{E}_{\pi}\big[ U(a, w) \big] = U\big(a, \mathbb{E}_{\pi}[w] \big),

with no approximation. The Bayes rule is the plug-in rule evaluated at the posterior mean, and for a Dirichlet posterior that mean is available in closed form:

\mathbb{E}_\pi[w] = \frac{n \hat w + \alpha}{n + C\alpha},

which is $\hat w$ shrunk toward the prior with weight $C\alpha/(n + C\alpha)$ .

So the entire difference between plug-in and Bayes is a shrinkage that vanishes like $1/n$ . Both are point rules: they collapse the posterior to a single workload and look it up. Only the minimax rule uses the shape of the posterior rather than its location, and it is the only one of the three with no closed form.

That asymmetry is worth holding onto, because it reappears as an engineering fact. In the laboratory below, the plug-in and Bayes rules are recomputed live and exactly in the browser, while the hedge is read from a precomputed grid — not an implementation compromise, but Proposition 1 showing up in the software.

share of traffic that is a bridge question = 45.0%queries observed (n) = 40

plug-in

agentic

Bayes

corrective

minimax (the hedge)

corrective

verdict

the hedge differs

Plug-in and Bayes are recomputed here exactly, live, from the utility kernel — because utility is linear in the workload, both have closed forms. The hedge does not, so it is the one rule read from a precomputed grid. That asymmetry is the topic's own result showing up in its laboratory. Slide across the boundary and the three bands part company only in a narrow region; slide n and the disagreement shrinks everywhere except on the boundary itself.

The six architectures and their quality and costs arrive unchanged from the two predecessor topics; nothing is re-measured here. Prices are fixed at λ = 0.00003, ρ = 1000. The posterior is Dirichlet with a uniform prior (α = 1) over the 6 kinds of question, and worst-case regret is its 0.95 quantile over 2000 draws rather than a maximum, which would grow with the draw count and could not be baked. Plug-in, Bayes and every utility on this page are recomputed live and exactly from the utility kernel; only the hedge and the realized regret are Monte-Carlo and therefore baked, and both are seeded.

The hedge is not a safe default

A natural suspicion about minimax rules is that they retreat to something bland — the option that is never terrible, chosen forever. Measured, that is not what happens here. Deep inside a cell the minimax arm is that cell’s arm: on bridge-heavy traffic it is the iterative architecture, on global-heavy traffic the graph one, and those are different arms. The hedge tracks where the posterior sits.

It departs from the plug-in rule in a narrow band around a boundary, and over workloads drawn without regard to where the boundaries are — which is the situation a deployment is actually in — the two rules differ on 7.67% of them.

How much traffic is enough

The natural question, once the estimate is admitted to be an estimate, is how much traffic settles it. The measurement is a sharp two-regime answer.

Inside a cell, almost none: the rules agree from n = 10 when the truth sits inside a cell, because the recommendation is simply not sensitive to where in the cell you are. On a boundary the same question is still paying at n = 5120 — about five hundred times as much traffic for a decision between two arms that are, at that point, nearly tied in value.

The worst-case regret itself decays monotonically with $n$ , and it is tempting to write down the $1/\sqrt n$ law that the multinomial rate would suggest. The halving ratios measured here are $0.98, 0.84, 0.88, 0.74, 0.77, 0.75, 0.72, 0.72, 0.69$ against the $0.707$ such a law implies: they approach it from above rather than sitting on it. What the topic asserts is the monotone decay and the tail, not the rate.

Remark (Why a maximum could not be used).

Worst-case regret is reported as the $0.95$ quantile of the regret distribution over 2000 draws, not as a maximum over them. A maximum is an extreme statistic whose value grows with the number of draws — run the simulation longer and the “worst case” gets worse, without anything about the problem having changed. Nothing computed that way can be baked into a figure or compared across runs. A quantile estimates a fixed property of the distribution instead, and a test pins its stability across seeds before any number here is quoted.

What the hedge costs

Hedging is usually discussed as though its only cost were a little foregone upside. Here the accounting is measurable, and it is less comfortable than that.

Both the Bayes rule and the hedge shrink toward the middle of the simplex. That is not a neutral act — it is a bet that the traffic is more central than the data says. Below the boundary the bet pays handsomely: the plug-in rule’s realized regret is several times the hedge’s, because the plug-in rule keeps being fooled by samples that stray across the boundary. Above it, where the truth genuinely sits in a narrow cell, the plug-in rule was correctly confident all along and the hedge drags it back out of the cell it was right about.

And the loss on that side is larger than the gain on the other. A hedge here is a trade, not insurance bought at a discount, and which way it goes depends on something you cannot observe: whether your true workload is nearer the middle of the simplex than your estimate suggests.

§ A week of logs, and a decision that does not wait text pdf news

The practical version: you are choosing one retrieval stack for a filings assistant, and what you have is a week of queries — a few hundred, unevenly distributed, from an early cohort that is not quite the eventual user base.

If that sample puts you clearly inside one cell, the decision is easy and no amount of extra data would change it; collect no more, and commit. If it puts you near a boundary, the uncomfortable finding is that a week is not the problem and a year would not obviously fix it — the traffic would have to grow by two or three orders of magnitude before the estimate separated the two candidate architectures.

The useful move there is not more data. It is to notice that being near a boundary means the two architectures are nearly tied in value, and to choose on something the model never saw: which one is easier to operate, which one degrades more gracefully if the mixture drifts, which one your team can debug at two in the morning.

⚠ Rigor flag

One result here is an identity and survives any corpus: utility is affine in the workload, so expected utility over any posterior equals utility at the posterior mean — exactly. That is what demotes the Bayes rule to the plug-in rule at a shrunk estimate, and it is also what makes the simulation affordable, since it collapses two thousand posterior draws into one matrix product.

Everything else is a measurement on a synthetic corpus inherited unchanged from two predecessors, and the decimals describe this arm set and no other. Three modeling choices change the numbers and are stated rather than buried: the prior is Dirichlet with $\alpha = 1$ , which is what defines the “middle of the simplex” the hedge shrinks toward; worst-case regret is a quantile rather than a maximum, for the reason given above; and the price and deployment size are pinned at the predecessor’s headline values, so every boundary here is a boundary at those prices.

The inherited utility table carries a sampling error of roughly 0.11 per coordinate, from twenty queries per class. Boundary locations are therefore soft, and the topic asserts that a boundary exists and that behavior differs sharply across it — never where it sits to three decimals.