When to Switch: Architecture Choice for a Workload That Moves

Three topics have now chosen a retrieval architecture, and each removed one assumption from the one before it. The first measured what six named patterns actually buy and where each one fails. The second showed that “which should we use?” resolves only after a workload mixture, a price ratio and an amortization horizon are fixed, and that no architecture wins everywhere. The third unfixed the mixture: you do not know your workload, you estimate it, and a partition of the simplex is precisely the structure that turns a small estimation error into a different recommendation.

One assumption survived all three. Each treated the workload as static — unknown, perhaps, but not moving. It moves. Traffic to a financial retrieval system in October is not the traffic it saw in June; a new filing season, a new asset class, a change in who is asking, and the mixture that the architecture was chosen for has drifted out from under it.

Once the workload moves, a second thing becomes true that was invisible before: changing architecture costs something. Reindexing a corpus under a different scheme, redeploying, and re-tuning the thresholds are real work, paid once per change rather than once per query. So the question stops being which architecture and becomes when to change to it.

That is a sequential decision problem, and its answer has a shape: a control band. Hold what you have until a rival beats it by more than a margin. What this topic is for is the measurement of what that margin is actually doing, which turns out not to be what the setup suggests.

The estimate is now a stream

The predecessor’s setup carries over intact. A deployment sees traffic made of six kinds of question, and its workload is the mixture $w$ over those kinds. Each architecture $a$ has a utility $U(a, w)$ that nets quality against both costs at fixed prices, and — the fact the whole arc leans on — that utility is affine in the workload:

$$U(a, w) = (M w)_a + c_a .$$

The predecessor observed traffic once and decided once. Here the deployment lives through $T = 60$ windows. In each window $t$ it observes $40$ queries, forms the empirical mixture $\hat w_t$, and has to decide whether to move. Writing $\hat U_t(a) = U(a, \hat w_t)$ for what the operator can see and $U(a, w_t)$ for the truth they are scored against, the policy is:

$$a_t = \begin{cases} \arg\max_a \hat U_t(a) & \text{if } \max_a \hat U_t(a) - \hat U_t(a_{t-1}) > \delta \\ a_{t-1} & \text{otherwise} \end{cases}$$

with $a_1 = \arg\max_a \hat U_1(a)$. The band $\delta$ is the whole policy. Setting $\delta = 0$ recovers the predecessor’s plug-in rule applied afresh at every window — switch whenever the estimated best architecture changes — and the module checks that identity as a sequence of arms, not merely as a count.

Regret is scored against the true workload at each window, never against the estimate, so a policy is charged for its mistakes rather than for its beliefs:

$$R(\delta) = \frac{1}{T} \sum_{t=1}^{T} \Big[ \max_a U(a, w_t) - U(a_t^\delta, w_t) \Big] .$$

The workload we send through this is a designed input, and worth naming as such: the bridge share — the fraction of questions whose answer requires following a mention from one filing to another — drifts linearly from $0.20$ to $0.70$, crossing one boundary of the predecessor’s partition. A designed path makes the phenomenon legible, and it also makes it fair to ask whether the conclusions are artifacts of a straight line, which is why a path that crosses, retreats and crosses back is run alongside it throughout.

A rule that re-decides every window changes its mind twelve times

Along that drifting path, the architecture that is genuinely best changes once. A rule that re-decides from each window’s traffic switches 12.3 times, averaged over ten deployments.

That gap is the topic. Eleven of those twelve changes are not responses to the workload, because the workload only did something once. They are responses to $40$ queries being a small sample.

The cleanest way to see that this is the estimator and not the workload is to remove the workload entirely. Hold the mixture perfectly constant — zero drift, for all sixty windows — and count switches anyway:

bridge share	real changes	myopic switches
0.20	0	1.8
0.35	0	19.3
0.43	0	30.8
0.45	0	29.5
0.55	0	10.3
0.70	0	1.0

Deep inside a cell, at $0.20$ or $0.70$, the deployment makes its first choice and stops. Near $0.43$ it changes architecture on roughly half the windows of its life, in response to a workload that never changed at all. This is the predecessor’s result arriving on the time axis: distance to a cell boundary decided how much an estimation error cost there, and it decides how much a deployment chatters here. Nothing new is needed to explain it, which is the point.

The band, and why it is not a trade

The fix is to make the incumbent harder to displace. The classical result behind this is familiar from inventory control: when changing state costs a fixed amount, the optimal policy is not a threshold but a band, and it is optimal to do nothing inside it. We take that as the frame and demonstrate the band on this instance rather than deriving its optimality in general.

Sweeping $\delta$ gives:

band $\delta$	switches	regret
0	12.3	0.00961
0.02	10.6	0.00901
0.05	7.8	0.00680
0.10	4.4	0.00420
0.20	2.0	0.00552
0.40	1.9	0.06720

The first column behaves as anyone would expect: a wider band switches less. The second column is the one worth pausing on. Regret does not pay for that stability — it improves, from $0.00961$ at $\delta = 0$ to $0.00420$ near $\delta = 0.10$, a reduction of more than half.

Both ends of that range lose. That is asserted seed by seed rather than on the average, because an interior optimum that exists only in the mean is a weaker object than one that exists in every run — and because this arc has been burned by a fragile interior optimum before. Across all ten deployments, the best band is interior every time and beats both the myopic rule and the widest band every time. Where it sits varies: the argmin lands on $0.02$, $0.05$, $0.10$ and $0.20$ depending on the seed. So the claim the module makes is that an interior optimum exists, and never where it is.

A · a workload that movesB · the control bandC · noise, not costD · detecting ≠ deciding

hysteresis band = 0.100deployment (seed) = 1 of 10

the best arm changes

1×

myopic switches

6×

banded switches

2×

myopic regret

0.00271

banded regret

0.00284

The bridge share drifts from 0.2 to 0.7 over 60 windows, crossing one boundary of the partition the predecessor drew. The top track is the architecture that is genuinely best at each window; it changes once. The middle track re-decides from each window’s 40 observed queries and changes far more often, and every red tick beyond the dashed line is a switch that the noise proposed rather than the workload. Widening the band collapses the middle track toward the top one, and it does that on every deployment: switching is non-increasing in the band on all 10 of them.

Regret is the subtler quantity, and this panel shows one deployment — a single draw of exactly the noise the topic is about. On every deployment some interior band beats switching freely. But which one varies, and on this first deployment the headline band of 0.10 is slightly worse than switching freely while a narrower band beats both. That is not a wrinkle to look past; it is the reason the topic’s claim is an average over deployments rather than a reading off any single one, and the reason the next panel is the one that actually makes it. Move the seed slider to see how much the picture moves with it.

The six architectures, their quality and their costs arrive unchanged from the three predecessor topics; nothing is re-measured here, and the only new thing is the time axis. A deployment observes 40 queries per window over 60 windows, and every figure is averaged over 10 deployments, because a single run is one draw of exactly the noise this topic is about. Every switch count, regret and optimal band on this page is recomputed live from the two utility streams the policy is a function of; only the KS and PSI readings and the static-workload table, which need SciPy, are baked.

Hysteresis is about noise, not about cost

Here is the part that inverts the setup. We introduced the band by way of a switching cost, and the classical result that motivates it is a result about switching costs. So the natural expectation is that the band is a device for amortizing an expensive change, and that making changes free should collapse it.

It does not. Sweeping the price $c$ of a single switch, and choosing the band that minimizes total cost $R(\delta) + c \cdot \#\{\text{switches}\} / T$:

price of a switch	optimal band	switches	regret
0	0.10	4.4	0.00420
0.001	0.10	4.4	0.00420
0.005	0.10	4.4	0.00420
0.02	0.10	4.4	0.00420
0.1	0.20	2.0	0.00552

At a price of exactly zero — switching is free, there is nothing to amortize — the optimal band is already $0.10$, the same band that is optimal across four decades of price. The price only moves it at $c = 0.1$, where it widens to $0.20$.

That flatness is the evidence. A band whose job was paying for switches would track the price of switches continuously. This one does not notice the price until the price is large, because its first job is something else: it is filtering an estimator. The band has to be wide enough to absorb the fluctuation in $\hat U_t$ and no wider, and how wide that is depends on how many queries each window observes — not on what a reindex costs. The switching cost is a second, and here a much smaller, reason to keep the band.

This is why the static table above is the topic’s control rather than a curiosity. A deployment whose workload never moves at all still needs a band, and needs the same band, because the thing the band is protecting against was never the workload.

Detecting is not deciding

There is an obvious objection to all of this: why infer drift from the decision at all? Monitor the workload directly, alarm when it moves, and switch then.

The drift detectors for exactly this arrive unchanged from the calibration topic — a two-sample Kolmogorov-Smirnov test and the population stability index, run on a twenty-window reference against the last twenty windows. Pointing them at the same streams the policy sees:

path	real changes	myopic	banded	KS $p$	PSI	the detectors say
static	0	35	17	0.9831	0.039	nothing happened
ramp	1	6	2	0.0000	2.569	drift
wobble	2	15	5	0.8320	0.303	drift (PSI only)

The two instruments fail on the same data in opposite directions.

On the static path the detectors are entirely correct — nothing happened, and they say so at $p = 0.98$ — and a myopic policy changes architecture $35$ times regardless. Detection is right and useless, because the problem was never that the workload moved.

On the wobbling path the situation reverses. The genuinely best architecture changes twice, and the policy has to move with it. The Kolmogorov-Smirnov test sees nothing at all, $p = 0.83$, and it is not wrong either. The excursion happens in the middle. The reference window spans $[0.43, 0.62]$ and the current window $[0.32, 0.67]$, while between them the share falls to $0.29$ — so the two windows the test compares really are similar, and everything that happened happened where the test does not look. PSI, comparing the same two windows with less power to say so, alarms weakly.

That contrast is also what keeps this topic distinct from the calibration one it borrows from. There, drift is detected in retrieval quality, and the topic stops at detection — which is the right place to stop when the response is a human investigation. Here the response is automatic and expensive, so the question is not whether the signal is real but whether acting on it pays, and those have different answers often enough to be worth separating.

What this establishes, and what it does not

The measurements are specific and the framing is general, so it is worth being explicit about which is which.

What the simulation establishes on this instance: a myopic rule chatters at an order of magnitude above the true rate of change; a control band suppresses that chatter while improving decision quality rather than trading against it; the best band is interior on every deployment tested; and it is already wide when switching is free, which locates its purpose in the estimator rather than in the switching cost. The non-monotone path reaches the same verdict on every one of those points, so none of them is an artifact of a straight-line drift.

What it does not establish: that $0.10$ is a band anyone else should use. The optimal band depends on how many queries a window observes, on how far the traffic sits from a cell boundary, and on the prices at which the utility was computed — all of which are held fixed here at one arc’s values. The transferable content is the shape of the answer and the diagnostic that goes with it. If a deployment is changing architecture far more often than its workload is changing, the band is too narrow, and the right response is to widen it rather than to improve the estimate.

The classical control-band result is cited as the frame, not re-derived: what is shown here is that a band is better than a threshold on this problem, not that the optimal policy for this problem is a band. The observation model is also the most favorable one available to a detector — a clean multinomial per window, independent across windows — and real traffic is autocorrelated, which makes every detector in the final movement look better here than it would in production. The gap that movement measures is therefore a lower bound on the gap between detecting and deciding, which is the direction that keeps its conclusion safe.