Pick the Plateau, Not the Peak

Swapping the reranker in a code-search pipeline is three decisions, not one: which model, where to set the knobs around it, and what becomes of every number that was tuned on the old model's scores. One real swap, a five-point sweep that zigzags, and a confidence floor that quietly stopped firing.

Changing the reranker in a retrieval pipeline looks like a one-line edit: a different model id in a config file. The model is the part everybody measures. Two other decisions ride along with it. The knobs around the model need settings, and the obvious way to pick them, taking whichever setting scored best, misleads on a small evaluation set. And every threshold downstream that was tuned against the old model's scores has to be checked again, a step that tends to get skipped because skipping it breaks no test.

The evidence here is one swap in vectr, an open-source code-search and agent-memory tool I maintain. Its default reranker changed from BAAI/bge-reranker-base to Alibaba-NLP/gte-reranker-modernbert-base, and the change was judged on an acceptance suite of 65 queries over the source tree of Django. Fifty-seven of those cases are scored automatically and eight are judged by hand; six are marked must-pass.

The model change was an easy call, 43 of 57 cases against 35. The pool depth that goes with it was not. As the candidate pool grows, the must-pass count goes 4, 5, 4, 5, 5, so the cheapest setting that reaches five sits alone between two fours, and the value that shipped came from the flat stretch after it, at nearly twice the latency. The swap also cost something no pass count recorded. A confidence floor at 0.30, tuned while the old model was in place, did not fire once in a 65-query study run under the new one.

Part 01
The Swap
01

What a Cross-Encoder Reranker Does in Two-Stage Code Search

Code search over a large repository usually runs in two stages, because the accurate way to judge relevance is too slow to run on everything. The first stage is cheap. An embedding model turns every chunk of code into a vector ahead of time, and at query time a nearest-neighbour lookup returns the chunks whose vectors sit closest to the query's. A keyword ranker, BM25, runs alongside it and catches exact identifier matches that vectors blur. Together they produce a candidate pool: a few dozen chunks that probably contain the answer.

The second stage reads each candidate properly. A cross-encoder takes the query and one chunk as a single input and outputs one relevance score. The sentence-transformers documentation draws the line against an embedding model in two sentences: "for a Cross-Encoder, we pass both sentences simultaneously to the Transformer network," and "A Cross-Encoder does not produce a sentence embedding." Because query and chunk are read together, every layer of the model can relate a word in one to a word in the other. That is where the accuracy comes from, and the cost too. Nothing can be computed in advance, so every (query, chunk) pair is a full pass through the model at query time.

The pattern goes back at least to Nogueira and Cho's 2019 BERT passage re-ranker, which scored the passages from a BM25 first stage and sorted by score. One sentence from their method section matters later: "We compute this probability for each passage independently and obtain the final list of passages by ranking them with respect to these probabilities." A chunk's score does not depend on which other chunks share the pool. The term for that is pointwise scoring.

Analogy: the CV screen and the interview

A company that receives thousands of applications cannot interview everyone. A keyword screen and a quick skim produce a shortlist; interviews then judge each shortlisted candidate against the actual job. The interview is far more accurate and far more expensive, so the length of the shortlist sets both the bill and the ceiling. A candidate who never made the shortlist cannot be hired, however good.

Two details carry over. Each interview judges one candidate against the job, not candidates against each other. And lengthening the shortlist never removes anyone who was already on it; it only adds competition. Both details come back when the pass counts start to zigzag.

What the second stage costs
rerank time ≈ fixed overhead + (time per pair) × (pairs in pool) pairs in pool ≈ 1.2 × cap (vectr unions two lists of size cap)

vectr's measured sweep (Section 03) gives a median rerank time of 599 ms at a pool cap of 10 and 2681 ms at a cap of 60. The configuration states the hardware for the shipped point only: an Apple M1 GPU at batch size 4. That is (2681 - 599) / (60 - 10) ≈ 41.6 ms per unit of cap, or roughly 35 ms per scored pair after the 1.2 factor. Both numbers are derived from five medians rather than measured per pair, but they settle the order of magnitude. Moving the cap from 10 to 60 multiplies rerank time by about 4.5. Swapping the model, on identical pools, multiplied it by 2.1 (Section 02). The cap is the bigger dial.

02

Why a Text Reranker Failed on Code, and What the Swap Bought

The old default, bge-reranker-base, is built on XLM-RoBERTa. Its model card lists Chinese and English and says the cross-encoder was trained "on a multilingual pair data". On code it did something worse than being noisy. For the query "get object by primary key" it scored QuerySet.get, the method the test expects, at 0.0275, and SingleObjectMixin.get_object at 0.9258. For "caching framework implementation" it scored BaseCache at 0.0019. The configuration comment that records these numbers also rules out the easy explanation: every chunk involved fit under the 512-token input cap, so nothing had been cut off.

Django's source shows what the model was rewarding. get_object is a view helper. Its docstring reads "Return the object the view is displaying", and its body carries the comment "Next, try looking up by primary key." shortly before it calls queryset.get(). The docstring of QuerySet.get never mentions a key: "Perform the query and return a single object matching the given keyword arguments." The wrapper contains the query's words, and the method contains the operation. Rewarding the words over the operation is what you would expect from a ranker trained on prose passages, and vectr's configuration describes bge as exactly that, "an NL passage ranker that inverts relevance on code".

In fairness, get_object is not an absurd answer: it does fetch an object by primary key, by calling the expected answer, and a different grader could accept it. What no grader would accept is 0.0275 for the method that does the work. The comment's diagnosis: "model miscalibration on the code domain, not a context-length artifact".

The replacement, gte-reranker-modernbert-base, is built on ModernBERT. Its model card reports results on CoIR, a code-retrieval benchmark, with a reranker average of 79.99 over 20 tasks. The two defaults compare as follows.

bge-reranker-base (old)gte-reranker-modernbert-base (new)
Cases passing, of 573543
Must-pass, of 635 (F23 still fails)
Default pool cap6040
Median rerank, as reported~1634 ms~1957 ms
Same pools, batch size 41252 ms2656 ms
On disk1.1 GB574 MB
Parameters~278M, of which ~192M is the token table~150M (card: 149M), ~39M token table
Transformer layers1222
Vocabulary250,00250,368
Input length used512 tokens (model allows 514)512 tokens (model allows 8,192)
Custom code on loadnot requirednot required

Pass counts, latency and size from vectr's configuration and commit d84a2e5; Apple M1 GPU, batch size 4. The bge latency is reported without a pool depth; the gte figure is pool 40. The same-pools row is the batch-size sweep in Section 07. Parameter, layer and length figures from each model's published config and weights file; vectr truncates every pair at 512 tokens.

The pass counts compare two complete configurations, not two models. The changelog describes both as measured "under shipped defaults", and the same change lowered the pool cap from 60 to 40. The new model at the old cap of 60 passes 44 (Section 03). If the old model's 35 was measured at its own default of 60, as that wording implies, then at a matched pool the model is worth nine cases and the move to 40 gave one of them back. F23, the "get object by primary key" query that exposed the old model, still fails under the new one at pool 40. The public log records the miss, not where in the pipeline it happens. A related failure, in which a chunk whose docstring paraphrases the query never reaches the candidate pool, is the subject of Embedding Dilution: Why Code Search Misses the Answer.

The new model is also smaller on disk and slower, and the reason is the vocabulary. bge's 250,002 tokens at 768 dimensions make a lookup table of about 192 million parameters, close to 69 percent of the model, and reading a row of a table costs almost nothing. The time goes into the stack of transformer layers above it: about 86 million parameters in 12 layers for bge, about 111 million in 22 layers for gte. To a first approximation, compute per token scales with those non-embedding parameters, which predicts gte at about 1.3 times bge's cost. On identical candidate pools it measured 2.1 times (2656 ms against 1252 ms per call). The two tokenizers also cut the same code into different numbers of tokens, and the two architectures run attention differently; which of these accounts for the rest of the gap was not profiled.

The last row matters to anyone shipping a model onto other people's machines. A model repository on Hugging Face can carry its own Python modelling code, and loading such a model requires trust_remote_code=True, which executes that code. gte-reranker-modernbert-base needs no such flag: its config names the standard modernbert architecture with no custom-code mapping, and the examples on its card load it with a plain call under "transformers>=4.48.0". Its multilingual sibling, gte-multilingual-reranker-base, does map to custom code, and its card passes trust_remote_code=True.

Part 02
The Setting
03

The Pool-Depth Knob That Actually Governs Rerank Cost

A reranker only scores what the first stage hands it, so the size of that pool, often called the rerank depth, is the real control. vectr had two settings for it. top_k: 40, commented "rerank this many hybrid candidates before trimming to n_results", reads like the one that matters. It applies only when a query carries a language or path filter. The other, top_k_unfiltered, applies otherwise. Not one of the 65 acceptance cases sets a language filter, so during the evaluation the knob with the obvious name was never read, and every query went through top_k_unfiltered. The configuration comment now says so in capitals: "THIS, not top_k above, is the knob that governs almost every real query".

A knob the evaluation never reads looks exactly like a plateau

Sweep top_k on this suite and every value gives the same pass count and the same latency: a perfectly flat line. A flat line is also what robustness looks like. Before reading anything into a sweep, confirm that the evaluated queries take the code path that reads the knob. Here that check is one count: how many cases set a language filter.

top_k_unfiltered was lowered from 60 to 40 alongside the swap, after a sweep over five values with the new model. From here on, "pool 40" is shorthand for a cap of 40.

Pool capMedian rerankCases passing, of 57Must-pass, of 6
10599 ms404
201074 ms415
301580 ms414
40 (shipped)1957 ms435
602681 ms445

gte-reranker-modernbert-base on the Django acceptance suite. The configuration gives Apple M1 GPU and batch size 4 for the shipped point, not row by row. The effective pool is about 1.2 times the cap. Source: the comment above top_k_unfiltered in vectr's config.yaml.

The case count rises almost steadily: 40, 41, 41, 43, 44. The must-pass column does not: 4, 5, 4, 5, 5. Pool 20 reaches five must-pass cases at 1074 ms, 55 percent of pool 40's latency, and on must-pass per millisecond it wins outright. It was not chosen. The configuration comment sums up the decision as "Robustness over the better headline".

04

How a Deterministic Search Pipeline Produces a Zigzag

The configuration names the cause: "must-pass-six is NON-MONOTONIC because F22 flips across the sweep (verified deterministic by a byte-identical rerun on a fresh daemon)". F22 is the query "delete rows from database table". It passes when QuerySet.delete is in the top five results and either QuerySet.delete or Model.delete is in the top three. If F22 is the only must-pass case that changes, which is how the counts read, it fails at 10, passes at 20, fails at 30, and passes again at 40 and 60.

The query is hard before any reranker sees it. Django defines delete on cache backends, session stores, file storage and the ORM, and the case's own label, delete-floods-cache-not-db, names the look-alikes that crowd the top results. I Deleted Every Ranking Heuristic From My Code Search follows this exact query: why rules about the query's words could not separate the right delete from the others, and what replaced them.

"Deterministic" should be read at the strength it was claimed. The evidence is one byte-identical rerun on a fresh daemon. That rules out run-to-run noise as the cause: the same index, query and settings produced the same output twice. It says nothing about whether the answer would survive a slightly different index.

A rerun says nothing about the next index

Run-to-run variance is what a rerun measures, and this one found none. Instance-to-instance variance is what happens when the corpus gains a file, a chunk boundary moves, or the same intent is phrased differently. A system can reproduce its output exactly and still sit on a knife-edge across instances. The F22 flip reproduced on this index; nothing has been measured about the next one.

Why nested pools and a pointwise scorer should not zigzag

If results came back in the cross-encoder's order, the flip could not happen, because of two properties of the pipeline. The pools are nested: vectr builds the pool as the union of the first k non-trivial chunks in the embedding ranking and the first k in the BM25 ranking, so everything in the 20-deep pool is also in the 30-deep pool. And the cross-encoder is pointwise, so a chunk's score does not change when other chunks join the pool.

Take a test of the form "the target is in the top five". Before the target enters the pool, the test fails. Once it enters, some set of chunks scores above it, and a deeper pool can only add to that set, never take anything away. The target's rank can only get worse as the pool grows. So the test passes from the depth where the target enters until the depth where a fifth higher-scored chunk arrives, and fails everywhere else: one unbroken interval of depths. Pass, fail, pass cannot happen.

F22 escapes that argument in two separate ways.

Escape one: a test with two acceptable answers

F22's top-three condition accepts either QuerySet.delete or Model.delete. Suppose QuerySet.delete is third in the 20-deep pool, a new competitor arrives at 30 and pushes it to fourth, and Model.delete first enters the pool at 40 and lands in the top three. Pass, fail, pass, with no randomness and nothing but a pointwise scorer. The interval argument holds for each answer on its own, but the union of two intervals need not be an interval.

Escape two: a final order that depends on pool size

vectr does not return the cross-encoder's order directly. After the rerank, each candidate gets a composite score: a base taken from its position in the reranked list, multiplied by factors for code quality, for how important the chunk's file and class are in the codebase, and for a second similarity score between the query and the chunk's qualified signature and docstring. Each chunk's factors are fixed for a given query; the pool size does not change them. The base does change with it.

A rank-based base score that depends on n
base(i) = 1 - i / n i = position after the rerank (0 = first), n = pool size score(i) = base(i) × q × f_file × f_class × f_purpose × f_def

Take two chunks. A is first after the rerank, with combined factor 1.00. B is second, with combined factor 1.05.

n = 10: A = 1.000 B = (1 - 1/10) × 1.05 = 0.945 A ranks first n = 30: A = 1.000 B = (1 - 1/30) × 1.05 = 1.015 B ranks first

B overtakes A whenever (1 - 1/n) × 1.05 > 1, which is any pool larger than 21. Now let the pool grow to 40 and admit one new chunk C, with factor 1.00, which the cross-encoder places between A and B. B drops to position 2:

n = 40: A = 1.000 C = (1 - 1/40) × 1.00 = 0.975 B = (1 - 2/40) × 1.05 = 0.9975 A ranks first again

The same two chunks change order twice as the pool grows, and nothing random happens anywhere. Gaps between adjacent base scores are 1/n, so a fixed multiplicative factor buys more positions in a bigger pool, while any newcomer that lands between two chunks pushes them apart again.

Deduplication runs in the same step and collapses near-duplicate results, which can also change which chunk occupies a position. And "pointwise" is a statement about the mathematics. In floating point, a pair's score can move in its last digits with the padding of the batch it lands in, and batch membership shifts as the pool grows. The batch-size check in Section 07 found no top-five change from that effect on Metal, so it is an unlikely cause here, but nobody has ruled it out for F22. The public record attributes the must-pass zigzag to F22 and does not say which mechanism flips it. Either of the two escapes is enough to produce the pattern.

In a pipeline built like this, each case's pass or fail is a step function of pool depth, and the must-pass count is the sum of six step functions. With six cases, one step is 16.7 percentage points. The must-pass column is not a smooth curve sampled with a little error. It is a handful of all-or-nothing events whose positions along the axis were fixed by this particular index.

05

The Optimizer's Curse: Why the Best Setting Overstates Itself

Choosing a setting because it scored best is an old problem with a precise name. Smith and Winkler, writing about decision analysis in 2006:

We show that if we take these value estimates at face value and select accordingly, we should expect the value of the chosen alternative to be less than its estimate, even if the value estimates are unbiased. Thus, when comparing actual outcomes to value estimates, we should expect to be disappointed on average, not because of any inherent bias in the estimates themselves, but because of the optimization-based selection process.

They called it the optimizer's curse, and it maps onto a sweep directly. Each setting's pass count estimates how that setting would do on the queries it will actually meet. Grant that each estimate is unbiased, which is generous for a suite whose case names (F31-clean-slate-F1-regression, F22-shortverb-delete-floods-cache-not-db) show it was built largely from past failures. Picking the maximum still selects partly on the error, so the chosen setting's score is, on average, too high. The larger the error and the more settings compared, the larger the overstatement.

The error that matters here is not run-to-run noise, of which the rerun found none. It is the instance-to-instance variance from Section 04. These 57 queries over one index are a sample of the queries and codebases the setting will meet, and a different sample would move every count.

How large the overstatement gets

A pass rate measured on n independent cases has a standard error of

SE = sqrt( p × (1 - p) / n )

For the six must-pass cases at a true rate of 0.8, SE = sqrt(0.16 / 6) ≈ 0.163, so one standard error is about one case (1/6 ≈ 0.167). For 57 cases at 0.75, SE ≈ 0.057.

If K settings were independent estimates with the same true value and the same SE, the best of them would land on average E[max] standard errors above the truth, where E[max] is the expected largest of K draws from a standard normal distribution:

K = 2: 0.564 K = 3: 0.846 K = 5: 1.163 K = 10: 1.539

Five independent settings scored on six cases would overstate the winner by about 1.163 × 0.163 ≈ 0.19, more than a full case, before any real difference between the settings enters. Real sweeps are not independent. Neighbouring pool depths share most of their candidates, so most cases score the same at both. That correlation shrinks the effect. It also means a single case flipping stands out as an isolated bump, like the one at pool 20. At n = 6 the normal approximation is rough, because a pass rate cannot exceed 1; the simulation below drops it.

The same arithmetic applies to the case-count column. With 57 cases, one standard error is about three cases, so 41 against 43 against 44 sits inside the noise too.

Interactive · Demo 1
How Far Does the Best of Several Identical Settings Overstate?

Every setting in this simulation has exactly the same true pass rate, so every difference between them is sampling noise. Change the number of cases and settings, and watch how far the best-looking setting overstates the truth, how often a dip appears in a sweep of identical settings, and how much averaging each setting with its neighbours pulls the overstatement back.

5
80%
0.80
The chart shows one simulated sweep; the numbers average 2,000. Each case's outcome across settings follows a correlated Gaussian walk thresholded at the true pass rate, so neighbouring settings agree on most cases. A dip is a setting, or a run of equal settings, that scores below the settings on both sides of it.

With the defaults (six cases, five identical settings, neighbours strongly correlated) the best setting reports about 92 percent against a true 80, and close to half of all sweeps contain a dip like the one at pool 30, from noise alone. At 57 cases the overstatement falls to about four points. Dips become more common as cases are added, though much shallower: a few points at 57 cases, while at six cases every dip is at least one whole case, 16.7 points. So a zigzag is weak evidence on its own. What matters is its depth measured in cases, and the must-pass column's one interior dip, at pool 30, is a whole case: 16.7 points.

Smith and Winkler's remedy is to stop taking the estimates at face value: adjust each one toward what was expected before looking, and adjust it harder the noisier it is. They describe the Bayesian version as "disciplined skepticism". For an ordered knob like pool depth there is an obvious thing to shrink toward, and that is the neighbours. If the true pass rate changes slowly with depth, a setting's neighbours carry information about it, and an estimate that agrees with its neighbours has earned more trust than one that disagrees with both.

06

Pick the Plateau, Not the Peak: Choosing a Pool Depth of 40

The rule that shipped, in the commit's words: "40 was chosen as the low end of a contiguous plateau rather than the isolated peak at 20." As a procedure: find the best must-pass count, find the longest run of adjacent settings that all reach it, and take the cheapest setting in that run. Pools 40 and 60 both reach 5 of 6. Pool 20 reaches it alone, between two settings at 4.

The plateau rule leaves the reported number alone. The setting it picks still reports 5 of 6, the same maximum the peak reports, so as an estimate it carries the same optimism. What changes is which setting carries the number: one whose neighbour also scored 5, which is weak evidence that the 5 at 40 is not a one-off. Shrinkage proper changes the number as well. Averaging each setting with its neighbours gives 40 a smoothed score of 4.67 (from 4, 5 and 5) and 60 a score of 5.0, and picks 60.

Other reasonable rules pick differently on the same five points, and where they land shows how thin the evidence for any one of them is.

Interactive · Demo 2
Five Rules, One Sweep: Which Pool Depth Would Each Choose?

Pick a selection rule and see which pool depth it chooses from the measured sweep, and what that choice costs in latency. Then add the one point that was never measured, pool 50, and see how much the choice depends on it.

Bars show must-pass cases out of 6. Measured values from vectr's configuration; pool 50 was never run, so its latency and case count are blank. Settings at the ends of the sweep have one neighbour, so the strict and smoothed rules see less evidence for them than for interior settings.

On the real sweep, the stricter rules all land on 60. Sixty is the only setting at 5 whose every measured neighbour agrees, it has the best smoothed score, and it passes the most cases (44, one more than 40, well inside the noise). Forty is a compromise between the robustness argument and a 724 ms difference in median latency per query, and it is better described as one. The public record states the rule only alongside the sweep's results. A rule picked after the data has been seen is one more degree of freedom, and writing it down before the sweep runs removes it.

The point that was never measured matters more than the choice of rule. If pool 50 scores 5, the run from 40 to 60 has three points and the plateau rule stays at 40. If it scores 4, there is no plateau at all. The plateau rule falls back to the cheapest of three isolated peaks, which is 20, and the smoothed rule picks 30, a setting that itself scored 4. One run at pool 50 decides which of those situations the choice is in, and it is the first experiment I would run on this knob.

Flat minima: an analogy, not a theorem

Analogy: flat minima in neural network training

Hochreiter and Schmidhuber defined a flat minimum as "a large connected region in weight space where the error remains approximately constant", and argued that such regions correspond to simpler networks with lower expected overfitting. Twenty years later Keskar and colleagues reported evidence that "large-batch methods tend to converge to sharp minimizers of the training and testing functions", while "small-batch methods consistently converge to flat minimizers", with the sharp ones generalising worse.

The intuition carries over to a sweep: a setting whose neighbourhood performs the same is less likely to owe its score to one accident. None of the mathematics comes with it. Those results concern millions of continuous weights and a training loss, and this is one integer knob and a pass count. The formal claim is also contested on its home ground. Dinh and colleagues argue that "most notions of flatness are problematic for deep models": the symmetries of networks built from rectifier units let one construct equivalent models whose minima are arbitrarily sharper, and a reparametrization can change the geometry of the parameters drastically without changing how the network generalises.

One more trap is in the vocabulary. The "batch" in Keskar's title is a training batch. It has nothing to do with the inference batch size in Section 07.

What the choice of 40 cost, and what it does not show

  • The plateau is two points. Pool 50 was never measured.
  • The low end of a plateau is next to its edge. Pool 30 is a dip, one step below 40.
  • One case paid for it. F30, "filter queryset by field value where clause", expects QuerySet.filter; it fails at 40 and passes at 60. The swap commit says so: "Known cost at pool 40: F30 regresses (present at pool 60)."
  • All of it is Django. From the configuration: "pool depth interacts with corpus chunk density in a way model choice does not, so cross-corpus validation is open." A cap of 40 chunks covers a different share of the relevant code in a repository cut into many small chunks than in one cut into few large ones.
Part 03
Cost-Only Knobs
07

Reranker Batch Size: A Knob That Changes Speed and Nothing Else

The new model was slower per call. A change committed four days before the swap took most of that extra latency back, using a parameter nobody had set. vectr's rerank called the library's predict() with no batch size, which in the commit's words meant "silently inheriting sentence_transformers' default of 32". Batch size is how many pairs go through the model in one forward pass. It changes how the work is laid out on the hardware, not what is computed for each pair.

DeviceModelBatch 32 (old default)Batch 4Speed-up
Apple GPU (Metal)bge-reranker-base2471 ms1252 ms1.97×
Apple GPU (Metal)gte-reranker-modernbert-base4067 ms2656 ms1.53×
Apple CPU, 5 threadsbge-reranker-base4450 ms2610 ms1.70×
Apple CPU, 5 threadsgte-reranker-modernbert-base6297 ms4219 ms1.49×
CUDA GPUbothnot measurednot measuredkept at 32

Metal (mps): 40 queries with candidate pools of 59 to 92 pairs, median per call. CPU: 10 queries, two repetitions, mean of the medians, with the thread cap production uses. Apple M1 only. Sources: vectr commits d7315f8 and f6b403c.

Batch size 32 was the worst value tested for both models on both backends, and on CPU 4 was the best. On Metal the rank order was checked as well: the top-1 result and the top-5 set were identical to batch size 32 at every tested value. A knob that moves latency and leaves everything downstream untouched is a cost-only knob, and it can be tuned by itself, per device, without re-running the quality evaluation. The rank check is recorded for Metal; the CPU sweep measured time.

The fix landed first, so its arithmetic shaped the swap. Before it, both models ran at 32, and moving from bge to gte would have taken a call from 2471 ms to 4067 ms, 65 percent slower than what was running. After it, gte at 4 costs 2656 ms, within 7.5 percent of what bge had been costing before anyone set the batch size. The fix speeds up both models, so on equal footing gte is still about 2.1 times slower per call. What changed is that the swap no longer required a latency regression against what users already had.

Why smaller batches would be faster is less clear than the measurement. The configuration's explanation is that "smaller batches parallelize better under Metal for this pool-size range", and the same direction on CPU suggests the effect is not specific to Metal. Padding is the usual first suspect, since every pair in a batch is padded to the longest one. But the sentence-transformers version vectr pins, 5.5.1, sorts the pairs by length before it splits them into batches, so padding within a batch is already kept small. The cause was not profiled, and I would not bet on any single explanation without a profile.

Keyed by device, defaulting to the library

The result runs against the usual intuition for accelerators, where a bigger batch keeps a large GPU busier. Nobody measured CUDA, so the configuration keys batch size by the device the model actually loaded on and leaves anything unmeasured at the library default:

agent/config.yaml · ranking.rerank.batch_size
batch_size:
  mps: 4
  cpu: 4
  cuda: 32
  default: 32

The default entry covers any other device, and the cpu figure was measured on Apple Silicon only, so x86 processors may see a different magnitude. The device comes from the loaded model's own CrossEncoder.device.type, the attribute the library populates itself, rather than from a second hardware check. The comment on the cuda entry names the risk of guessing: "guessing a small value here would risk silently regressing CUDA throughput with no local way to catch it."

Quality knobs and cost knobs

Pool depth changes which chunks get scored, so it trades quality against cost. Batch size changes only how they are scheduled. Sweeping both together yields a frontier where nobody can tell which knob bought which millisecond. Checking that a knob is cost-only (here, identical top-1 results and top-5 sets) earns the right to tune it on its own, per device, and leave the quality evaluation alone.

Part 04
The Constants
08

Score Thresholds After a Reranker Swap: The Floor That Stopped Firing

The acceptance suite checks where results land, and vectr's ranking reads only the cross-encoder's order: the base score in Section 04 is 1 - i/n, whatever the model's raw scores were. A replacement model that scored every pair on a different scale but in the same order would produce the same ranking. The scale matters in the decisions made around the ranking: which results to drop, whether to warn the caller, which results to show in full.

vectr puts a low-confidence banner in front of the calling agent when its results are probably not the answer. The banner says no strong match was found and suggests an exact symbol lookup, a rephrased query, or grep. While it is up, pointer mode strips the code out of every result and leaves a bare pointer the agent can fetch if it wants the body. Two independent signals raise the flag. One is about vocabulary: a content word of the query with zero document frequency in the indexed corpus marks a topic the codebase does not contain. The other is about score: the top result's cross-encoder relevance falls below an absolute floor, min_top_relevance: 0.30.

The score signal needs an absolute number, and the composite that orders the results cannot supply one, because it is rebuilt from each query's own candidate order. The comment that introduced the banner puts the consequence plainly: "the top result of any query always displays close to 1.0 even when every candidate is irrelevant." The cross-encoder's own output, kept as ce_relevance, is the only per-pair absolute judgment in the pipeline. The comment above the floor justifies it on that basis: a cross-encoder score is "a different, calibrated relevance judgment (trained to predict query-doc relevance directly), so it is safe to floor on." The value 0.30 was, in a later commit's words, "calibrated against the previous cross-encoder".

What a cross-encoder score is, and what it is not

"Calibrated" is the word to check. A cross-encoder's final layer outputs one unbounded number per pair, a logit z. sentence-transformers passes a single-output model's logit through a sigmoid, σ(z) = 1 / (1 + e^-z), which maps it into the range 0 to 1. The result looks like a probability. Whether it behaves like one, so that 0.3 means roughly a three-in-ten chance of relevance, depends on how the model was trained and on what data it is now reading.

Why the absolute level is not pinned

Suppose a reranker is trained with a softmax over a group of candidates for the same query, one relevant and several not. The loss for that query is

L = -log( e^(z_pos) / Σ_j e^(z_j) )

Add the same constant c to every logit in the group and L does not change, because e^c cancels between numerator and denominator. The loss constrains the gaps between one query's scores, never their level, so σ(z) can sit anywhere along the curve in a perfectly trained ranker.

A pointwise model trained as a binary relevant-or-not classifier, the way Nogueira and Cho trained theirs, does have its level pinned, but only for its training distribution, including its ratio of relevant to irrelevant pairs. Their negatives were passages from BM25's top 1,000 for each query. A code-search pool has a different mix of near-misses and a different domain.

Either way, a threshold on σ(z) is a property of one model on one kind of data. bge's model card says as much about its raw output: "The reranker is optimized based cross-entropy loss, so the relevance score is not bounded to a specific range." The standard repair is to recalibrate per model and domain with Platt scaling: fit a scale a and an offset b in σ(a·z + b) on labelled pairs from the target corpus, so that a score of 0.3 again means what it says. That needs labelled present and absent queries for every corpus the threshold will serve, which is the material that was missing.

Under the new model, 0.30 never fired

After the swap, a distribution study ran 65 queries through the search path on Django and on TigerBeetle, with the new model at a pool cap of 40, and grouped each query's cross-encoder relevance by what the right outcome would be:

CorpusQuery classQueriesce_relevance range
Djangoknown-good200.767-0.974
Djangoknown-miss-in-topic200.834-0.952
Djangoknown-absent80.515-0.894
TigerBeetleknown-good80.877-0.948
TigerBeetleknown-absent60.786-0.879

gte-reranker-modernbert-base, pool cap 40. Ranges and class labels as published in vectr commit 17f4ab8; known-miss-in-topic is the commit's label for on-topic queries whose expected answer the search misses. The listed classes account for 62 of the 65 measurements; the commit does not break out the other three.

Every class sits above 0.5. The commit's summary: "No single absolute value separates the classes under the current reranker; min_top_relevance (0.30) never fired via its own sub-signal across all 65 measurements". The three low-confidence flags that did appear all came from the vocabulary signal, on queries that really were off-topic. The acceptance queries tell the same story. Across 44 of them at pool 40, the top score ran from 0.7256 to 0.9742 with a median of 0.915, and all 220 top-five scores fell between 0.66 and 0.98.

No test failed when this happened, and no pass count moved: a floor that never fires changes no ranking, so a suite that checks rankings cannot see it. The score half of the banner was switched off in effect by the same change that lifted the pass count.

The old model was no cleaner, only wrong in a different way. The configuration's summary covers both: no single value separates the classes "under either the current reranker (all three classes overlap in a narrow 0.5-0.97 band) or the prior bge-reranker-base default (all three classes overlap in a 0.02-0.99 band ...)". Under bge the floor fired, sometimes on a correct answer, as Section 09 shows. Under gte it went silent.

Interactive · Demo 3
Drag a Score Floor Across Two Models' Score Ranges

Move the floor and see, for each class of query, whether a banner keyed to it would fire. Under the old model, look at what 0.30 does to the correct answer that scored 0.024. Under the new one, look for any floor that fires on every absent query and on no good one.

0.30
00.250.50.751
Only published ranges are shown, not individual queries, so a floor inside a range fires on some unknown share of it. The old model's two bands are approximate, measured on the acceptance fixture and published in vectr's configuration comment on ce_override_min_relevance. The 0.024 point is the correct top result reproduced with bge-reranker-base in commit 17f4ab8.
A threshold that stops firing makes no noise

An alarm that fires all the time gets fixed, because someone complains about it. An alarm that never fires produces no complaints, and from outside it looks just like a codebase where nothing was ever missing. The silent floor here showed up only in a per-class score study run after the swap.

vectr had met this once before with a different signal. Its first not-found floor was an absolute cutoff on embedding cosine similarity. Measured on a large corpus, absent-topic and on-topic queries landed in the same band of roughly 0.60 to 0.86, and the constant never fired. The configuration's verdict on that one was "The defect was the signal, not the threshold." On the published ranges, the same verdict fits the cross-encoder floor under the new model.

Why neither obvious replacement shipped

Raising the floor does not work. Django's known-absent class reaches 0.894, above the bottom of the known-good range, so any floor that catches every absent query also flags good ones. A relative test, whether rank 1 clearly leads rank 2, was the next candidate, and a preliminary check on Django "showed no clean separation".

What shipped instead is the mechanism with its threshold left open. The relative test asks whether the top result leads the second by a margin measured against the headroom left above the top score:

A gap that scales with the headroom
decisive lead: s1 ≥ min_top and (s1 - s2) ≥ min_drop × (1 - s1) s1, s2 = the top two ce_relevance scores; min_top = 0.30, min_drop = 0.50

With min_drop = 0.5, a top score of 0.95 needs a lead of only 0.025 over the second, and a top score of 0.60 needs a lead of 0.20. A fixed required gap treats those two situations the same, though at 0.95 only 0.05 of the scale is left above the top result and at 0.60 there is 0.40. Under the new model the top five results of every acceptance query sat between 0.66 and 0.98, so leads near the top of the scale are small. A margin that shrinks with the headroom can be read the same way at 0.95 and at 0.60; a fixed one cannot.

It ships with enabled: false, an exact no-op, and with a min_drop the configuration calls a placeholder. Next to it is a harness of 35 known-absent queries, "each with a written reason the topic is absent from that corpus." The commit's last line: "The threshold decision waits on measuring react, cpython and uv."

09

Coupled Constants and the Rank-Based Rule That Survives a Swap

The floor had a second problem, older than the swap and independent of any scale. While the banner is up, pointer mode strips result bodies. A per-result override, pointer_mode_retain, lets a result keep a bounded excerpt of its body if its own ce_relevance clears a floor. That floor was 0.30, the same scalar as the banner trigger.

Follow the case the override exists for. The banner fires through the score signal only when the top result scores below 0.30. The top result keeps its body only if it scores at least 0.30. Both cannot be true at once. The commit that found it calls this "a structural deadlock, not a coincidence", and it holds "on any reranker or corpus".

It was measured on a live query: rank 1 correct, ce_relevance 0.024, banner fired, zero code delivered, and the calling agent pushed into a second fetch round trip for an answer it had already been handed. According to the commit, the 0.024 was reproduced by running the previous model, bge-reranker-base, against the same Django index, on the content of Django's Query class. The measured instance belongs to the old model, which put correct answers in the 0.02 range as readily as it put QuerySet.get at 0.0275. Under the new model's compressed range the trigger does not fire, so the deadlock is dormant rather than gone. The first model or corpus that drives a correct top result under 0.30 brings it back.

Analogy: the exit that unlocks only when there is no fire

Picture a building whose emergency exit unlocks only while the smoke reading is below the level that sounds the evacuation alarm. Every drill without smoke passes: the door opens. The one situation the door exists for, alarm sounding, is the one in which it stays locked. No choice of threshold fixes it, because both conditions read the same number from opposite sides of the same line.

The fix is a rule about rank. retain_rank1_always: true keeps the top result's excerpt whatever its score. The commit gives the reason: "Rank-based, not score-based, so it needs no calibrated absolute number and survives any future reranker swap." It extends an older rule in the result floor, "the single strongest result is always kept so the low-confidence banner always leads a best guess", from staying in the list to keeping its body.

The same commit also lowered the floor for ranks 2 and below from 0.30 to 0.15, half the banner value. Twelve minutes later a follow-up put it back at 0.30, because "the change substituted one uncalibrated constant for another." Nothing in the distribution study favoured 0.15 over 0.30; it was half of the old value and nothing more.

The rank rule guarantees that the best guess gets delivered; whether the best guess is any good is a question it cannot answer. Telling a query with no answer apart from a query with a bad ranking still needs a signal on an absolute scale, which is why the banner's score half is waiting on measurements rather than being replaced by a rank rule.

Every constant that reads the reranker's scale

After the swap, vectr's configuration holds six settings that read the cross-encoder's output. Here they are, with what the published numbers say about each:

SettingValueWhat it doesAfter the swap
min_top_relevance0.30Raises the low-confidence flag when the top score is below itNever fired in 65 measurements; the 1.12.0 changelog calls the constant stale
ce_override_min_relevance0.70Silences the vocabulary flag when the top score is at least thisSet under bge, where on the acceptance fixture misses topped out around 0.04-0.19 and matches started around 0.72. Under gte, known-absent queries reach 0.894, so where the vocabulary flag fires on such a query, the override now silences a correct warning. Inferred from the ranges; not measured
pointer_mode_retain.min_relevance0.30Ranks 2 and below keep an excerpt in a flagged set if at or above itEvery published gte score is above it. Left at 0.30 until separation evidence exists
result_floor.min_relevance0.02Drops results scored below itWritten for absent-topic results that rendered at "~0.000" under the old model. Lowest published gte score: 0.515
relative_cliff0.30, 0.50Rank-1 floor and headroom share for the relative-gap testUncalibrated; enabled: false
retain_rank1_alwaystrueTop result always keeps its excerptRank-based; nothing to recalibrate

Published gte scores cover each query's top result in the distribution study and the top five for 44 acceptance queries; scores further down the list are not published.

Only the last row survives a swap by construction. The other five are numbers on a scale the model defines, so each one needs the per-class distributions measured again under the new model before its value means anything. Two of the five already look wrong on the published ranges: the banner floor never fires, and the override sits below scores that known-absent queries reach.

Part 05
The Procedure
10

A Checklist for the Next Reranker Swap

  1. Find the knob the evaluated queries read. Check which settings the evaluation's code path actually reaches. A knob no case reaches sweeps flat and looks robust.
  2. Compare models at matched settings first. The shipped 35-to-43 mixes a model change with a pool change. Measure both models at one pool depth before moving any knob, so the model's effect is not mixed with the setting's.
  3. Sweep the quality knob and read the whole curve. Where it is non-monotonic, find the cases that flip and the mechanism: a test with several acceptable answers, or a final score that depends on pool size.
  4. Choose from a plateau and say what it cost. Write the selection rule down before the sweep runs. Prefer a setting whose neighbours agree over a peak between dips, take the cheapest such setting if latency matters, and record the named cases the choice gave up (here F30) and the unmeasured points that could change it (here 50).
  5. Separate cost-only knobs. Confirm they leave rankings identical, tune them per device, and leave unmeasured devices at the library default.
  6. Inventory every constant on the model's score scale. Treat each as unvalidated, re-measure per-class score distributions, and use a rank-based rule wherever one does the job.
  7. Ship an unknown as unknown. Where no replacement value is supported, ship the mechanism disabled, together with the harness that will calibrate it.
Close
The Line to Keep

Of the three decisions, the model is the only one a pass count can judge, and it judged it fairly: eight more cases, or nine at a matched pool if the old 35 was measured at 60. The pool depth rests on six cases, where one case is 16.7 points, and on a plateau two measured points wide. The floor went quiet without failing anything. If I kept one habit from this swap, it would be writing the Section 09 inventory before the next model is downloaded, so the list exists before any number on it goes stale.

Three things in this swap are still open, and each has a specific next measurement. Pool 50 decides whether 40 sits on a plateau at all. React, CPython and uv decide whether the banner's score half gets a number or stays off. And F23, the query that exposed the old model, still fails under the new one, with nothing public yet saying where in the pipeline it is lost.

↑ Back to top
11

Sources