Sequential testing ·
Why you can’t look at a leaderboard twice
Two models are exactly as good as each other. Watch what happens to the claim “A beats B” as you keep checking.
A confidence interval is a promise about one look. Read it once, at a sample size you fixed before you started, and it is wrong 5% of the time. That is the whole deal, and it is a good one.
A leaderboard breaks it immediately. Nobody reads a leaderboard once. It is read every time someone refreshes the page, and it is recomputed every time a new vote lands. Each of those readings is another chance for noise to look like a result, and the 5% was only ever priced for one. Drag the slider.
At a single look the interval does what it promised: about one board in twenty names a winner between two models that are genuinely tied. By forty looks it is closer to one in three. Nothing about the models changed and no new bias crept in — the only thing that changed is how often somebody looked.
This is not a subtle effect and it is not a hypothetical one. Public model leaderboards are read continuously, by exactly the people deciding which model to ship.
What to use instead
The fix is to stop buying a promise about one look and buy one about all looks. A confidence sequence is an interval at every sample size at once, built so that the probability it is ever wrong — across the whole infinite sequence of looks — stays under 5%. You may check it as often as you like, stop when you like, and decide to keep going because of what you just saw. The guarantee does not care.
That is the teal line above, and it is wider than the dashed one. That width is the price, and it is the right thing to pay: a bound that is honest about having been read a hundred times is more useful than a tight one that quietly stopped being true after the second look.
Try it on your own numbers
If you have pairwise results of your own, the leaderboard calculator will put an anytime-valid bound on every claim in them, and tell you how many votes an adversary would have to corrupt before each claim fell over.
Open the calculator Rank Confidence Sequences Certified Corruption Budgets