Leaderboard calculator

Anytime-valid bounds and corruption budgets for your own pairwise results

Paste one comparison per line — model A, model B, wins for A, wins for B. Everything is computed in your browser; nothing is uploaded.

What it is doing

Each claim gets a confidence sequence rather than a confidence interval: a bound that is valid at every sample size simultaneously, so you may look at it as often as you like, stop when you like, and keep the guarantee. Here that is a sub-Gaussian normal-mixture sequence on the leader’s win rate. With hold all claims at once ticked, the level is split across the pairs, so the whole table is right together rather than each row being right on its own.

The budgets answer a different question: not “is this claim significant” but how much tampering would it take to overturn it. Flipping a vote that was already cast moves the count twice as far as injecting a fresh one, which is why the flip budget is consistently about half the forge budget.

This is a demonstration of the methods from the two papers below, not the papers’ own experiment code — it takes the simplest version of each so it can run live in a browser. For the full treatment, including rankings over all pairs at once and adversaries who adapt to what they see, read the papers.

Rank Confidence Sequences Certified Corruption Budgets The note behind it