A benchmark that can be saturated will be

A fixed benchmark is a target, and targets get hit. The tasks are published, so they enter the next training corpus. The scores crowd the ceiling, so the differences that matter stop being visible. The ranking freezes while the models keep moving, and the number that was the whole point becomes a number nobody can act on. This is not a failure of any particular benchmark. It is what happens to all of them, on a clock, and computer-use agents are moving fast enough that the clock is short.

So we did not build a task list. We built an arena, and the tasks are other people's: real browser work, posted by the person who wants it done, run by two frontier agents in parallel, judged blind by a human. There is no seed set to leak because the corpus does not exist in advance. That is the whole thesis, and the rest of this post is the part that decides whether it is worth anything: what has to be true for a blind human judgment to mean something, and what we do about each one.

If the judge can see the vendor, you are measuring the brand

Published de-anonymization rates for frontier models run above 95% from a bag of words, and simply asking a model who it is works most of the time. Moving a top-five rank costs fewer than a thousand votes. A reader is therefore right to assume that an arena with no blindness machinery is poisoned by default, and to want to see the machinery rather than the promise.

Ours scrubs vendor names at the database write, not at the render. That distinction is the entire lesson: we applied it in the shared runner loop first, and one lane did not go through the shared loop, so it wrote its own summary into three judge-visible fields verbatim. A guarantee applied where a caller can route around it is not a guarantee. Blindness is also a property of the (battle, viewer) pair rather than of the battle, because the first vote resolves a battle while two more judges are still expected. A battle-level rule would have handed judges two and three the model names before they labelled anything, and their labels would have measured brand recognition instead of agreement, which is the one number the redundancy sample exists to produce.

And a battle whose own visible text names its model was never blind, so its votes do not count. Both final answers and every step's reasoning are scanned at the moment the battle becomes votable, which is before the first judge can arrive rather than before a delivery. The scan fails closed: an unscannable battle is recorded as leaked.

Equal odds at the next battle is not fairness

This one sounds wrong and is the most interesting thing we changed. Matchmaking was a uniform random draw over pairs, which is the answer everyone reaches for. It gives a model rostered this morning exactly the same share of the next battle as one with three hundred battles behind it, and that is the problem: a rating's interval scales as one over the square root of its battle count, so the newcomer stays unrankable for as long as uniform sampling takes to drag it over the floor, while the incumbent keeps collecting battles that narrow an interval nobody is waiting on.

Equal chance of being picked and equal chance of being ranked are different properties. Only the second is what an entrant is actually asking for. So the draw is weighted by remaining uncertainty, one over the square root of measured battles plus 4, which has the useful property of being self-cancelling: under that allocation every agent converges to the same count from any starting position, so the tilt is temporary by construction rather than a handicap somebody has to remember to remove. Opponents are additionally weighted by how informative the pairing would be and by how under-covered that specific pair is, because a Bradley-Terry fit reads the graph's edges and an unplayed edge is an unidentified comparison.

Three guardrails keep that from becoming its own kind of thumb on the scale. A flat 20% of every draw is uniform, so no agent and no pairing can ever reach probability zero. There is no per-agent constant anywhere in the sampler, so every asymmetry it produces is a function of what has been measured and nothing else. And the selection function takes no caller input at all: not the submitter, not the prompt, not the task, not the time of day. There is nothing for a person to shape.

A ranking should be recomputable, not accumulated

Running Elo is a convenience and not an estimator. It depends on the order the votes arrived in, so an agent whose losses happened to land early sits permanently below one whose identical losses landed late. It needs a seed for every new agent, which asserts the agent is exactly average before anyone has looked. And it has no memory of which vote moved it, so a retraction can never be undone.

The published number is a maximum-likelihood Bradley-Terry fit over every admissible comparison, refitted from scratch. It is order-independent, it needs no seed, and a retraction or a newly untrusted judge takes effect on the next refresh rather than being carried forever. Standard errors cluster on the judge, because a corpus where one person cast most of the labels does not contain as many independent facts as it contains votes. And below more judges than parameters no interval is published at all, which is the honest output rather than a narrow one computed from a corpus that cannot support it. The live ticker is still shown, under its own name, labelled as a ticker, with a step that decays from 48 to 12 as an agent accumulates battles so that a model is not penalised for the date it was added.

Being stricter is not the same as being right

The best correction we have shipped was to a rule of our own. We decided that a battle judged only by the person who submitted it should not rate anybody. Obviously defensible, and it discarded 39 of 44 judged battles in the live corpus, leaving three of thirteen models with a score. It replaced a ranking built on forty-four blind human labels with one built on two.

The error was in the threat model. Self-judging is dangerous when a judge can aim, and here they cannot: identities are withheld from every viewer until a battle resolves, so a submitter's own vote is blind by construction, and the pair is drawn from measurement history so choosing the task cannot target a model either. What self-judging actually carries is annotator concentration, which is a variance problem with a correct treatment (cluster the errors on the judge) rather than a reason to delete the evidence. The rule now says independent labels win where they exist, the submitter's blind label decides where none has arrived yet, and the share of the ranking resting on self-judged battles is published. It tightens by itself as the judge pool grows, with no threshold to tune.

Every number carries the rule that produced it

“Success rate 71%” is not a result. Success according to whom, over which runs, counting a provider outage as whose fault? Most arguments about a benchmark turn out to be arguments about a denominator, so the denominator ships with the number: 57 measures, each published with what is counted, what it is counted over, which direction is good (or an explicit statement that neither is), and what it does not establish. Machine-readable, with the definitions in the same payload as the values, so a metric cannot quietly change meaning without its description changing in the same response.

Every one of those measures is a proxy and each says so. There is no oracle telling us a click was correct, so “clicks that did nothing” means a click whose own observation reported an error or an unchanged page. That is a real signal and it is not the same thing as a miss, and the difference is printed next to the number rather than left for a reader to discover.

Where this is still weak

A thesis post with no open problems is marketing. Ours, today: the ranking rests on a small number of distinct judges, and most of the corpus is still labelled by the people who submitted it, so the intervals are wide and we publish that rather than a headline number. Judge calibration is computed but does not yet travel with the exported data, so a buyer cannot separate a calibrated annotator's label from an uncalibrated one. Agreement statistics need 3 annotators on an item to exist at all, and only a sampled fraction of battles gets them. None of that is fixed by wording, and all of it is listed, with the mechanism that would earn each claim, on the pages below.

The reason to believe any of this is not that it is well argued. It is that every sentence of it is wired to a file: the mission page pairs each belief with the mechanism that enforces it, and a belief whose mechanism is deleted stops rendering instead of quietly becoming false. npm run verify:claims fails the build when that happens.

Mission · Governance · Every metric, defined · The standings