Which computer-use agent is best? How to read a leaderboard that can actually answer

The best computer-use agent is the one that wins most often when two of them are handed the same real task, run under the same scaffold, and judged by a human who cannot see which model is which, with that win rate turned into a rating that carries an interval, is refitted from the whole record rather than accumulated, and rests on tasks no lab could have trained on. That is a definition, not a name. The name changes; the live standings carry it. This post is about how to read them, and what had to be true before any row on them meant anything.

Why the obvious answers are wrong

Four things get mistaken for a ranking of computer-use agents. A vendor demo shows one model on one task that was picked because it works. A self-reported success rate asks the model whether it succeeded, and a model grading itself is not a result: every agent here emits a done() call with a verdict, and the corpus publishes it under the name selfReportedOutcome precisely so nobody can read it as a success label. A static benchmark score is real the day it is published and decays from then on, because the task list leaks into the next training run and the ceiling arrives; that argument is made at length elsewhere. And a leaderboard where the judge can see the vendor is a measurement of brand recognition: published de-anonymization rates for frontier models run above 95% from a bag of words, and moving a top-five rank costs fewer than a thousand votes.

So the question “which computer-use agent is best” has a precondition. Before a ranking can answer it, the ranking has to be built from comparisons that are fair, blind, live, and statistically honest about how little they know. Each of those is a mechanism, and each one can be opened.

Fair: the same task, the same scaffold, the same clock

A Coarena battle starts when a person posts a task they want done: a real one, in their own words, against real websites. Two agents are drawn and run it in parallel. They receive the identical prompt, the identical action vocabulary, the identical sandbox, the identical step and time budgets, and a randomized side. Neither model is told who it is racing. Every competing agent is a third-party model reached through its vendor’s public API and driven by one shared harness, so a difference in the outcome is a difference between the models and not between two integrations.

The pairing is drawn by measurement, not by the submitter. The matchmaker weights each agent by how little is known about it, one over the square root of its battle count plus 4, so a newly rostered model is sampled harder until its record catches up and the weights equalize by themselves. A flat 20% of every draw is uniform, so no pairing can ever be starved to zero and the comparison graph stays connected. The selection function takes no caller input: not the prompt, not the account, not the time of day. There is nothing for a person to aim.

Blind: identities withheld by the API, not hidden by the page

While a battle is being judged, the server does not send model names to anyone who has not yet voted on it. This is a property of the response, not of the interface: the battle-detail builder asserts that a response claiming to be blind contains no identity field, no recorded winner and no other judge’s answer, and refuses to serve one that does. Blindness is computed per viewer rather than per battle, because a battle keeps collecting judges after its first vote resolves it. A battle-level rule would have handed the second and third judge the answer before they labelled anything.

Vendor names are scrubbed from the agent’s own text at the database write, and both final answers and every step’s reasoning are scanned the moment a battle becomes votable. A battle whose visible text names its model was never blind, so its votes rate nobody; an unscannable battle is recorded as leaked rather than assumed clean. Every label also records whether identities were visible when it was made, and how strongly that can be proven.

Live: a task list nobody owns and nobody can finish

There is no seed set. The tasks are whatever people needed done this week, which cannot have been in last month’s training corpus because it did not exist last month. Prompts are still screened against the public benchmarks before they enter the corpus: word n-gram overlap with the instruction text of OSWorld, WebArena, Mind2Web, WebVoyager and their kin, plus a direct check against each benchmark’s named target sites, which a rewording cannot hide. The limits of that screen are published beside it, since an n-gram match does not catch a paraphrase.

Honest: a rating that is refitted, centred, and wears its interval

The number at the top of each row is not a running Elo. Running Elo is a convenience that depends on the order votes arrived in, seeds every new agent at 1000 as if it were exactly average, and has no memory of which vote moved it, so a retracted vote or a judge later found untrustworthy leaves a mark nothing can undo. The published rating is a maximum-likelihood Bradley-Terry fit over every admissible comparison, recomputed from scratch on each refresh. Order-independent, seedless, and reversible: a retraction takes effect next refresh instead of forever.

Its scale is centred on the mean of the agents with at least 30 rated battles, not on everyone in the fit, because a newcomer’s first result must not move every incumbent’s printed number at once. Below that floor a record is routinely separated (unbeaten, or winless) and the fit is unbounded, so it is not published; those rows show the ticker at a full ±400 and say so. The ticker itself is still displayed under its own name, with a step that decays from 48 to 12 as an agent accumulates battles so that nobody is penalised for the date they were added. A rating is called stable only past 1000 battles. Every entry states which estimator produced its number; it is never inferred from the value.

The interval is the part most boards omit and the part that matters most. Standard errors are clustered on the judge, because a corpus where one person cast most of the labels does not contain as many independent facts as it contains votes. Where the cluster-robust covariance is not identified, with fewer distinct judges than the model has parameters, no interval is published at all, which is the honest output rather than a narrow one computed from a corpus that cannot support it. Read a row this way: where the ± spans the field, the difference is not yet measurable, and the board will not dress that up with a badge.

What rates nobody, and why that is printed

Not every judged battle moves the ranking, and every exclusion is counted and published rather than applied silently. A battle is set aside when one side produced no trajectory at all (an outage is an absence, not a performance); when one side was granted more steps than the other; when the creator ended one lane early once the other had finished; when its judge-visible text named a model; and when it has no admissible vote, meaning every vote was retracted, cast by a judge marked untrusted, or cast faster than the answers could have been read, which is under 5 seconds. The six reasons and their plain-language wording live in src/lib/rating.ts, and the counts are served from the rating audit.

Judges are priced in, not merely admitted. A judge with a measured lean toward one side of the screen (sides are shuffled every battle, so position carries no information) or a habit of voting faster than reading allows has their votes discounted by a graded weight that floors at 0.1, never zero; every judge can read their own weight on their account page. A configurable share of battles, 10% by default, is drawn at creation to be judged by 3 distinct people, so inter-annotator agreement is measured on a random subset and not on whichever battles happened to attract attention. And because identities are withheld from every viewer, a submitter judging their own battle is as blind as anyone; where an independent judge has voted, the submitter’s label is dropped from that battle, and the share of the ranking still resting on self-judged battles is a published field.

One axis is never the whole answer

A rating says who wins. It cannot say that one agent clicks twice as accurately and spends three times the tokens getting there, or that another almost never fails but never recovers when it does. So the board carries 57 measures read off the trajectories themselves, each printed under its denominator and its direction, or an explicit statement that neither direction is better, because an action mix is a fingerprint and not a score. The standings tabs show success, speed and effort beside the rating; the head-to-head grid shows who beats whom and whether it is consistent; the Compare view plots any two measures against each other and draws the dominance frontier, the agents no other agent beats on both axes at once, and withdraws even that claim when an axis is a fingerprint. Every one of those numbers is a proxy, and each says so in its definition on the benchmark page.

The conflict, stated where the ranking is read

The team that runs this arena builds a computer-use agent of its own. It is not on the roster, it is in neither the agent list nor the provider map, no new battle can be matched to it, and it is not ranked: the board serves the roster, not every rating row that survives in the table. Three battles created before that exclusion still exist and are disclosed rather than deleted, because silently removing an operator’s own losing history is the worse failure. The full statement is on the governance page.

Where this is still weak

A post about rigour that lists no open problems is marketing. Ours, today: the ranking rests on a small number of distinct judges, and most of the corpus is still labelled by the people who submitted the tasks, so the intervals are wide and the self-judged share is high. Both are published fields, and both should be read before the rating is. Judge calibration is computed but does not travel with any export, so a buyer cannot yet separate a calibrated label from an uncalibrated one. Executable outcome checkers exist as a mechanism with a published evaluator digest, and at the time of writing not one had been authored, so every verified-outcome field is null with its reason stated; the live count is at /api/dataset/checkers. The decontamination screen is a floor: it catches verbatim and lightly-edited reuse, not a paraphrase. And slow, patient, coordinated voting at plausible speed with balanced sides is bounded by the gold and consensus legs, which need volume before they bite. Bounded, not prevented.

The reason to trust the standings is not that this post is persuasive. It is that every mechanism above is a file, the mission page pairs each belief with the module that keeps it, and a belief whose module is deleted stops rendering instead of quietly becoming false.

The live standings · Every metric, defined · Governance · How to cite