Anatomy of a computer-use benchmark: one scaffold, a real desktop, and every measure defined

A computer-use benchmark is not a list of tasks and a score. It is the whole apparatus the score was read off: which actions the agent was allowed, on what screen, for how many steps and minutes, what was written down after each one, and which rule turned the record into a number. Most published evals describe the tasks and leave the rest implicit. This post is the rest: every part of the scaffold a Coarena battle runs on, in enough detail to reproduce it, with every figure imported from the code that enforces it.

One scaffold, shared by both sides of every battle

Two agents race the identical task. They are given the same prompt, the same vocabulary of actions, the same sandbox image, the same 1280×800 screen, the same step and time budgets, and a randomized left-or-right side. The action space is drawn once per battle, either the browser alone or the browser plus a desktop, and both lanes inherit it, so the two sides of a battle are never given different vocabularies. Which one a battle ran under is stamped on the row and delivered with every record, because a trajectory that does not say what its agent was allowed to do is uninterpretable.

Every competitor is a third-party model reached through its vendor’s public API (Anthropic, OpenAI, Google, Meta, xAI, and open-weight models hosted by Fireworks AI), driven by one harness. A lane is told nothing about its opponent. Agents run logged out; no task is run from an authenticated session on a third party’s site, and no credentials are ever supplied to one.

The action vocabulary

The browser space has 12 verbs: navigate, click (with its button and held modifiers recorded as required fields, so a record can never be silent about which button), hover, drag, type, press a key, scroll, switch tab, read the page’s text, take a screenshot, deliver a file, and finish with a verdict of success, failure or infeasible. A drag carries its whole path, up to 50 points, never just its endpoints: the intermediate points are the gesture for a slider, a canvas stroke or a reorder, and once they are flattened nothing downstream can recover or even detect the loss. Every verb has a declared mapping onto the shipped lab computer-use APIs, or a declared reason it has none; a verb added without a row in that table fails to compile.

A desktop battle adds 8 OS-level verbs (screen capture, pointer move, click, drag, key, type, launch an application, run a command) on the same coordinate space, because the X display and the viewport are the same 1280×800. They are separate names rather than a flag on the browser verbs on purpose: a screen pixel and a page pixel are different references, and the verb says which one a coordinate belongs to.

The desktop, and why there is no terminal on it

The desktop is a real one, Chromium as a window on an X display, with exactly 3 launchable applications named from a closed server-side list: xcalc, xedit and thunar. The model names one; our code resolves the name to an argv. No model-authored text ever reaches a shell. There is deliberately no terminal emulator, no spreadsheet and no paint program: a model that can open a terminal and type into it has arbitrary shell, which defeats every other control at once, and its trajectory would stop being comparable with a lane that only clicked.

What it has instead is a command verb that is not a terminal. The model supplies a program name and a list of arguments; the pieces reach execve as a list and nothing ever parses them, so a semicolon in an argument is a character in an argument. The program must be one of 80 on an allowlist checked twice, here and again inside the sandbox helper; paths are confined to the agent’s own working directory with .. refused rather than resolved, and loopback and link-local addresses are refused in any argument. One command may run for 45 seconds and return 8,000 characters, and one run may issue 80 of them, because a run that spent all its steps shelling out is not a computer-use trajectory and the corpus should not carry it as one. The list is generous by design (interpreters are on it, and say so), because the boundary is the ephemeral sandbox, not the list, and a policy that pretended otherwise would be theatre. Every rule is pure and provable without a sandbox; npm run verify:shell proves it.

Budgets sized against each other

A run gets 100 actions and 20 minutes, and the two numbers were set together, because whichever is smaller is the only limit that exists. The step budget was raised when the measured corpus showed runs dying at the ceiling while visibly working; the clock was raised with it, because raising steps alone would have converted “step limit” deaths into “out of time” deaths and changed nothing a user could see. The example tasks are written to need roughly seventy to a hundred and twenty actions: fetch figures from two or three sites, work them through the calculator a digit at a time, type up a structured document, save it, verify it in the file manager. The budget clears that design band with headroom rather than truncating it.

When a run hits its limit it pauses rather than dies: the sandbox is held, the page state is intact, and the battle’s creator is offered 10 more steps against a visible countdown, up to a ceiling of 200. A battle where one side was granted more steps than the other is excluded from the ranking, because the two lanes no longer ran under the same budget. Tasks may carry files, up to 10 per task at 50 MB each, which land in both sandboxes identically.

What is written down at every step

The step is the unit of the corpus, and each one records the action with its full parameters; the model’s reasoning text where the run emitted one; the observation, which is the sentence the executor fed back, and which for a failed action begins with a fixed error sentinel so that “did this step fail” is a classification and not a regex over prose; the raw screen capture and, where one could be produced, the masked capture with a redaction receipt stating which detector ran, on what basis, and how many rectangles were painted and proven in the delivered bytes; the page URL the action landed on, read after it resolved so it matches the frame; a gzipped accessibility-tree snapshot, so the step stays verifiable after the site changes and the pixels no longer describe anything that exists; an absolute wall-clock timestamp; and the identity of the tab the action landed on, since a click that opened a new tab looks identical in a record that does not say so.

A command step records the command’s exit code and its output in the observation, and the replay shows exactly that where the screenshot would be, because a command runs headless, the frame after it is the frame before it, and a picture of nothing changing is not evidence of anything.

57 measures, in 10 families, each under its rule

“Success rate 71%” is not a result until it says success according to whom, over which runs, counting a provider outage as whose fault. Most arguments about a benchmark are arguments about a denominator, so the denominator ships with the number. Every measure is published with what is counted, what it is counted over, which direction is good (or an explicit statement that neither is), and what it does not establish, in the same payload as the values, so a metric cannot change meaning without its description changing in the same response. The families, and the question each one answers:

• OutcomeDid it do the job, and does it know whether it did?
• RouteHow direct was the path, and how much of it was wasted motion?
• RecoveryWhen something went wrong, what did it do next?
• PrecisionDoes it hit what it aims at? The pixel layer, which only a computer-use arena can see.
• TempoHow fast, how steady, and how bad is the tail?
• CostWhat does it spend, and what does a SUCCESS cost?
• ExpressionHow much does it think out loud, and per what?
• OutputWhat did it hand back?
• Head to headWho beats whom, and is it consistent?
• Human judgmentWhat did the people who watched it say went wrong?

Some of these rank and some deliberately do not. An action mix, a click spread and a flaw profile are fingerprints: their job is to answer the question a leaderboard cannot, which is what two agents with the same win rate are doing differently, and no chart draws a direction under them. Every measure is a proxy and each one says so. There is no oracle telling us a click was correct, so “clicks that did nothing” means a click whose own observation reported an error or an unchanged page: a real signal, and not the same thing as a miss. The full catalog, with the live values where they exist, is the benchmark page.

The human layer

After the replay, one question: which one did it better. One tap, blind. The judge may then mark the steps where the loser went wrong, which turns a pairwise preference into a step-level label, and may attach a flaw or a character stamp to a run. A share of battles, 10% by default, is drawn at creation to be judged by 3 distinct people so agreement is measured on an unbiased subset; judges are scored against known outcomes and can see their own calibration; and a judge’s measured behaviour, a lean toward one side of the screen or a habit of voting faster than reading allows, discounts their votes by a graded weight that is published in the rating audit.

Verified outcomes, and their coverage

A task may carry an executable checker: sets of assertions over the run’s recorded end state (final URL, final answer, delivered files, the last accessibility snapshot). It is executed by us, and the exact function that ran ships with the data along with a SHA-256 of the evaluator, so a buyer can prove the checker they were sent is the one that decided the label. A verdict is null whenever it cannot be stood behind, with the reason published; null is never read as failure. Assertions that cannot fail are rejected. And the honest part: at the time of writing, zero checkers had been authored, so every delivered verified outcome was null with reason no_evaluator_authored. The mechanism is real; the coverage is what /api/dataset/checkers says it is.

Where this is still weak

The desktop’s frames are whole-screen captures with no DOM behind them, so no mask is attempted on any of them and none is claimed; they are shown inside the arena to the submitter and to the judges a battle is offered to, and delivered to nobody. The command allowlist includes interpreters, which can open a socket, and the sandbox is the boundary that bounds that; the list is not, and says so. Verified outcomes have no coverage yet. A click ring drawn on a desktop frame for a browser verb sits above its real target by the height of the browser chrome, because the window rectangle is not measured anywhere yet; that is written down rather than approximated. None of this is fixed by wording, and all of it is listed with the file that would earn each claim on the governance page.

Every metric, defined · The standings · The dataset · Mission