Real-world evaluations of computer-use agents.
57 measures in 10 families, each with its definition, denominator and direction. They describe recorded behaviour in Computer Use Arena, not task correctness; KnowledgeBench results are separate. Also published as JSON.
Did it do the job, and does it know whether it did?
How runs ended, by the agent's own report.
Runs whose terminal `done` call reported success.
Over runs that produced a trajectory (harness errors excluded)
Runs whose terminal `done` call reported failure, plus runs that stopped without calling done.
Over runs that produced a trajectory
Runs where the agent reported the objective does not exist on this environment: the control is gone, a login wall bars it.
Over runs that produced a trajectory
Runs that hit the shared wall-clock limit before finishing.
Over runs that produced a trajectory
Runs that produced no trajectory at all: an upstream outage, a rejected credential, a withdrawn model.
Over all runs, including ones with no trajectory
Where the agent's own verdict is checkable against something else: the opponent on the same task, or a judge who read the answer.
Runs where the agent declared the task infeasible while its opponent COMPLETED the identical task in the same battle.
Over runs declared infeasible whose opponent produced a terminal verdict
How the run stopped: on purpose, or because something ran out.
Runs whose final action was an explicit `done` call rather than being cut off.
Over runs that produced a trajectory
How direct was the path, and how much of it was wasted motion?
Actions spent, and how that compares to the other agent on the same task.
Actions executed in a run, including the terminal one.
Over runs that produced a trajectory
When something went wrong, what did it do next?
How often actions failed at all.
Actions whose observation came back as an error.
Over actions
Does it hit what it aims at? The pixel layer, which only a computer-use arena can see.
Where clicks land, how they cluster, and whether they repeat.
Clicks whose own observation reported an error or an unchanged page.
Over click actions
Clicks that did nothing figures on request: founders@coasty.ai
Clicks within 24px of any viewport boundary.
Over click actions
Clicks at the frame edge figures on request: founders@coasty.ai
How fast, how steady, and how bad is the tail?
Wall clock, including the tails a median hides.
Wall-clock milliseconds from run start to terminal state.
Over runs that produced a trajectory
What does it spend, and what does a SUCCESS cost?
Tokens in and out.
Tokens sent upstream over a run.
Over runs that produced a trajectory
How much does it think out loud, and per what?
Text the model wrote before acting.
Characters of model-written reasoning recorded per action.
Over actions
What did it hand back?
Files produced for the user.
Runs that wrote a downloadable artifact for the user.
Over runs that produced a trajectory
Who beats whom, and is it consistent?
The pairwise grid and what it implies.
Wins, losses and ties against each individual opponent, from ranked battles only.
Over ranked battles against that opponent
What did the people who watched it say went wrong?
Which failure a judge attached to it, and how often.
Share of the agent's flaw annotations falling under each of the seven flaw labels.
Over flaw annotations on this agent's runs
Runs that reported success and then collected a human 'wrong or made up' or 'didn't finish the job' label.
Over runs that reported success AND received at least one flaw annotation
Runs that consumed their entire step budget.
Over runs that produced a trajectory and recorded a budget
Runs that were granted at least one continuation by the battle's creator.
Over runs that produced a trajectory
Actions executed, counting only runs that reported success.
Over runs that reported success
Geometric mean of (this agent's actions / the opponent's actions) across battles where BOTH sides completed the identical task.
Over battles where both sides completed
Actions that repeated, changed nothing, or went back where it came from.
Actions byte-identical to the action immediately before them.
Over actions after the first, per run
Actions whose observation was identical to the previous step's, once the tab marker is stripped.
Over actions after the first
Actions the browser explicitly reported as having moved nothing: scrolling past the end of a document, scrolling a panel that does not scroll.
Over actions
Actions after which the page URL was unchanged from the previous step.
Over actions after the first, on steps where both URLs were recorded
Steps landing on a URL the run had already visited and since left.
Over steps with a recorded URL
How much of the page and the site it touched, and how it split looking from doing.
Read-only actions (screenshot, read_page, hover) as a share of all actions that were either read-only or world-changing.
Over read-only plus world-changing actions (terminal actions excluded)
Distinct page URLs observed across a run.
Over runs with at least one recorded URL
Navigations to an origin other than the first one the run observed.
Over navigate actions
Shannon entropy of the run's verb distribution, normalized by log2 of the twelve verbs AVAILABLE.
Over runs using at least two distinct verbs
Share of each of the twelve verbs across all of the agent's actions.
Over all actions
Longest run of consecutive failing actions within a run.
Over runs that produced a trajectory
Whether the next action was different, and whether it worked.
Failing actions whose NEXT action did not also fail.
Over failing actions that had a next action
Failing actions immediately followed by the byte-identical action: the same call, expecting a different answer.
Over failing actions that had a next action
Clicks at an (x, y) already clicked earlier in the same run.
Over click actions
Clicked the same pixel again figures on request: founders@coasty.ai
Mean pairwise distance between a run's clicks, divided by the viewport diagonal.
Over runs with at least two clicks
Click spread figures on request: founders@coasty.ai
Drags, modifiers, and whether a path is a path or two endpoints.
Coordinate pairs supplied per drag action.
Over drag actions
Points per drag figures on request: founders@coasty.ai
Drags supplied with two points or fewer.
Over drag actions
Endpoint-only drags figures on request: founders@coasty.ai
Clicks holding Alt, Control, Meta or Shift.
Over click actions
Modified clicks figures on request: founders@coasty.ai
How it types: bursts or characters, keys or strings.
Characters supplied per `type` action.
Over type actions
Characters per typing action figures on request: founders@coasty.ai
`press_key` actions as a share of all text-entry actions (type plus press_key).
Over type plus press_key actions
Keys vs strings figures on request: founders@coasty.ai
Elapsed milliseconds when the first action resolved.
Over runs with at least one step
Median gap between consecutive step timestamps within a run.
Over runs with at least two steps
Whether the pace is even or spiky.
Interquartile range of run durations divided by their median.
Over runs that produced a trajectory
Tokens generated over a run, including reasoning where the provider bills it.
Over runs that produced a trajectory
Spend per action, and spend per finished task: the number that decides whether you can run it.
A run's output tokens divided by its actions.
Over runs with at least one step
A run's input tokens divided by its actions.
Over runs with at least one step
Total tokens across all of an agent's runs divided by the number that reported success.
Over runs that reported success
Actions recorded with no reasoning text at all.
Over actions
Shape of the final answer: the live confound on every human preference label.
Characters in the final answer handed to the judge.
Over runs with a final answer
Runs that reported success with a final answer under 20 characters.
Over runs that reported success
Upsets, ties, and how separable its battles are.
Wins against opponents whose published rating was higher at the time of the fit.
Over ranked battles against higher-rated opponents
Ranked battles judged a tie or both-bad.
Over ranked battles
Runs that collected at least one flaw annotation.
Over runs that were annotated at all
The low-effort labels judges reach for.
Share of the agent's stamps across the five run-level character labels.
Over stamps on this agent's runs
Strength and separability of the labels its battles produced.
Mean judge-reported strength (barely / clearly / not close, as 1-3) on battles this agent won.
Over wins where the judge tapped the optional strength chip