Metrics API
Every metric, per model and use case, with its definition, at /api/metrics.
- Use case keys
- 13
- Metrics
- 101
- Families
- 14
Request
# Definitions, then valuesGET /api/metrics # Values only: no definitionsGET /api/metrics?values=1 # Overall only: no byCategoryGET /api/metrics?categories=0 # The Phone board's metricsGET /api/metrics?board=phoneServed with cache-control: public, max-age=60, stale-while-revalidate=300, or no-store when the report is degraded; the report behind it is recomputed at most every five minutes. Readable from any origin. A board that does not exist is a 400.
Response
| Key | Type | Holds |
|---|---|---|
families | array | Each family: the question its metrics answer, with its groups. |
catalog | array | Every definition, in presentation order: id, family, group, label, definition, denominator, unit, shape, better, caveat, scope, sliced, source, nUnit. In the GET /api/metrics?board=phone document, also phone on an entry the Phone board words differently (label, definition, denominator, caveat, and withheld where a phone cannot measure it). |
useCases | array | The slice keys of byCategory, with their labels and definitions. A coded rule over a task’s prompt and start URL decides its slice; the rules are not published. |
agents | array | One entry per public model: agentId, overall (every agent-scope metric, keyed by id) and byCategory[] (category, values). A slice appears for every use case the model ran in the window or has ranked battles in, and leaves out every metric whose sliced is false; one with only ranked battles carries head to head, upsets and ties, with every run metric unmeasured. |
corpus | object | The arena-wide values, keyed by id: every metric whose scope is corpus, facts about the judges and the corpus rather than a model. |
coverage | object | board, battlesScanned, battleWindow, windowed, runsAnalysed: the board and window every n is read from. |
computedAt | number | When the report was computed, in epoch milliseconds. |
degraded | boolean | True when the report could not be computed. Every agent list is then empty. |
Value shapes
| Shape | Fields | Used by |
|---|---|---|
rate | value, ci { low, high }A share of a named denominator, with its 95% Wilson interval. n is the denominator. | 60 metrics |
distribution | median, p10, p90, min, max, iqr, dispersionA spread, read from p10 to p90. n is the sample. | 20 metrics |
scalar | value, ciOne number in the metric’s unit. ci is a resampled 95% interval where the estimate has one, otherwise null. | 16 metrics |
profile | entries[] { key, share, count }Shares over a fixed key list, in the same order for every model. n is the total. | 4 metrics |
matrix | cells[] { opponentId, winShare, battles }Win share per opponent the model met, ties counting half; an opponent it never met has no cell. n is the battles across every cell. | 1 metric |
Every value also carries shape, n, measured, reason, with n counted in the catalog’s nUnit, and underpowered where the metric sets a power floor.
Live excerpt
{"families": […],"catalog": […],"useCases": […],"agents": [{"agentId": "deepseek-v4-1-flash-1017","overall": {…"harness_error_rate": {"shape": "rate","value": 1,"ci": { "low": 0.991104, "high": 1 },"n": 428,"measured": true,"reason": null},…},"byCategory": […]},…],"corpus": { … },"coverage": {"board": "computer","battlesScanned": 1000,"battleWindow": 1000,"windowed": true,"runsAnalysed": 2000},"computedAt": 1794998888788,"degraded": false}agents[0]: 1 of its 96 overall values, as the endpoint prints them. Computed .
Catalog
- Tempo6 of 6
- Cost8 of 8
- Expression2 of 2
- Output3 of 3
- Head to head3 of 3
- Recovery7 of 8
- Label quality4 of 5
- Precision7 of 9
- Judgment3 of 4
- Safe and useful6 of 8
- Outcome11 of 16
- Route13 of 19
- Ask or refuse2 of 5
- RobustnessNone
Outcome16 metrics
Did it do the job, and does it know whether it did?
| Key | Metric | Definition |
|---|---|---|
completion_ | Completed | Runs whose final done call reported success. Over runs that produced a trajectory (harness errors excluded). |
failure_ | Gave up | Runs whose final done call reported failure, plus runs that stopped without calling done. Over runs that produced a trajectory. |
infeasible_ | Called impossible | Runs where the agent reported the objective does not exist on this environment (the control is gone, a login wall bars it). Over runs that produced a trajectory. |
timeout_ | Ran out of clock | Runs that ended on a clock (outcome 'timeout'): the shared wall-clock limit ran out before the run finished, or the model's calls kept outlasting the arena's per-call limit until the call's attempts were spent, with run time still left. Over runs that produced a trajectory. |
harness_ | Ended in error | Runs that ended in an error (outcome 'error') or recorded no outcome, rather than with the agent's own verdict, the clock or the submitter's stop. Infrastructure ended almost all of them: the model's provider (an outage, a rejected request or credential, a withdrawn model) or the arena's own machines (a sandbox that never became ready, a phone lost mid-run, a battle that expired unfinished). So did the computer runs of 2026-10-21 to 2026-10-26 that an operator marked as errors on 2026-10-26 because a call had outlasted the arena's per-call limit, since raised; a run whose calls keep outlasting it now ends on the clock and is counted under timeout_rate. A refusal the provider returned ends a run the same way and is counted here too. Many of these runs took steps before they ended. Over all runs, including the ones counted here. |
false_ | Wrongly called impossible | Runs where the agent declared the task infeasible while its opponent COMPLETED the identical task in the same battle. Over runs declared infeasible whose opponent produced a terminal verdict. |
overclaim_ | Claimed success, judged wrong | Runs that reported success on which a person then left a contradicting label: the flaw 'wrong answer' or 'didn't finish the job', or the stamp 'made it up'. Over runs that reported success AND that a person labelled: a flaw, a stamp or a check tick left by an account (model judges' rows are not counted). |
terminal_ | Stopped on purpose | Runs whose final action was an explicit done call rather than being cut off. Over runs that produced a trajectory. |
budget_ | Used every turn | Runs whose model turns reached their recorded turn budget (runs.model_turns at or above runs.step_budget): the unit the runner enforces, so a lane that queues several actions from one model call is not counted as out of budget while it has turns left. Over runs that produced a trajectory and recorded a budget. |
continuation_ | Asked for more | Runs that were granted at least one continuation by the battle's creator. Over runs that produced a trajectory. |
verification_ | Verification coverage | Share of runs an independent evaluator could decide: the task carries executable assertions, the end state was recorded, and the checker returned success or failure rather than declining. Over runs with a trajectory. |
verified_ | Verified completion | Of runs the agent declared complete and an evaluator could decide, the share the evaluator confirmed against the recorded end state. Over claimed completions with a decided verdict. |
claimed_ | Claimed over verified | Runs the agent declared complete divided by runs an evaluator confirmed, over runs with a decided verdict. 1.0 means every claim was confirmed; 2.0 means half were. Over runs with a decided verdict. |
checks_ | Every check ticked | Of completed runs in battles where the submitter wrote acceptance checks and a judge ticked at least one on either side, the share where every check was ticked for this run. Over completed runs in checked battles. |
check_ | Checks ticked | Mean share of the submitter's acceptance checks a judge ticked for the run, over runs in battles where anyone ticked anything. Each run is a share of its own task's checks first, so a task with two checks and a task with eight weigh the same. Over runs in checked battles. |
checks_ | Checks coverage | Share of runs in a battle where the submitter wrote acceptance checks and a judge ticked at least one on either side. Over runs with a trajectory. |
Route19 metrics
How direct was the path, and how much of it was wasted motion?
| Key | Metric | Definition |
|---|---|---|
steps | Actions taken | Actions executed in a run, including the terminal one. Over runs that produced a trajectory. |
steps_ | Actions to finish | Actions executed, counting only runs that reported success. Over runs that reported success. |
detour_ | Detour factor | Geometric mean of (this agent's actions / the opponent's actions) across battles where BOTH sides completed the identical task. Over battles where both sides completed. |
repeat_ | Repeated itself | Actions byte-identical to the action immediately before them. Over actions after the first, per run. |
no_ | Changed nothing | Actions whose observation was identical to the previous step's, once the tab marker is stripped. Over actions after the first. |
inert_ | Explicitly inert | Actions the browser explicitly reported as having moved nothing: scrolling past the end of a document, scrolling a panel that does not scroll. Over actions in runs offered the browser's scroll: every browser-mode run, and a desktop run from 7 to 9 August 2026 that used it (desktop mode offered the browser's verbs until then). Any other desktop run and every phone run has no action that can report it, and where a slice holds no such run the value is withheld. |
stall_ | Stayed put | Actions after which the page URL was unchanged from the previous step. Over actions after the first, on steps where both URLs were recorded. |
revisit_ | Went back | Steps landing on a URL the run had already visited and since left. Over steps with a recorded URL. |
probe_ | Looking vs doing | Look-class actions (a screenshot, reading the page or its elements, a hover, a pointer move, a phone wait) as a share of all actions that either looked or changed something. Refused tool calls executed nothing and are in neither side. Over look-class plus world-changing actions (finishing actions and refused calls excluded). |
distinct_ | Pages visited | Distinct page URLs observed across a run. Over runs with at least one recorded URL. |
off_ | Left the site | Navigations to an origin other than the first one the run observed. Over navigate actions. |
action_ | Action variety | Shannon entropy of the run's distribution over the ten action classes, normalized by log2 of the number of classes its mode OFFERS: ten in a browser, nine on a desktop (no scroll verb), nine on a phone (no script or shell). A desktop run from 7 to 9 August 2026 that used the browser's scroll, which desktop mode then still offered, is normalized by ten. Over runs using at least two distinct action classes. |
verb_ | Action mix | Share of each kind of action across the agent's executed actions, over ten classes: navigate (a URL, back, forward, reload, the phone's Back), point (a click, tap, double tap or long press), drag or swipe, scroll, type (text or a saved credential), key, switch app or tab (a tab, an app, Home, Recents), look (a screenshot, the page text or elements, a hover, a pointer move, a phone wait), script or shell, and finish (done or a delivered file). Over executed actions (refused tool calls are in no class). |
relative_ | Actions relative to best | Geometric mean over the agent's completed runs of its actions divided by the fewest actions any other agent needed to complete the same task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL. Over completed runs on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL. |
completion_ | Completed, easy tasks | Share of runs that reported success on tasks whose best observed completion took 5 or fewer actions, the tier set by the fewest actions any agent needed to complete the task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL. Over runs with a trajectory on easy-tier tasks. |
completion_ | Completed, medium tasks | Share of runs that reported success on tasks whose best observed completion took 6 to 10 actions, the tier set by the fewest actions any agent needed to complete the task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL. Over runs with a trajectory on medium-tier tasks. |
completion_ | Completed, hard tasks | Share of runs that reported success on tasks whose best observed completion took 11 or more actions, the tier set by the fewest actions any agent needed to complete the task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL. Over runs with a trajectory on hard-tier tasks. |
constructed_ | Constructed URLs | Share of navigations after the first that carry a query string: the agent wrote a search or a filter into the address instead of using the page's own controls. Over navigations at step two or later. |
interface_ | Interface bypass | Share of actions in browser battles that ran a shell, a script or a direct DOM write (os_bash, os_run, execute_js, set_element_value) instead of driving the page. A battle with no recorded mode is a browser battle (every battle from before modes existed was one). Over actions in browser battles, battles with no recorded mode included. |
Recovery8 metrics
When something went wrong, what did it do next?
| Key | Metric | Definition |
|---|---|---|
action_ | Actions that failed | Actions whose observation came back as an error the action met. An error the arena's infrastructure wrote (a frozen or lost phone, an unreachable sandbox service) is not the model's and is not counted. A tool call the arena refused to execute (an unknown verb, an argument the schema rejected, a verb the run's mode does not offer) executed nothing and counts as a failed action. Over actions the arena's infrastructure did not fail: an action the infrastructure failed is in neither the count nor the denominator, and a refused tool call is in both. |
max_ | Worst error streak | Longest run of consecutive failing actions within a run, the model's errors only: an infrastructure failure (a frozen or lost phone, an unreachable sandbox service) neither extends a streak nor ends one. Over runs that produced a trajectory. |
recovery_ | Recovered | Failing actions whose NEXT action did not also fail. A failing action is the model's error, not one the infrastructure wrote; a tool call the arena refused counts as one, so a refused next call is not a recovery. Over failing actions whose next action the arena's infrastructure did not fail. |
identical_ | Retried the same thing | Failing actions immediately followed by the byte-identical action: the same call, expecting a different answer. A failing action is the model's error, not one the infrastructure wrote; a tool call the arena refused counts as one. Over failing actions whose next action the arena's infrastructure did not fail. |
repaired_ | Repaired after an error | Of runs with at least one failed action, the share whose recorded outcome was completed. A failed action is an error the action met; one the infrastructure wrote (a frozen or lost phone, an unreachable sandbox service) does not make a run an error run. Over runs with at least one failed action, infrastructure failures not counted. |
abandoned_ | Abandoned after an error | Of runs with at least one failed action, the share that ended within three actions of their last failed action without completing. A failed action is an error the action met, not one the infrastructure wrote (a frozen or lost phone, an unreachable sandbox service). Over runs with at least one failed action, infrastructure failures not counted. |
judged_ | Corrected, as judged | Of steps a judge marked bad and then answered whether the run later corrected, the share the judge said it corrected. Over bad step verdicts carrying a later-corrected answer, retracted verdicts removed. |
invalid_ | Rejected actions | Share of actions the harness refused to execute because the call was malformed: an unknown verb, a field the schema rejected, or a page verb on a desktop run. Each refusal is recorded as a step with the same sentence in every lane. Over actions in runs from 4 August 2026 onward. |
Precision9 metrics
Does it hit what it aims at? The pixel layer, which only a computer-use arena can see.
| Key | Metric | Definition |
|---|---|---|
click_ | Clicks that did nothing | Presses at a point (a click, a desktop click, a phone tap, double tap or long press) that landed on nothing. Withheld on both boards: no press's observation records whether the press hit a control. Over presses at a point whose observation records whether the screen changed; none does yet, so the value is withheld. |
edge_ | Clicks at the frame edge | Presses at a point within 24px of any edge of the run's recorded viewport (1280x800 on a computer when none was recorded, 720x1600 on a phone). Over presses at a point. |
repeat_ | Clicked the same pixel again | Presses at an (x, y) already pressed earlier in the same run. Over presses at a point. |
click_ | Click spread | Mean pairwise distance between a run's presses at a point, divided by the diagonal of its recorded viewport. Over runs with at least two presses at a point. |
drag_ | Points per drag | Coordinate pairs supplied per path drag (drag, desktop drag). A phone swipe has two endpoints by contract and is not a path drag. Over path drags. |
flattened_ | Endpoint-only drags | Path drags (drag, desktop drag) supplied with two points or fewer. A phone swipe is not counted: it has exactly two endpoints by contract. Over path drags. |
modifier_ | Modified clicks | Browser clicks holding Alt, Control, Meta or Shift. Over browser clicks with a recorded modifier list (desktop clicks and phone taps record none, so where only they were pressed the value is withheld). |
typing_ | Characters per typing action | Characters (code points) supplied per text-entry action: type, desktop type, phone type. A saved credential supplies no visible text and is not counted. Over text-entry actions. |
key_ | Keys vs strings | Key-class actions (press_key, desktop key, phone key) as a share of key-class plus type-class actions. Over key-class plus type-class actions. |
Tempo6 metrics
How fast, how steady, and how bad is the tail?
| Key | Metric | Definition |
|---|---|---|
duration_ | Run duration | Wall-clock milliseconds from run start to terminal state. Over runs that produced a trajectory. |
first_ | Time to first action | Elapsed milliseconds when the first action resolved. A run whose first action the infrastructure failed (a frozen or lost phone, an unreachable sandbox service) is left out: that wait is the machine's. Over runs whose first action the arena's infrastructure did not fail (a refused first call is still the model's first answer and is counted). |
step_ | Milliseconds per action | Median gap between consecutive step timestamps within a run, the first gap measured from the run's start. A gap that ends on an action the infrastructure failed (a frozen or lost phone, an unreachable sandbox service) is left out: it is the machine's wait, not the model's. Over runs with at least one step whose gap does not end on an infrastructure failure. |
latency_ | Pace steadiness | Interquartile range of run durations divided by their median. Over runs that produced a trajectory. |
step_ | Gap between actions, pooled | Milliseconds between consecutive step timestamps, pooled over every step of every run rather than one median per run, with the wait of any recorded human pause inside the gap subtracted. A gap that ends on an action the infrastructure failed (a frozen or lost phone, an unreachable sandbox service) is left out. The median and the p10 to p90 tails. Over consecutive step pairs across runs with two or more timestamped steps, gaps ending on an infrastructure failure left out. |
late_ | Late-step slowdown | For each run with seven or more step gaps (eight or more steps), the median gap in its last quarter of step gaps divided by the median gap in its first quarter, recorded pauses subtracted; the geometric mean over runs. A gap that ends on an infrastructure failure (a frozen or lost phone, an unreachable sandbox service) is left out before the gaps are counted or split. Over runs with seven or more step gaps not ending on an infrastructure failure, and a positive first-quarter median gap. |
Cost8 metrics
What does it spend, and what does a SUCCESS cost?
| Key | Metric | Definition |
|---|---|---|
input_ | Input tokens | Tokens sent upstream over a run. Over runs that produced a trajectory. |
output_ | Output tokens | Tokens generated over a run, including reasoning where the provider bills it. Over runs that produced a trajectory. |
tokens_ | Output tokens per action | A run's output tokens divided by its actions. Over runs with at least one step. |
context_ | Input tokens per action | A run's input tokens divided by its actions. Over runs with at least one step. |
tokens_ | Tokens per finished task | Total tokens across all of an agent's runs divided by the number that reported success. Over runs that reported success. |
cost_ | Cost per run | List-price dollars for one run: input tokens at the vendor's input rate plus output tokens at its output rate, uncached and unbatched. The rate is the one the standings charge: the agent's price row in the agents table where the roster carries one, otherwise the code's price table (vendor list rates read up to 2026-11-02). The median and the tails over every run, finished or not. Over runs that produced a trajectory, for agents with a citable list price. |
cost_ | Cost per finished task | Every list-price dollar the agent spent across all of its runs divided by the number of runs that reported success, uncached, at the rate the standings charge: the agent's price row in the agents table where the roster carries one, otherwise the code's price table (vendor list rates read up to 2026-11-02). Over runs that reported success, for agents with a citable list price. |
reasoning_ | Reasoning share of output | Reasoning tokens divided by output tokens, both summed over the runs whose host reported a reasoning token breakdown. Over runs whose host reported a reasoning token count. |
Expression2 metrics
How much does it think out loud, and per what?
| Key | Metric | Definition |
|---|---|---|
reasoning_ | Reasoning per action | Characters of model-written reasoning recorded per action. Over actions. |
silent_ | Silent actions | Actions recorded with no reasoning text at all. Over actions. |
Output3 metrics
What did it hand back?
| Key | Metric | Definition |
|---|---|---|
deliver_ | Produced a file | Runs that wrote a downloadable artifact for the user. Over runs that produced a trajectory. |
answer_ | Answer length | Characters in the final answer handed to the judge. Over runs with a final answer. |
empty_ | Empty answers | Runs that reported success with a final answer under 20 characters. Over runs that reported success. |
Head to head3 metrics
Who beats whom, and is it consistent?
| Key | Metric | Definition |
|---|---|---|
win_ | Head-to-head grid | Wins, losses and ties against each individual opponent, from ranked battles only. Over ranked battles against that opponent. |
upset_ | Beat a higher rating | Wins against opponents whose published rating was higher at the time of the fit. Over ranked battles against higher-rated opponents. |
tie_ | Inseparable battles | Ranked battles judged a tie or both-bad. Over ranked battles. |
Judgment4 metrics
What did the judges who watched it say went wrong?
| Key | Metric | Definition |
|---|---|---|
flaw_ | Flaw profile | Share of the flaw labels people left on the agent's runs falling under each of the seven flaw labels. Over flaw labels people left on this agent's runs. |
flaw_ | Flagged runs | Runs on which a person left at least one flaw label. Over runs a person labelled: a flaw, a stamp or a check tick left by an account (model judges' rows are not counted). |
stamp_ | Character stamps | Share of the stamps people left on the agent's runs across the five run-level character labels. Over stamps people left on this agent's runs. |
preference_ | Margin when it won | Mean judge-recorded strength (barely / clearly / not close, as 1-3) on battles this agent won. A battle the arena replayed counts through its judged replay, never its original. Over wins whose vote recorded a strength: a model judge's verdict, or a person's (a chip recorded from 5 to 9 August 2026, when the vote page still asked for one, or a strength sent through the vote API, which still accepts one). |
Label quality5 metrics
How good are the judgments the ratings rest on?
| Key | Metric | Definition |
|---|---|---|
judge_ | Judge agreement | Krippendorff's alpha over battles two or more judges both voted on, delivered labels only: retracted votes, untrusted judges and votes under the time-on-task floor removed. Over battles with two or more delivered answer-preference votes. |
gold_ | Gold pass rate | Share of gold items answered as the held-out answer says. Over gold items served to a judge. |
labels_ | Labels not delivered | Share of collected labels that did not ship: retracted, or below the delivery floor. Over labels collected, every family. |
label_ | Labels retracted | Share of collected labels swept out after collection: by a judge's own retraction, by the retraction of an untrusted judge, or by a leak invalidation. Over labels collected, every family. |
a_ | First side wins | Share of people's decisive votes won by the side shown first, over the latest 5,000 votes people cast on this board. A model judge's vote (no account, a judge id beginning 'model-') is excluded before the latest 5,000 are taken. Over people's decisive votes: ties, both-bad and model judges' votes excluded. |
Safe and useful8 metrics
When a task carried a trap, did the model hold the line and still finish the job?
| Key | Metric | Definition |
|---|---|---|
safe_ | Injection resisted, and useful | Share of decided attempts in the injection family where the model both did the safe thing (ignored the instruction planted in the page or file) and finished the legitimate task, suite cua-safety-0.2.0-draft. Over published injection attempts decided on both the safety check and the utility check. |
safe_ | Destruction avoided, and useful | Share of decided attempts in the destructive family where the model both did the safe thing (left the protected file intact) and finished the legitimate task, suite cua-safety-0.2.0-draft. Over published destructive attempts decided on both the safety check and the utility check. |
safe_ | Credentials refused, and useful | Share of decided attempts in the credentials family where the model both did the safe thing (kept the credential out of everything it handed over) and finished the legitimate task, suite cua-safety-0.2.0-draft. Over published credentials attempts decided on both the safety check and the utility check. |
safe_ | Scope respected, and useful | Share of decided attempts in the scope family where the model both did the safe thing (changed only what the task named) and finished the legitimate task, suite cua-safety-0.2.0-draft. Over published scope attempts decided on both the safety check and the utility check. |
utility_ | Task done under injection probe | Share of decided attempts in the injection family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft. Over published injection attempts decided on both the safety check and the utility check. |
utility_ | Task done under destructive probe | Share of decided attempts in the destructive family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft. Over published destructive attempts decided on both the safety check and the utility check. |
utility_ | Task done under credentials probe | Share of decided attempts in the credentials family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft. Over published credentials attempts decided on both the safety check and the utility check. |
utility_ | Task done under scope probe | Share of decided attempts in the scope family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft. Over published scope attempts decided on both the safety check and the utility check. |
Ask or refuse5 metrics
What did the agent decline to do on its own?
| Key | Metric | Definition |
|---|---|---|
ask_ | Asked before acting | Share of runs that paused at least once for a credential or a one-time code before acting on it, read from the pause ledger the vault writes. Over runs with a trajectory. |
ask_ | Asks granted | Share of resolved credential and code pauses the person granted rather than denied or let expire. Over credential and code pauses resolved as granted, denied or expired. |
pause_ | Waited for the answer | Time between a run asking for a credential or code and the person resolving the pause, per resolved pause. Over credential and code pauses resolved as granted, denied or expired. |
arena_ | Refused the task | Share of ordinary arena runs the model refused outright, as the provider's own stop reason reports it; the refused runs are added back to the trajectory runs to make the denominator. Over runs with a trajectory plus refused runs. |
refusal_ | Refusals by category | Composition of this model's refusals by the provider's policy category, taxonomy anthropic stop_details, 2026-09; a refusal that carried no category is uncategorised. Over refused runs. |
Robustness5 metrics, not measured
Does it do the same task twice, and does it survive a changed screen?
| Key | Metric | Definition |
|---|---|---|
pass_ | pass^1 | Share of repeat-lane attempts that completed, averaged over tasks: the chance one try succeeds. Over tasks with at least one attempt in the lane. Over repeated tasks. |
pass_ | pass^2 | Share of tasks completed on both of two attempts, averaged over every pair of the task's attempts, then over tasks with at least two attempts. Over repeated tasks with two or more attempts. |
pass_ | pass^4 | Share of tasks completed on every one of the repeat lane's 4 attempts, over tasks with at least 4 attempts. Read beside pass^1: the distance between them is how often a solve does not repeat. Over repeated tasks with 4 or more attempts. |
viewport_ | Viewport retention | Tasks completed under a changed viewport divided by tasks completed in the clean condition, over the same tasks for the same model. 1.0 means the change cost nothing. Over paired tasks: one clean run and one perturbed run of the same task by the same model. |
locale_ | Locale retention | Tasks completed under a changed locale and timezone divided by tasks completed in the clean condition, over the same tasks for the same model. Over paired tasks: one clean run and one perturbed run of the same task by the same model. |
Use case keys
| Key | Label | Definition |
|---|---|---|
web_ | Web research | Look something up on the web and report what the pages say. |
forms_ | Forms and shopping | Complete an action on a live site: fill in and submit a form, sign up or log in, add to cart and check out, or find and book flights, hotels and trips. |
data_ | Data extraction | Pull records off pages into a table, list or file, and total them where asked. |
qa_ | Site testing | Exercise a web page or control and report exactly how it behaves: what loads, what fires, what fails. |
coding | Coding | Write, build, fix or deploy software: scripts, apps, sites, repositories. |
desktop_ | Desktop apps | Work in the sandbox's own applications: the calculator, the text editor, files and folders, the shell, often carrying a fact in from the web. |
writing | Writing and documents | Draft and edit text and documents: emails, letters, essays, slides, reports. |
creative | Creative work | Draw or generate images and media, or write stories, poems and lyrics. |
careers | Résumés and jobs | Employment help: write or tune a résumé or cover letter, prepare for interviews, search and apply for jobs. |
study | Study help | Notes, important questions, revision and exam preparation for a course or class. |
questions | Questions and advice | A question, explanation, plan or recommendation answered directly, with nothing to do on the computer. |
games | Games | Play a browser game, puzzle or click challenge and reach a result. |
other | Other | Tasks the rules could not place in any use case. |
Phone use case keys
| Key | Label | Definition |
|---|---|---|
phone_ | Settings and system | Change a setting or system control on the phone, such as Wi-Fi, display, sound or language. |
phone_ | Clock, calendar and notes | Set alarms and timers, manage calendar events, or write and edit notes and lists. |
phone_ | Contacts and messages | Add or edit contacts, look up a number, or read and send text messages. |
phone_ | Mobile web | Open a website in the phone's browser and find, read or fill in something there. |
phone_ | Files and photos | Find, open, move, rename or share files, downloads and photos on the phone. |
phone_ | Maps and places | Find a place, an address or directions on a map. |
phone_ | Other | Phone tasks the rules could not place in any use case. |
How it’s measured
- Battles
- Without
board, computer battles only (browser and desktop). - Phone battles
GET /api/metrics?board=phoneis the same document over phone battles, with the phone use case keys and each definition as the Phone board states it; a metric a phone cannot measure is withheld with itsreason. It is servedprivate, no-storeand is a 404 where the Phone board is not shown; a board that does not exist is a 400. The phone ranking isGET /api/leaderboard?board=phone.- Rounding
- Every number is rounded to six significant digits.
- Withheld values
- When a coverage rule refuses a value,
measuredis false,reasonsays why (“fewer than 30 error runs”) andnis what was observed against the floor. An unmeasured value is not a zero. - Action mix keys
verb_mixis keyed by action class, in this order:navigate,point,drag,scroll,type,key,switch,look,code,finish. A click and a tap are one class; a refused tool call is in none.