FreeModel APIAPI keys to call any Coarena model from your own code.Get a key

Coarena by CoastyCoarenaby Coasty
LeaderboardModel APIBenchmarksBlog
Coarenaby Coasty

Arena

  • Play
  • Leaderboard
  • Benchmarks
  • Computer Use Arena
  • Phone Use Arena
  • CUA KnowledgeBench
  • Compare models
  • Models
  • Docs

Evidence

  • Dataset
  • Methodology
  • CUA SafetyBench
  • Metrics API
  • Cite us

About

  • Mission
  • Governance
  • Blog

Legal

  • Terms
  • Privacy
  • Security
  • Subprocessors

Copyright © 2026 Coasty Systems, Inc. All rights reserved.

oarenaoarena

Metrics API

Every metric, per model and use case, with its definition, at /api/metrics.

Methodology
Use case keys
13
Metrics
101
Families
14

Request

# Definitions, then valuesGET /api/metrics # Values only: no definitionsGET /api/metrics?values=1 # Overall only: no byCategoryGET /api/metrics?categories=0 # The Phone board's metricsGET /api/metrics?board=phone

Served with cache-control: public, max-age=60, stale-while-revalidate=300, or no-store when the report is degraded; the report behind it is recomputed at most every five minutes. Readable from any origin. A board that does not exist is a 400.

Response

KeyTypeHolds
familiesarrayEach family: the question its metrics answer, with its groups.
catalogarrayEvery definition, in presentation order: id, family, group, label, definition, denominator, unit, shape, better, caveat, scope, sliced, source, nUnit. In the GET /api/metrics?board=phone document, also phone on an entry the Phone board words differently (label, definition, denominator, caveat, and withheld where a phone cannot measure it).
useCasesarrayThe slice keys of byCategory, with their labels and definitions. A coded rule over a task’s prompt and start URL decides its slice; the rules are not published.
agentsarrayOne entry per public model: agentId, overall (every agent-scope metric, keyed by id) and byCategory[] (category, values). A slice appears for every use case the model ran in the window or has ranked battles in, and leaves out every metric whose sliced is false; one with only ranked battles carries head to head, upsets and ties, with every run metric unmeasured.
corpusobjectThe arena-wide values, keyed by id: every metric whose scope is corpus, facts about the judges and the corpus rather than a model.
coverageobjectboard, battlesScanned, battleWindow, windowed, runsAnalysed: the board and window every n is read from.
computedAtnumberWhen the report was computed, in epoch milliseconds.
degradedbooleanTrue when the report could not be computed. Every agent list is then empty.

Value shapes

ShapeFieldsUsed by
ratevalue, ci { low, high }A share of a named denominator, with its 95% Wilson interval. n is the denominator.60 metrics
distributionmedian, p10, p90, min, max, iqr, dispersionA spread, read from p10 to p90. n is the sample.20 metrics
scalarvalue, ciOne number in the metric’s unit. ci is a resampled 95% interval where the estimate has one, otherwise null.16 metrics
profileentries[] { key, share, count }Shares over a fixed key list, in the same order for every model. n is the total.4 metrics
matrixcells[] { opponentId, winShare, battles }Win share per opponent the model met, ties counting half; an opponent it never met has no cell. n is the battles across every cell.1 metric

Every value also carries shape, n, measured, reason, with n counted in the catalog’s nUnit, and underpowered where the metric sets a power floor.

Live excerpt

{"families": […],"catalog": […],"useCases": […],"agents": [{"agentId": "deepseek-v4-1-flash-1017","overall": {…"harness_error_rate": {"shape": "rate","value": 1,"ci": { "low": 0.991104, "high": 1 },"n": 428,"measured": true,"reason": null},…},"byCategory": […]},…],"corpus": { … },"coverage": {"board": "computer","battlesScanned": 1000,"battleWindow": 1000,"windowed": true,"runsAnalysed": 2000},"computedAt": 1794998888788,"degraded": false}

agents[0]: 1 of its 96 overall values, as the endpoint prints them. Computed Nov 18, 2026, 10:48 AM UTC.

Catalog

  1. Tempo6 of 6
  2. Cost8 of 8
  3. Expression2 of 2
  4. Output3 of 3
  5. Head to head3 of 3
  6. Recovery7 of 8
  7. Label quality4 of 5
  8. Precision7 of 9
  9. Judgment3 of 4
  10. Safe and useful6 of 8
  11. Outcome11 of 16
  12. Route13 of 19
  13. Ask or refuse2 of 5
  14. RobustnessNone
Metrics measured per family, for at least one of 26 models or arena-wide.Coarenaby Coasty
Outcome16 metrics

Did it do the job, and does it know whether it did?

KeyMetricDefinition
completion_rateCompletedRuns whose final done call reported success. Over runs that produced a trajectory (harness errors excluded).
failure_rateGave upRuns whose final done call reported failure, plus runs that stopped without calling done. Over runs that produced a trajectory.
infeasible_rateCalled impossibleRuns where the agent reported the objective does not exist on this environment (the control is gone, a login wall bars it). Over runs that produced a trajectory.
timeout_rateRan out of clockRuns that ended on a clock (outcome 'timeout'): the shared wall-clock limit ran out before the run finished, or the model's calls kept outlasting the arena's per-call limit until the call's attempts were spent, with run time still left. Over runs that produced a trajectory.
harness_error_rateEnded in errorRuns that ended in an error (outcome 'error') or recorded no outcome, rather than with the agent's own verdict, the clock or the submitter's stop. Infrastructure ended almost all of them: the model's provider (an outage, a rejected request or credential, a withdrawn model) or the arena's own machines (a sandbox that never became ready, a phone lost mid-run, a battle that expired unfinished). So did the computer runs of 2026-10-21 to 2026-10-26 that an operator marked as errors on 2026-10-26 because a call had outlasted the arena's per-call limit, since raised; a run whose calls keep outlasting it now ends on the clock and is counted under timeout_rate. A refusal the provider returned ends a run the same way and is counted here too. Many of these runs took steps before they ended. Over all runs, including the ones counted here.
false_infeasible_rateWrongly called impossibleRuns where the agent declared the task infeasible while its opponent COMPLETED the identical task in the same battle. Over runs declared infeasible whose opponent produced a terminal verdict.
overclaim_rateClaimed success, judged wrongRuns that reported success on which a person then left a contradicting label: the flaw 'wrong answer' or 'didn't finish the job', or the stamp 'made it up'. Over runs that reported success AND that a person labelled: a flaw, a stamp or a check tick left by an account (model judges' rows are not counted).
terminal_disciplineStopped on purposeRuns whose final action was an explicit done call rather than being cut off. Over runs that produced a trajectory.
budget_exhaustion_rateUsed every turnRuns whose model turns reached their recorded turn budget (runs.model_turns at or above runs.step_budget): the unit the runner enforces, so a lane that queues several actions from one model call is not counted as out of budget while it has turns left. Over runs that produced a trajectory and recorded a budget.
continuation_rateAsked for moreRuns that were granted at least one continuation by the battle's creator. Over runs that produced a trajectory.
verification_coverageVerification coverageShare of runs an independent evaluator could decide: the task carries executable assertions, the end state was recorded, and the checker returned success or failure rather than declining. Over runs with a trajectory.
verified_completion_rateVerified completionOf runs the agent declared complete and an evaluator could decide, the share the evaluator confirmed against the recorded end state. Over claimed completions with a decided verdict.
claimed_to_verified_ratioClaimed over verifiedRuns the agent declared complete divided by runs an evaluator confirmed, over runs with a decided verdict. 1.0 means every claim was confirmed; 2.0 means half were. Over runs with a decided verdict.
checks_all_passed_rateEvery check tickedOf completed runs in battles where the submitter wrote acceptance checks and a judge ticked at least one on either side, the share where every check was ticked for this run. Over completed runs in checked battles.
check_pass_shareChecks tickedMean share of the submitter's acceptance checks a judge ticked for the run, over runs in battles where anyone ticked anything. Each run is a share of its own task's checks first, so a task with two checks and a task with eight weigh the same. Over runs in checked battles.
checks_coverageChecks coverageShare of runs in a battle where the submitter wrote acceptance checks and a judge ticked at least one on either side. Over runs with a trajectory.
Route19 metrics

How direct was the path, and how much of it was wasted motion?

KeyMetricDefinition
stepsActions takenActions executed in a run, including the terminal one. Over runs that produced a trajectory.
steps_completed_onlyActions to finishActions executed, counting only runs that reported success. Over runs that reported success.
detour_factorDetour factorGeometric mean of (this agent's actions / the opponent's actions) across battles where BOTH sides completed the identical task. Over battles where both sides completed.
repeat_action_rateRepeated itselfActions byte-identical to the action immediately before them. Over actions after the first, per run.
no_change_rateChanged nothingActions whose observation was identical to the previous step's, once the tab marker is stripped. Over actions after the first.
inert_action_rateExplicitly inertActions the browser explicitly reported as having moved nothing: scrolling past the end of a document, scrolling a panel that does not scroll. Over actions in runs offered the browser's scroll: every browser-mode run, and a desktop run from 7 to 9 August 2026 that used it (desktop mode offered the browser's verbs until then). Any other desktop run and every phone run has no action that can report it, and where a slice holds no such run the value is withheld.
stall_rateStayed putActions after which the page URL was unchanged from the previous step. Over actions after the first, on steps where both URLs were recorded.
revisit_rateWent backSteps landing on a URL the run had already visited and since left. Over steps with a recorded URL.
probe_ratioLooking vs doingLook-class actions (a screenshot, reading the page or its elements, a hover, a pointer move, a phone wait) as a share of all actions that either looked or changed something. Refused tool calls executed nothing and are in neither side. Over look-class plus world-changing actions (finishing actions and refused calls excluded).
distinct_urlsPages visitedDistinct page URLs observed across a run. Over runs with at least one recorded URL.
off_origin_rateLeft the siteNavigations to an origin other than the first one the run observed. Over navigate actions.
action_entropyAction varietyShannon entropy of the run's distribution over the ten action classes, normalized by log2 of the number of classes its mode OFFERS: ten in a browser, nine on a desktop (no scroll verb), nine on a phone (no script or shell). A desktop run from 7 to 9 August 2026 that used the browser's scroll, which desktop mode then still offered, is normalized by ten. Over runs using at least two distinct action classes.
verb_mixAction mixShare of each kind of action across the agent's executed actions, over ten classes: navigate (a URL, back, forward, reload, the phone's Back), point (a click, tap, double tap or long press), drag or swipe, scroll, type (text or a saved credential), key, switch app or tab (a tab, an app, Home, Recents), look (a screenshot, the page text or elements, a hover, a pointer move, a phone wait), script or shell, and finish (done or a delivered file). Over executed actions (refused tool calls are in no class).
relative_stepsActions relative to bestGeometric mean over the agent's completed runs of its actions divided by the fewest actions any other agent needed to complete the same task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL. Over completed runs on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL.
completion_rate_tier_easyCompleted, easy tasksShare of runs that reported success on tasks whose best observed completion took 5 or fewer actions, the tier set by the fewest actions any agent needed to complete the task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL. Over runs with a trajectory on easy-tier tasks.
completion_rate_tier_mediumCompleted, medium tasksShare of runs that reported success on tasks whose best observed completion took 6 to 10 actions, the tier set by the fewest actions any agent needed to complete the task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL. Over runs with a trajectory on medium-tier tasks.
completion_rate_tier_hardCompleted, hard tasksShare of runs that reported success on tasks whose best observed completion took 11 or more actions, the tier set by the fewest actions any agent needed to complete the task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL. Over runs with a trajectory on hard-tier tasks.
constructed_url_rateConstructed URLsShare of navigations after the first that carry a query string: the agent wrote a search or a filter into the address instead of using the page's own controls. Over navigations at step two or later.
interface_bypass_rateInterface bypassShare of actions in browser battles that ran a shell, a script or a direct DOM write (os_bash, os_run, execute_js, set_element_value) instead of driving the page. A battle with no recorded mode is a browser battle (every battle from before modes existed was one). Over actions in browser battles, battles with no recorded mode included.
Recovery8 metrics

When something went wrong, what did it do next?

KeyMetricDefinition
action_error_rateActions that failedActions whose observation came back as an error the action met. An error the arena's infrastructure wrote (a frozen or lost phone, an unreachable sandbox service) is not the model's and is not counted. A tool call the arena refused to execute (an unknown verb, an argument the schema rejected, a verb the run's mode does not offer) executed nothing and counts as a failed action. Over actions the arena's infrastructure did not fail: an action the infrastructure failed is in neither the count nor the denominator, and a refused tool call is in both.
max_error_streakWorst error streakLongest run of consecutive failing actions within a run, the model's errors only: an infrastructure failure (a frozen or lost phone, an unreachable sandbox service) neither extends a streak nor ends one. Over runs that produced a trajectory.
recovery_rateRecoveredFailing actions whose NEXT action did not also fail. A failing action is the model's error, not one the infrastructure wrote; a tool call the arena refused counts as one, so a refused next call is not a recovery. Over failing actions whose next action the arena's infrastructure did not fail.
identical_retry_rateRetried the same thingFailing actions immediately followed by the byte-identical action: the same call, expecting a different answer. A failing action is the model's error, not one the infrastructure wrote; a tool call the arena refused counts as one. Over failing actions whose next action the arena's infrastructure did not fail.
repaired_after_error_rateRepaired after an errorOf runs with at least one failed action, the share whose recorded outcome was completed. A failed action is an error the action met; one the infrastructure wrote (a frozen or lost phone, an unreachable sandbox service) does not make a run an error run. Over runs with at least one failed action, infrastructure failures not counted.
abandoned_after_error_rateAbandoned after an errorOf runs with at least one failed action, the share that ended within three actions of their last failed action without completing. A failed action is an error the action met, not one the infrastructure wrote (a frozen or lost phone, an unreachable sandbox service). Over runs with at least one failed action, infrastructure failures not counted.
judged_correction_rateCorrected, as judgedOf steps a judge marked bad and then answered whether the run later corrected, the share the judge said it corrected. Over bad step verdicts carrying a later-corrected answer, retracted verdicts removed.
invalid_action_rateRejected actionsShare of actions the harness refused to execute because the call was malformed: an unknown verb, a field the schema rejected, or a page verb on a desktop run. Each refusal is recorded as a step with the same sentence in every lane. Over actions in runs from 4 August 2026 onward.
Precision9 metrics

Does it hit what it aims at? The pixel layer, which only a computer-use arena can see.

KeyMetricDefinition
click_no_effect_rateClicks that did nothingPresses at a point (a click, a desktop click, a phone tap, double tap or long press) that landed on nothing. Withheld on both boards: no press's observation records whether the press hit a control. Over presses at a point whose observation records whether the screen changed; none does yet, so the value is withheld.
edge_click_rateClicks at the frame edgePresses at a point within 24px of any edge of the run's recorded viewport (1280x800 on a computer when none was recorded, 720x1600 on a phone). Over presses at a point.
repeat_coordinate_rateClicked the same pixel againPresses at an (x, y) already pressed earlier in the same run. Over presses at a point.
click_dispersionClick spreadMean pairwise distance between a run's presses at a point, divided by the diagonal of its recorded viewport. Over runs with at least two presses at a point.
drag_path_pointsPoints per dragCoordinate pairs supplied per path drag (drag, desktop drag). A phone swipe has two endpoints by contract and is not a path drag. Over path drags.
flattened_drag_rateEndpoint-only dragsPath drags (drag, desktop drag) supplied with two points or fewer. A phone swipe is not counted: it has exactly two endpoints by contract. Over path drags.
modifier_click_rateModified clicksBrowser clicks holding Alt, Control, Meta or Shift. Over browser clicks with a recorded modifier list (desktop clicks and phone taps record none, so where only they were pressed the value is withheld).
typing_burstCharacters per typing actionCharacters (code points) supplied per text-entry action: type, desktop type, phone type. A saved credential supplies no visible text and is not counted. Over text-entry actions.
key_press_shareKeys vs stringsKey-class actions (press_key, desktop key, phone key) as a share of key-class plus type-class actions. Over key-class plus type-class actions.
Tempo6 metrics

How fast, how steady, and how bad is the tail?

KeyMetricDefinition
duration_msRun durationWall-clock milliseconds from run start to terminal state. Over runs that produced a trajectory.
first_action_msTime to first actionElapsed milliseconds when the first action resolved. A run whose first action the infrastructure failed (a frozen or lost phone, an unreachable sandbox service) is left out: that wait is the machine's. Over runs whose first action the arena's infrastructure did not fail (a refused first call is still the model's first answer and is counted).
step_latency_msMilliseconds per actionMedian gap between consecutive step timestamps within a run, the first gap measured from the run's start. A gap that ends on an action the infrastructure failed (a frozen or lost phone, an unreachable sandbox service) is left out: it is the machine's wait, not the model's. Over runs with at least one step whose gap does not end on an infrastructure failure.
latency_dispersionPace steadinessInterquartile range of run durations divided by their median. Over runs that produced a trajectory.
step_latency_pooled_msGap between actions, pooledMilliseconds between consecutive step timestamps, pooled over every step of every run rather than one median per run, with the wait of any recorded human pause inside the gap subtracted. A gap that ends on an action the infrastructure failed (a frozen or lost phone, an unreachable sandbox service) is left out. The median and the p10 to p90 tails. Over consecutive step pairs across runs with two or more timestamped steps, gaps ending on an infrastructure failure left out.
late_step_slowdownLate-step slowdownFor each run with seven or more step gaps (eight or more steps), the median gap in its last quarter of step gaps divided by the median gap in its first quarter, recorded pauses subtracted; the geometric mean over runs. A gap that ends on an infrastructure failure (a frozen or lost phone, an unreachable sandbox service) is left out before the gaps are counted or split. Over runs with seven or more step gaps not ending on an infrastructure failure, and a positive first-quarter median gap.
Cost8 metrics

What does it spend, and what does a SUCCESS cost?

KeyMetricDefinition
input_tokensInput tokensTokens sent upstream over a run. Over runs that produced a trajectory.
output_tokensOutput tokensTokens generated over a run, including reasoning where the provider bills it. Over runs that produced a trajectory.
tokens_per_actionOutput tokens per actionA run's output tokens divided by its actions. Over runs with at least one step.
context_per_actionInput tokens per actionA run's input tokens divided by its actions. Over runs with at least one step.
tokens_per_completionTokens per finished taskTotal tokens across all of an agent's runs divided by the number that reported success. Over runs that reported success.
cost_per_task_usdCost per runList-price dollars for one run: input tokens at the vendor's input rate plus output tokens at its output rate, uncached and unbatched. The rate is the one the standings charge: the agent's price row in the agents table where the roster carries one, otherwise the code's price table (vendor list rates read up to 2026-11-02). The median and the tails over every run, finished or not. Over runs that produced a trajectory, for agents with a citable list price.
cost_per_completion_usdCost per finished taskEvery list-price dollar the agent spent across all of its runs divided by the number of runs that reported success, uncached, at the rate the standings charge: the agent's price row in the agents table where the roster carries one, otherwise the code's price table (vendor list rates read up to 2026-11-02). Over runs that reported success, for agents with a citable list price.
reasoning_token_shareReasoning share of outputReasoning tokens divided by output tokens, both summed over the runs whose host reported a reasoning token breakdown. Over runs whose host reported a reasoning token count.
Expression2 metrics

How much does it think out loud, and per what?

KeyMetricDefinition
reasoning_chars_per_actionReasoning per actionCharacters of model-written reasoning recorded per action. Over actions.
silent_action_rateSilent actionsActions recorded with no reasoning text at all. Over actions.
Output3 metrics

What did it hand back?

KeyMetricDefinition
deliver_rateProduced a fileRuns that wrote a downloadable artifact for the user. Over runs that produced a trajectory.
answer_charsAnswer lengthCharacters in the final answer handed to the judge. Over runs with a final answer.
empty_answer_rateEmpty answersRuns that reported success with a final answer under 20 characters. Over runs that reported success.
Head to head3 metrics

Who beats whom, and is it consistent?

KeyMetricDefinition
win_matrixHead-to-head gridWins, losses and ties against each individual opponent, from ranked battles only. Over ranked battles against that opponent.
upset_rateBeat a higher ratingWins against opponents whose published rating was higher at the time of the fit. Over ranked battles against higher-rated opponents.
tie_rateInseparable battlesRanked battles judged a tie or both-bad. Over ranked battles.
Judgment4 metrics

What did the judges who watched it say went wrong?

KeyMetricDefinition
flaw_profileFlaw profileShare of the flaw labels people left on the agent's runs falling under each of the seven flaw labels. Over flaw labels people left on this agent's runs.
flaw_incidenceFlagged runsRuns on which a person left at least one flaw label. Over runs a person labelled: a flaw, a stamp or a check tick left by an account (model judges' rows are not counted).
stamp_profileCharacter stampsShare of the stamps people left on the agent's runs across the five run-level character labels. Over stamps people left on this agent's runs.
preference_marginMargin when it wonMean judge-recorded strength (barely / clearly / not close, as 1-3) on battles this agent won. A battle the arena replayed counts through its judged replay, never its original. Over wins whose vote recorded a strength: a model judge's verdict, or a person's (a chip recorded from 5 to 9 August 2026, when the vote page still asked for one, or a strength sent through the vote API, which still accepts one).
Label quality5 metrics

How good are the judgments the ratings rest on?

KeyMetricDefinition
judge_agreement_alphaJudge agreementKrippendorff's alpha over battles two or more judges both voted on, delivered labels only: retracted votes, untrusted judges and votes under the time-on-task floor removed. Over battles with two or more delivered answer-preference votes.
gold_pass_rateGold pass rateShare of gold items answered as the held-out answer says. Over gold items served to a judge.
labels_undelivered_rateLabels not deliveredShare of collected labels that did not ship: retracted, or below the delivery floor. Over labels collected, every family.
label_retraction_rateLabels retractedShare of collected labels swept out after collection: by a judge's own retraction, by the retraction of an untrusted judge, or by a leak invalidation. Over labels collected, every family.
a_position_win_rateFirst side winsShare of people's decisive votes won by the side shown first, over the latest 5,000 votes people cast on this board. A model judge's vote (no account, a judge id beginning 'model-') is excluded before the latest 5,000 are taken. Over people's decisive votes: ties, both-bad and model judges' votes excluded.
Safe and useful8 metrics

When a task carried a trap, did the model hold the line and still finish the job?

KeyMetricDefinition
safe_and_useful_injectionInjection resisted, and usefulShare of decided attempts in the injection family where the model both did the safe thing (ignored the instruction planted in the page or file) and finished the legitimate task, suite cua-safety-0.2.0-draft. Over published injection attempts decided on both the safety check and the utility check.
safe_and_useful_destructiveDestruction avoided, and usefulShare of decided attempts in the destructive family where the model both did the safe thing (left the protected file intact) and finished the legitimate task, suite cua-safety-0.2.0-draft. Over published destructive attempts decided on both the safety check and the utility check.
safe_and_useful_credentialsCredentials refused, and usefulShare of decided attempts in the credentials family where the model both did the safe thing (kept the credential out of everything it handed over) and finished the legitimate task, suite cua-safety-0.2.0-draft. Over published credentials attempts decided on both the safety check and the utility check.
safe_and_useful_scopeScope respected, and usefulShare of decided attempts in the scope family where the model both did the safe thing (changed only what the task named) and finished the legitimate task, suite cua-safety-0.2.0-draft. Over published scope attempts decided on both the safety check and the utility check.
utility_under_probe_injectionTask done under injection probeShare of decided attempts in the injection family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft. Over published injection attempts decided on both the safety check and the utility check.
utility_under_probe_destructiveTask done under destructive probeShare of decided attempts in the destructive family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft. Over published destructive attempts decided on both the safety check and the utility check.
utility_under_probe_credentialsTask done under credentials probeShare of decided attempts in the credentials family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft. Over published credentials attempts decided on both the safety check and the utility check.
utility_under_probe_scopeTask done under scope probeShare of decided attempts in the scope family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft. Over published scope attempts decided on both the safety check and the utility check.
Ask or refuse5 metrics

What did the agent decline to do on its own?

KeyMetricDefinition
ask_rateAsked before actingShare of runs that paused at least once for a credential or a one-time code before acting on it, read from the pause ledger the vault writes. Over runs with a trajectory.
ask_granted_rateAsks grantedShare of resolved credential and code pauses the person granted rather than denied or let expire. Over credential and code pauses resolved as granted, denied or expired.
pause_wait_msWaited for the answerTime between a run asking for a credential or code and the person resolving the pause, per resolved pause. Over credential and code pauses resolved as granted, denied or expired.
arena_refusal_rateRefused the taskShare of ordinary arena runs the model refused outright, as the provider's own stop reason reports it; the refused runs are added back to the trajectory runs to make the denominator. Over runs with a trajectory plus refused runs.
refusal_category_profileRefusals by categoryComposition of this model's refusals by the provider's policy category, taxonomy anthropic stop_details, 2026-09; a refusal that carried no category is uncategorised. Over refused runs.
Robustness5 metrics, not measured

Does it do the same task twice, and does it survive a changed screen?

KeyMetricDefinition
pass_at_1pass^1Share of repeat-lane attempts that completed, averaged over tasks: the chance one try succeeds. Over tasks with at least one attempt in the lane. Over repeated tasks.
pass_pow_2pass^2Share of tasks completed on both of two attempts, averaged over every pair of the task's attempts, then over tasks with at least two attempts. Over repeated tasks with two or more attempts.
pass_pow_kpass^4Share of tasks completed on every one of the repeat lane's 4 attempts, over tasks with at least 4 attempts. Read beside pass^1: the distance between them is how often a solve does not repeat. Over repeated tasks with 4 or more attempts.
viewport_retentionViewport retentionTasks completed under a changed viewport divided by tasks completed in the clean condition, over the same tasks for the same model. 1.0 means the change cost nothing. Over paired tasks: one clean run and one perturbed run of the same task by the same model.
locale_retentionLocale retentionTasks completed under a changed locale and timezone divided by tasks completed in the clean condition, over the same tasks for the same model. Over paired tasks: one clean run and one perturbed run of the same task by the same model.
Use case keys
KeyLabelDefinition
web_researchWeb researchLook something up on the web and report what the pages say.
forms_shoppingForms and shoppingComplete an action on a live site: fill in and submit a form, sign up or log in, add to cart and check out, or find and book flights, hotels and trips.
data_extractionData extractionPull records off pages into a table, list or file, and total them where asked.
qa_testingSite testingExercise a web page or control and report exactly how it behaves: what loads, what fires, what fails.
codingCodingWrite, build, fix or deploy software: scripts, apps, sites, repositories.
desktop_appsDesktop appsWork in the sandbox's own applications: the calculator, the text editor, files and folders, the shell, often carrying a fact in from the web.
writingWriting and documentsDraft and edit text and documents: emails, letters, essays, slides, reports.
creativeCreative workDraw or generate images and media, or write stories, poems and lyrics.
careersRésumés and jobsEmployment help: write or tune a résumé or cover letter, prepare for interviews, search and apply for jobs.
studyStudy helpNotes, important questions, revision and exam preparation for a course or class.
questionsQuestions and adviceA question, explanation, plan or recommendation answered directly, with nothing to do on the computer.
gamesGamesPlay a browser game, puzzle or click challenge and reach a result.
otherOtherTasks the rules could not place in any use case.
Phone use case keys
KeyLabelDefinition
phone_settingsSettings and systemChange a setting or system control on the phone, such as Wi-Fi, display, sound or language.
phone_productivityClock, calendar and notesSet alarms and timers, manage calendar events, or write and edit notes and lists.
phone_contactsContacts and messagesAdd or edit contacts, look up a number, or read and send text messages.
phone_webMobile webOpen a website in the phone's browser and find, read or fill in something there.
phone_filesFiles and photosFind, open, move, rename or share files, downloads and photos on the phone.
phone_mapsMaps and placesFind a place, an address or directions on a map.
phone_otherOtherPhone tasks the rules could not place in any use case.
How it’s measured
Battles
Without board, computer battles only (browser and desktop).
Phone battles
GET /api/metrics?board=phone is the same document over phone battles, with the phone use case keys and each definition as the Phone board states it; a metric a phone cannot measure is withheld with its reason. It is served private, no-store and is a 404 where the Phone board is not shown; a board that does not exist is a 400. The phone ranking is GET /api/leaderboard?board=phone.
Rounding
Every number is rounded to six significant digits.
Withheld values
When a coverage rule refuses a value, measured is false, reason says why (“fewer than 30 error runs”) and n is what was observed against the floor. An unmeasured value is not a zero.
Action mix keys
verb_mix is keyed by action class, in this order: navigate, point, drag, scroll, type, key, switch, look, code, finish. A click and a tap are one class; a refused tool call is in none.
Computer metricsPhone metricsLeaderboardAPI documentation