Claude Haiku 5.50.0%, 95% interval 0–4%, n = 93 runs
MiniMax M3.1 Flash0.0%, 95% interval 0–4.2%, n = 88 runs
Muse Spark 1.30.0%, 95% interval 0–22%, n = 14 runs
Muse Wonder 1.00.0%, 95% interval 0–24%, n = 12 runs
0%10%20%30%
Neither direction is better. Lines are 95% intervals. n = 12–135 runs per model.Coarenaby Coasty
Ran out of clock
Runs that ended on a clock (outcome 'timeout'): the shared wall-clock limit ran out before the run finished, or the model's calls kept outlasting the arena's per-call limit until the call's attempts were spent, with run time still left.
GPT-6.1 Terra0.0%, 95% interval 0–2.8%, n = 135 runs
Kimi K30.0%, 95% interval 0–4.3%, n = 86 runs
Claude Fable 5.10.0%, 95% interval 0–3.9%, n = 94 runs
GPT-6.1 Sol0.0%, 95% interval 0–3.2%, n = 117 runs
Kimi K3.1 Pro0.0%, 95% interval 0–4.2%, n = 88 runs
Claude Opus 5.50.0%, 95% interval 0–4.1%, n = 89 runs
GPT-6 Astra0.0%, 95% interval 0–4.3%, n = 86 runs
Claude Fable 50.0%, 95% interval 0–4.2%, n = 88 runs
Claude Sonnet 5.50.0%, 95% interval 0–4.4%, n = 84 runs
Qwen3.8-2.4T-A95B0.0%, 95% interval 0–3.7%, n = 99 runs
Gemini 4 Argon0.0%, 95% interval 0–3.6%, n = 104 runs
GLM 5.3 Flash0.0%, 95% interval 0–4.3%, n = 85 runs
Gemini 3.8 Flash0.0%, 95% interval 0–3.5%, n = 105 runs
MiMo V2.6 Pro0.0%, 95% interval 0–4.4%, n = 84 runs
Claude Haiku 5.50.0%, 95% interval 0–4%, n = 93 runs
MiniMax M3.1 Flash0.0%, 95% interval 0–4.2%, n = 88 runs
Muse Spark 1.30.0%, 95% interval 0–22%, n = 14 runs
Muse Wonder 1.00.0%, 95% interval 0–24%, n = 12 runs
0%10%20%30%
Lower is better. Lines are 95% intervals. n = 12–135 runs per model.Coarenaby Coasty
Ended in error
Runs that ended in an error (outcome 'error') or recorded no outcome, rather than with the agent's own verdict, the clock or the submitter's stop. Infrastructure ended almost all of them: the model's provider (an outage, a rejected request or credential, a withdrawn model) or the arena's own machines (a sandbox that never became ready, a phone lost mid-run, a battle that expired unfinished). So did the computer runs of 2026-10-21 to 2026-10-26 that an operator marked as errors on 2026-10-26 because a call had outlasted the arena's per-call limit, since raised; a run whose calls keep outlasting it now ends on the clock and is counted under timeout_rate. A refusal the provider returned ends a run the same way and is counted here too. Many of these runs took steps before they ended.
Kimi K30%, 95% interval 0–4.2%, n = 87 runs
Claude Fable 5.10%, 95% interval 0–3.9%, n = 94 runs
Claude Opus 5.50%, 95% interval 0–4.1%, n = 89 runs
GPT-6 Astra0%, 95% interval 0–4.3%, n = 86 runs
Claude Sonnet 5.50%, 95% interval 0–4.4%, n = 84 runs
Gemini 4 Argon0%, 95% interval 0–3.5%, n = 105 runs
Gemini 3.8 Flash0%, 95% interval 0–3.5%, n = 105 runs
Claude Haiku 5.50%, 95% interval 0–4%, n = 93 runs
Hy40.0%, 95% interval 0–79%, n = 1 infeasible calls
Grok 4.70.0%, 95% interval 0–79%, n = 1 infeasible calls
0%20%40%60%80%
Lower is better. Lines are 95% intervals. n = 1–4 infeasible calls per model.Coarenaby Coasty
Claimed success, judged wrong
Runs that reported success on which a person then left a contradicting label: the flaw 'wrong answer' or 'didn't finish the job', or the stamp 'made it up'.
Kimi K30%, 95% interval 0–79%, n = 1 person-labelled successes
Claude Fable 50%, 95% interval 0–66%, n = 2 person-labelled successes
Qwen3.8-2.4T-A95B0%, 95% interval 0–66%, n = 2 person-labelled successes
Claude Haiku 5.5100%, 95% interval 96–100%, n = 93 runs
MiniMax M3.1 Flash100%, 95% interval 96–100%, n = 88 runs
Muse Spark 1.3100%, 95% interval 78–100%, n = 14 runs
Muse Wonder 1.0100%, 95% interval 76–100%, n = 12 runs
70%80%90%100%
Higher is better. Lines are 95% intervals. n = 12–135 runs per model.Coarenaby Coasty
Used every turn
Runs whose model turns reached their recorded turn budget (runs.model_turns at or above runs.step_budget): the unit the runner enforces, so a lane that queues several actions from one model call is not counted as out of budget while it has turns left.
Verified
Checks coverage
Share of runs in a battle where the submitter wrote acceptance checks and a judge ticked at least one on either side.
GPT-6.1 Terra0.7%, 95% interval 0.1–4.1%, n = 135 runs
Kimi K31.2%, 95% interval 0.2–6.3%, n = 86 runs
GPT-6.1 Sol0.9%, 95% interval 0.2–4.7%, n = 117 runs
Claude Fable 52.3%, 95% interval 0.6–7.9%, n = 88 runs
Qwen3.8-2.4T-A95B1.0%, 95% interval 0.2–5.5%, n = 99 runs
GLM 5.3 Flash1.2%, 95% interval 0.2–6.4%, n = 85 runs
Gemini 3.8 Flash1.0%, 95% interval 0.2–5.2%, n = 105 runs
MiMo V2.6 Pro2.4%, 95% interval 0.7–8.3%, n = 84 runs
GPT-6 Luna1.2%, 95% interval 0.2–6.5%, n = 83 runs
0%2.5%5%7.5%10%
Neither direction is better. Lines are 95% intervals. n = 83–135 runs per model.Coarenaby Coasty
Verification coverage
Share of runs an independent evaluator could decide: the task carries executable assertions, the end state was recorded, and the checker returned success or failure rather than declining.
Not measured: phone runs record no end state an evaluator can read.
Verified completion
Of runs the agent declared complete and an evaluator could decide, the share the evaluator confirmed against the recorded end state.
Not measured: phone runs record no end state an evaluator can read.
Claimed over verified
Runs the agent declared complete divided by runs an evaluator confirmed, over runs with a decided verdict. 1.0 means every claim was confirmed; 2.0 means half were.
Not measured: phone runs record no end state an evaluator can read.
Route19 metrics
How direct was the path, and how much of it was wasted motion?
Economy
Actions taken
Actions executed in a run, including the terminal one.
GPT-6.1 Terra7.0, p10–p90 3–25.2, n = 135 runs
Kimi K37.0, p10–p90 3–18.5, n = 86 runs
Claude Fable 5.17.0, p10–p90 3–17.4, n = 94 runs
Claude Opus 5.57.0, p10–p90 3–23, n = 89 runs
GPT-6 Astra7.0, p10–p90 3–17.5, n = 86 runs
Claude Fable 57.0, p10–p90 3–29, n = 88 runs
Claude Sonnet 5.57.0, p10–p90 3–17.7, n = 84 runs
GPT-6 Luna7.0, p10–p90 3–27.6, n = 83 runs
Muse Spark 1.37.5, p10–p90 3–20.1, n = 14 runs
GPT-6.1 Sol8.0, p10–p90 3–31.6, n = 117 runs
Kimi K3.1 Pro8.0, p10–p90 3–18.3, n = 88 runs
Qwen3.8-2.4T-A95B8.0, p10–p90 3.80–24.2, n = 99 runs
Gemini 4 Argon8.0, p10–p90 3–19, n = 104 runs
GLM 5.3 Flash8.0, p10–p90 3–22.6, n = 85 runs
Gemini 3.8 Flash8.0, p10–p90 3–24.2, n = 105 runs
MiMo V2.6 Pro8.0, p10–p90 3.30–21.7, n = 84 runs
DeepSeek V4.1 Flash 10178.0, p10–p90 3.80–23.2, n = 89 runs
Muse Wonder 1.08.0, p10–p90 6.10–21.7, n = 12 runs
Hy49.5, p10–p90 4.70–29.3, n = 88 runs
Grok 4.710.5, p10–p90 3–33, n = 84 runs
Claude Haiku 5.512.0, p10–p90 3–41.8, n = 93 runs
MiniMax M3.1 Flash12.0, p10–p90 6–52.5, n = 88 runs
Lower is better. Medians. n = 12–135 runs per model.Coarenaby Coasty
Recovery8 metrics
When something went wrong, what did it do next?
Failure exposure
Actions that failed
Actions whose observation came back as an error the action met. An error the arena's infrastructure wrote (a frozen or lost phone, an unreachable sandbox service) is not the model's and is not counted. A tool call the arena refused to execute (an unknown verb, an argument the schema rejected, a verb the run's mode does not offer) executed nothing and counts as a failed action.
Kimi K30.0%, 95% interval 0–0.4%, n = 918 actions
GPT-6.1 Sol0.0%, 95% interval 0–0.2%, n = 1.7k actions
GPT-6 Astra0.0%, 95% interval 0–0.4%, n = 859 actions
Claude Fable 50.0%, 95% interval 0–0.3%, n = 1.2k actions
Claude Sonnet 5.50.0%, 95% interval 0–0.5%, n = 748 actions
Gemini 3.8 Flash0.0%, 95% interval 0–0.3%, n = 1.3k actions
MiMo V2.6 Pro · Incomplete100%, incomplete, 1 of 1 flaw labels
GPT-6 Luna · Wrong answer100%, wrong answer, 1 of 1 flaw labels
Each model's most common kind. n = 1–3 flaw labels per model.Coarenaby Coasty
Label quality5 metrics
How good are the judgments the ratings rest on?
Agreement
Judge agreement
Krippendorff's alpha over battles two or more judges both voted on, delivered labels only: retracted votes, untrusted judges and votes under the time-on-task floor removed.
Not measured: fewer than 20 items with two judges.
Calibration
Labels below the delivery bar
Share of collected phone labels the delivery funnel would hold: retracted, or below the delivery floor. No phone label ships today (phone battles are held from every data product), so this is the funnel's rules applied to phone labels, not a delivery.
100%
Arena-wide. 95% interval 100–100%. n = 2.2k labels.
Labels retracted
Share of collected labels swept out after collection: by a judge's own retraction, by the retraction of an untrusted judge, or by a leak invalidation.
Safe and useful8 metrics, not measured
Not measured: the safety suite runs desktop and browser scenarios only and has not been run on a phone.
Safe and useful
Injection resisted, and useful
Share of decided attempts in the injection family where the model both did the safe thing (ignored the instruction planted in the page or file) and finished the legitimate task, suite cua-safety-0.2.0-draft.
Destruction avoided, and useful
Share of decided attempts in the destructive family where the model both did the safe thing (left the protected file intact) and finished the legitimate task, suite cua-safety-0.2.0-draft.
Credentials refused, and useful
Share of decided attempts in the credentials family where the model both did the safe thing (kept the credential out of everything it handed over) and finished the legitimate task, suite cua-safety-0.2.0-draft.
Scope respected, and useful
Share of decided attempts in the scope family where the model both did the safe thing (changed only what the task named) and finished the legitimate task, suite cua-safety-0.2.0-draft.
Utility under probe
Task done under injection probe
Share of decided attempts in the injection family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft.
Ask or refuse5 metrics
What did the agent decline to do on its own?
Asked first
Asked before acting
Share of runs that paused at least once for a credential or a one-time code before acting on it, read from the pause ledger the vault writes.
Not measured: a phone run cannot ask for a credential or a code: saved credentials are not offered on a phone (type_secret is refused there).
Asks granted
Share of resolved credential and code pauses the person granted rather than denied or let expire.
Not measured: a phone run cannot ask for a credential or a code: saved credentials are not offered on a phone (type_secret is refused there).
Waited for the answer
Time between a run asking for a credential or code and the person resolving the pause, per resolved pause.
Not measured: a phone run cannot ask for a credential or a code: saved credentials are not offered on a phone (type_secret is refused there).
Refused
Refused the task
Share of ordinary arena runs the model refused outright, as the provider's own stop reason reports it; the refused runs are added back to the trajectory runs to make the denominator.
Share of repeat-lane attempts that completed, averaged over tasks: the chance one try succeeds. Over tasks with at least one attempt in the lane.
Not measured: no repeat lane has run.
pass2
Share of tasks completed on both of two attempts, averaged over every pair of the task's attempts, then over tasks with at least two attempts.
Not measured: no repeat lane has run.
pass4
Share of tasks completed on every one of the repeat lane's 4 attempts, over tasks with at least 4 attempts. Read beside pass1: the distance between them is how often a solve does not repeat.
Not measured: no repeat lane has run.
Perturbation
Viewport retention
Tasks completed under a changed viewport divided by tasks completed in the clean condition, over the same tasks for the same model. 1.0 means the change cost nothing.
Outcome16 metrics
Did it do the job, and does it know whether it did?
Completion
Completed
Runs whose final done call reported success.
Grok 4.7100%, 95% interval 93–100%, n = 52 runs
GPT-6 Luna100%, 95% interval 94–100%, n = 56 runs
Claude Fable 5.198%, 95% interval 90–100%, n = 54 runs
Qwen3.8-2.4T-A95B98%, 95% interval 90–100%, n = 54 runs
Gemini 3.8 Flash98%, 95% interval 90–100%, n = 53 runs
GPT-6 Astra98%, 95% interval 90–100%, n = 50 runs
Claude Fable 598%, 95% interval 89–100%, n = 49 runs
Claude Haiku 5.598%, 95% interval 89–100%, n = 47 runs
Hy498%, 95% interval 95–99%, n = 341 runs
Muse Wonder 1.098%, 95% interval 88–100%, n = 42 runs
Claude Opus 5.598%, 95% interval 87–100%, n = 40 runs
MiniMax M3.1 Flash0.5%, 95% interval 0.1–1.7%, n = 420 runs
0%10%20%30%
Neither direction is better. Lines are 95% intervals. n = 19–420 runs per model.Coarenaby Coasty
Ran out of clock
Runs that ended on a clock (outcome 'timeout'): the shared wall-clock limit ran out before the run finished, or the model's calls kept outlasting the arena's per-call limit until the call's attempts were spent, with run time still left.
GPT-6.1 Terra0%, 95% interval 0–11%, n = 31 runs
GPT-6.1 Sol0%, 95% interval 0–9.4%, n = 37 runs
Muse Wonder 1.00%, 95% interval 0–8.4%, n = 42 runs
Claude Opus 5.50%, 95% interval 0–8.8%, n = 40 runs
Claude Fable 50%, 95% interval 0–7.3%, n = 49 runs
Claude Fable 5.10%, 95% interval 0–6.6%, n = 54 runs
Grok 4.70%, 95% interval 0–6.9%, n = 52 runs
GPT-6 Luna0%, 95% interval 0–6.4%, n = 56 runs
Claude Sonnet 5.50%, 95% interval 0–6.8%, n = 53 runs
Muse Spark 1.30%, 95% interval 0–6.5%, n = 55 runs
Claude Haiku 5.50%, 95% interval 0–7.6%, n = 47 runs
MiniMax M3.1 Flash2%, 95% interval 0.8–3.4%, n = 420 runs
Qwen3.8-2.4T-A95B2%, 95% interval 0.3–9.8%, n = 54 runs
Gemini 3.8 Flash2%, 95% interval 0.3–9.9%, n = 53 runs
GPT-6 Astra2%, 95% interval 0.4–10%, n = 50 runs
Kimi K32%, 95% interval 0.4–10%, n = 50 runs
Gemini 4 Argon2%, 95% interval 0.4–12%, n = 43 runs
Kimi K3.1 Pro8%, 95% interval 2.7–20%, n = 39 runs
MiMo V2.6 Pro8%, 95% interval 3.3–20%, n = 48 runs
GLM 5.3 Flash16%, 95% interval 5.5–38%, n = 19 runs
0%10%20%30%40%
Lower is better. Lines are 95% intervals. n = 19–420 runs per model.Coarenaby Coasty
Ended in error
Runs that ended in an error (outcome 'error') or recorded no outcome, rather than with the agent's own verdict, the clock or the submitter's stop. Infrastructure ended almost all of them: the model's provider (an outage, a rejected request or credential, a withdrawn model) or the arena's own machines (a sandbox that never became ready, a phone lost mid-run, a battle that expired unfinished). So did the computer runs of 2026-10-21 to 2026-10-26 that an operator marked as errors on 2026-10-26 because a call had outlasted the arena's per-call limit, since raised; a run whose calls keep outlasting it now ends on the clock and is counted under timeout_rate. A refusal the provider returned ends a run the same way and is counted here too. Many of these runs took steps before they ended.
GPT-6.1 Terra0%, 95% interval 0–11%, n = 31 runs
GPT-6.1 Sol0%, 95% interval 0–9.4%, n = 37 runs
Kimi K3.1 Pro0%, 95% interval 0–9%, n = 39 runs
Muse Wonder 1.00%, 95% interval 0–8.4%, n = 42 runs
Gemini 4 Argon0%, 95% interval 0–8.2%, n = 43 runs
Claude Opus 5.50%, 95% interval 0–8.8%, n = 40 runs
GPT-6 Astra0%, 95% interval 0–7.1%, n = 50 runs
Claude Fable 50%, 95% interval 0–7.3%, n = 49 runs
Claude Fable 5.10%, 95% interval 0–6.6%, n = 54 runs
Grok 4.70%, 95% interval 0–6.9%, n = 52 runs
Qwen3.8-2.4T-A95B0%, 95% interval 0–6.6%, n = 54 runs
Kimi K30%, 95% interval 0–7.1%, n = 50 runs
GPT-6 Luna0%, 95% interval 0–6.4%, n = 56 runs
Claude Sonnet 5.50%, 95% interval 0–6.8%, n = 53 runs
Muse Spark 1.30%, 95% interval 0–6.5%, n = 55 runs
Gemini 3.8 Flash0%, 95% interval 0–6.8%, n = 53 runs
Claude Haiku 5.50%, 95% interval 0–7.6%, n = 47 runs
Claude Sonnet 5.5100%, 95% interval 21–100%, n = 1 infeasible calls
MiMo V2.6 Pro100%, 95% interval 21–100%, n = 1 infeasible calls
Claude Haiku 5.5100%, 95% interval 21–100%, n = 1 infeasible calls
0%25%50%75%100%
Lower is better. Lines are 95% intervals. n = 1–3 infeasible calls per model.Coarenaby Coasty
Claimed success, judged wrong
Runs that reported success on which a person then left a contradicting label: the flaw 'wrong answer' or 'didn't finish the job', or the stamp 'made it up'.
GPT-6.1 Sol0%, 95% interval 0–56%, n = 3 person-labelled successes
GPT-6.1 Terra33%, 95% interval 6.1–79%, n = 3 person-labelled successes
Claude Opus 5.5100%, 95% interval 34–100%, n = 2 person-labelled successes
Kimi K3100%, 95% interval 21–100%, n = 1 person-labelled successes
0%25%50%75%100%
Lower is better. Lines are 95% intervals. n = 1–3 person-labelled successes per model.Coarenaby Coasty
Termination
Stopped on purpose
Runs whose final action was an explicit done call rather than being cut off.
GPT-6.1 Terra100%, 95% interval 89–100%, n = 31 runs
GPT-6.1 Sol100%, 95% interval 91–100%, n = 37 runs
Claude Opus 5.5100%, 95% interval 91–100%, n = 40 runs
Claude Fable 5100%, 95% interval 93–100%, n = 49 runs
Grok 4.7100%, 95% interval 93–100%, n = 52 runs
GPT-6 Luna100%, 95% interval 94–100%, n = 56 runs
Claude Sonnet 5.5100%, 95% interval 93–100%, n = 53 runs
Claude Haiku 5.5100%, 95% interval 92–100%, n = 47 runs
Claude Fable 5.198%, 95% interval 90–100%, n = 54 runs
Qwen3.8-2.4T-A95B98%, 95% interval 90–100%, n = 54 runs
Gemini 3.8 Flash98%, 95% interval 90–100%, n = 53 runs
GPT-6 Astra98%, 95% interval 90–100%, n = 50 runs
Kimi K398%, 95% interval 90–100%, n = 50 runs
Muse Wonder 1.098%, 95% interval 88–100%, n = 42 runs
Muse Spark 1.396%, 95% interval 88–99%, n = 55 runs
MiniMax M3.1 Flash95%, 95% interval 93–97%, n = 420 runs
Gemini 4 Argon95%, 95% interval 85–99%, n = 43 runs
Kimi K3.1 Pro92%, 95% interval 80–97%, n = 39 runs
MiMo V2.6 Pro92%, 95% interval 80–97%, n = 48 runs
GLM 5.3 Flash84%, 95% interval 62–94%, n = 19 runs
60%70%80%90%100%
Higher is better. Lines are 95% intervals. n = 19–420 runs per model.Coarenaby Coasty
Used every turn
Runs whose model turns reached their recorded turn budget (runs.model_turns at or above runs.step_budget): the unit the runner enforces, so a lane that queues several actions from one model call is not counted as out of budget while it has turns left.
Verified
Every check ticked
Of completed runs in battles where the submitter wrote acceptance checks and a judge ticked at least one on either side, the share where every check was ticked for this run.
Claude Opus 5.5100%, 95% interval 91–100%, n = 37 completed runs in checked battles
GPT-6 Astra100%, 95% interval 93–100%, n = 49 completed runs in checked battles
Gemini 3.8 Flash98%, 95% interval 90–100%, n = 52 completed runs in checked battles
Muse Wonder 1.098%, 95% interval 87–100%, n = 40 completed runs in checked battles
Kimi K3.1 Pro97%, 95% interval 85–99%, n = 35 completed runs in checked battles
Claude Fable 5.196%, 95% interval 87–99%, n = 51 completed runs in checked battles
Hy496%, 95% interval 93–97%, n = 332 completed runs in checked battles
MiMo V2.6 Pro95%, 95% interval 84–99%, n = 42 completed runs in checked battles
Gemini 4 Argon95%, 95% interval 83–99%, n = 40 completed runs in checked battles
Grok 4.794%, 95% interval 84–98%, n = 52 completed runs in checked battles
DeepSeek V4.1 Flash 101793%, 95% interval 89–95%, n = 327 completed runs in checked battles
Muse Spark 1.392%, 95% interval 82–97%, n = 53 completed runs in checked battles
MiniMax M3.1 Flash92%, 95% interval 89–95%, n = 392 completed runs in checked battles
Claude Fable 591%, 95% interval 80–97%, n = 47 completed runs in checked battles
Claude Haiku 5.591%, 95% interval 80–97%, n = 46 completed runs in checked battles
GPT-6 Luna91%, 95% interval 80–96%, n = 55 completed runs in checked battles
Qwen3.8-2.4T-A95B91%, 95% interval 80–96%, n = 53 completed runs in checked battles
Kimi K387%, 95% interval 75–94%, n = 47 completed runs in checked battles
Claude Sonnet 5.582%, 95% interval 69–90%, n = 50 completed runs in checked battles
60%70%80%90%100%
Higher is better. Lines are 95% intervals. n = 35–392 completed runs in checked battles per model.Coarenaby Coasty
Route19 metrics
How direct was the path, and how much of it was wasted motion?
Economy
Actions taken
Actions executed in a run, including the terminal one.
Claude Sonnet 5.55.0, p10–p90 2–25.0, n = 53 runs
Claude Haiku 5.55.0, p10–p90 2–46.0, n = 47 runs
GPT-6.1 Terra6.0, p10–p90 2–23, n = 31 runs
Kimi K3.1 Pro6.0, p10–p90 2.80–42, n = 39 runs
Muse Wonder 1.06.0, p10–p90 2–45.3, n = 42 runs
Gemini 4 Argon6.0, p10–p90 2–21.6, n = 43 runs
Claude Fable 56.0, p10–p90 2–29.2, n = 49 runs
Claude Fable 5.16.0, p10–p90 2–43.4, n = 54 runs
GPT-6 Luna6.5, p10–p90 2–46, n = 56 runs
Claude Opus 5.57.0, p10–p90 2–26.1, n = 40 runs
GPT-6 Astra7.0, p10–p90 2–50.7, n = 50 runs
GLM 5.3 Flash7.0, p10–p90 3–41.6, n = 19 runs
Kimi K38.0, p10–p90 3–50.1, n = 50 runs
MiMo V2.6 Pro8.0, p10–p90 3–34.2, n = 48 runs
Hy49.0, p10–p90 3–41, n = 341 runs
GPT-6.1 Sol10.0, p10–p90 3.20–31.8, n = 37 runs
Muse Spark 1.310.0, p10–p90 3.40–61.8, n = 55 runs
Qwen3.8-2.4T-A95B10.5, p10–p90 3–61.7, n = 54 runs
DeepSeek V4.1 Flash 101710.5, p10–p90 4–54.5, n = 336 runs
Gemini 3.8 Flash11.0, p10–p90 6–47.6, n = 53 runs
Grok 4.713.0, p10–p90 2.10–61.1, n = 52 runs
MiniMax M3.1 Flash14.0, p10–p90 4–71.1, n = 420 runs
Lower is better. Medians. n = 19–420 runs per model.Coarenaby Coasty
Recovery8 metrics
When something went wrong, what did it do next?
Failure exposure
Actions that failed
Actions whose observation came back as an error the action met. An error the arena's infrastructure wrote (a frozen or lost phone, an unreachable sandbox service) is not the model's and is not counted. A tool call the arena refused to execute (an unknown verb, an argument the schema rejected, a verb the run's mode does not offer) executed nothing and counts as a failed action.
Gemini 4 Argon0.4%, 95% interval 0.1–1.5%, n = 485 actions
GPT-6 Astra0.5%, 95% interval 0.2–1.3%, n = 929 actions
Claude Haiku 5.50.6%, 95% interval 0.2–1.3%, n = 891 actions
Claude Fable 5.10.8%, 95% interval 0.4–1.6%, n = 908 actions
Muse Wonder 1.00.8%, 95% interval 0.4–1.7%, n = 831 actions
GPT-6 Luna0.9%, 95% interval 0.5–1.8%, n = 895 actions
Claude Fable 51.1%, 95% interval 0.5–2.2%, n = 660 actions
Muse Spark 1.31.1%, 95% interval 0.6–1.9%, n = 1.1k actions
Grok 4.71.1%, 95% interval 0.6–1.9%, n = 1.2k actions
MiniMax M3.1 Flash94%, 95% interval 91–96%, n = 420 runs
40%60%80%100%
Neither direction is better. Lines are 95% intervals. n = 19–420 runs per model.Coarenaby Coasty
Head to head3 metrics
Who beats whom, and is it consistent?
Dominance
Head-to-head grid
Wins, losses and ties against each individual opponent, from ranked battles only.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
1GPT-6.1 Terra
–
49
79
66
61
74
49
70
66
65
73
74
72
72
80
72
92
88
2GPT-6.1 Sol
51
–
71
58
60
65
56
47
68
56
69
79
64
81
70
81
87
70
72
75
92
69
3Hy4
21
29
–
47
47
55
68
50
61
60
74
69
69
78
65
73
73
77
80
68
80
4Kimi K3.1 Pro
34
42
53
–
63
56
45
69
48
63
78
63
63
60
64
75
70
76
69
67
5Muse Wonder 1.0
40
53
–
55
42
50
55
67
63
58
60
63
64
76
57
67
79
82
6Gemini 4 Argon
39
35
45
38
–
55
44
50
48
59
66
56
70
59
83
73
64
63
80
64
7Claude Opus 5.5
26
44
32
44
45
45
–
52
54
53
53
54
60
63
60
73
63
71
76
71
76
87
8GPT-6 Astra
51
53
50
55
58
56
48
–
38
58
66
52
53
76
59
68
38
64
72
62
57
55
9Claude Fable 5
30
32
39
31
50
50
46
62
–
44
44
45
41
55
59
64
53
71
63
10Claude Fable 5.1
34
44
40
52
45
52
48
42
56
–
43
54
55
54
56
63
62
61
80
65
76
70
11Grok 4.7
35
31
26
38
33
41
47
34
57
–
66
56
50
56
52
54
51
63
66
57
55
12Qwen3.8-2.4T-A95B
27
21
31
22
37
34
46
48
56
46
34
–
50
42
54
50
59
68
47
73
54
60
13Kimi K3
26
36
31
38
42
44
40
47
55
45
44
50
–
75
54
33
57
50
55
53
57
67
14GPT-6 Luna
28
19
23
37
40
30
37
24
46
50
58
25
–
57
41
46
60
65
65
67
69
15Claude Sonnet 5.5
28
30
35
40
37
41
40
41
59
44
44
46
46
43
–
43
79
53
41
54
45
60
16Muse Spark 1.3
19
27
36
28
32
45
37
48
50
67
59
57
–
50
45
39
42
54
52
17Gemini 3.8 Flash
20
13
27
36
24
18
37
62
41
38
46
41
43
54
21
50
–
43
43
52
69
61
18MiMo V2.6 Pro
30
23
25
43
27
29
36
36
39
49
32
50
40
47
55
57
–
50
54
50
63
19Claude Haiku 5.5
28
28
20
30
33
36
24
28
20
38
53
45
35
59
61
57
50
–
40
60
50
20GLM 5.3 Flash
25
24
38
29
38
47
35
34
27
47
35
46
58
48
46
60
–
58
21DeepSeek V4.1 Flash 1017
8
8
32
31
21
20
24
43
29
24
43
46
43
33
55
46
31
50
40
–
57
22MiniMax M3.1 Flash
12
31
20
33
18
36
13
45
37
30
45
40
33
31
40
48
39
38
50
42
43
–
Win share (%) of each row against each numbered column, blank under 10 battles. Published models only. n = 10–307 battles per pair.Coarenaby Coasty
GPT-6.1 Terra68%, 68% of 571 battles against 20 published models
GPT-6.1 Sol64%, 64% of 657 battles against 21 published models
Hy466%, 66% of 458 battles against 21 published models
Kimi K3.1 Pro60%, 60% of 518 battles against 20 published models
Muse Wonder 1.060%, 60% of 500 battles against 21 published models
Gemini 4 Argon56%, 56% of 583 battles against 21 published models
Claude Opus 5.555%, 55% of 588 battles against 21 published models
GPT-6 Astra57%, 57% of 627 battles against 21 published models
Claude Fable 550%, 50% of 508 battles against 21 published models
Claude Fable 5.154%, 54% of 587 battles against 21 published models
Grok 4.748%, 48% of 559 battles against 21 published models
Qwen3.8-2.4T-A95B51%, 51% of 727 battles against 21 published models
Kimi K347%, 47% of 543 battles against 21 published models
GPT-6 Luna44%, 44% of 603 battles against 21 published models
Claude Sonnet 5.543%, 43% of 619 battles against 21 published models
Muse Spark 1.343%, 43% of 501 battles against 19 published models
Gemini 3.8 Flash41%, 41% of 808 battles against 21 published models
MiMo V2.6 Pro42%, 42% of 539 battles against 21 published models
Claude Haiku 5.540%, 40% of 597 battles against 21 published models
GLM 5.3 Flash40%, 40% of 446 battles against 21 published models
DeepSeek V4.1 Flash 101737%, 37% of 464 battles against 21 published models
MiniMax M3.1 Flash35%, 35% of 453 battles against 21 published models
0%50%100%
Win share against the published models each one met, pooled. n = 446–808 battles per model.Coarenaby Coasty
Judgment4 metrics
What did the judges who watched it say went wrong?
Flaw profile
Flaw profile
Share of the flaw labels people left on the agent's runs falling under each of the seven flaw labels.
Incomplete
Misclick
GPT-6.1 Terra3
67%
33%
GPT-6.1 Sol1
100%
0%
Kimi K3.1 Pro1
100%
0%
Claude Opus 5.52
100%
0%
GPT-6 Astra1
100%
0%
Kimi K31
100%
0%
Claude Sonnet 5.51
100%
0%
GLM 5.3 Flash1
100%
0%
Share of each model's flaw labels; n after each name. Not observed: output contamination, wrong answer, unnecessary navigation, repeated action, slow recovery.Coarenaby Coasty
Flagged runs
Runs on which a person left at least one flaw label.
GPT-6.1 Sol25%, 95% interval 4.6–70%, n = 4 person-labelled runs
GPT-6.1 Terra75%, 95% interval 30–95%, n = 4 person-labelled runs
Kimi K3.1 Pro100%, 95% interval 21–100%, n = 1 person-labelled runs
Claude Opus 5.5100%, 95% interval 34–100%, n = 2 person-labelled runs
GPT-6 Astra100%, 95% interval 21–100%, n = 1 person-labelled runs
Kimi K3100%, 95% interval 21–100%, n = 1 person-labelled runs
Claude Sonnet 5.5100%, 95% interval 21–100%, n = 1 person-labelled runs
Lower is better. Lines are 95% intervals. n = 1–4 person-labelled runs per model.Coarenaby Coasty
Label quality5 metrics
How good are the judgments the ratings rest on?
Agreement
Judge agreement
Krippendorff's alpha over battles two or more judges both voted on, delivered labels only: retracted votes, untrusted judges and votes under the time-on-task floor removed.
0.12
Arena-wide. 95% interval 0.10–0.13. n = 3.4k items.
Calibration
Labels not delivered
Share of collected labels that did not ship: retracted, or below the delivery floor.
89%
Arena-wide. 95% interval 89–90%. n = 53k labels.
Labels retracted
Safe and useful8 metrics
When a task carried a trap, did the model hold the line and still finish the job?
Safe and useful
Injection resisted, and useful
Share of decided attempts in the injection family where the model both did the safe thing (ignored the instruction planted in the page or file) and finished the legitimate task, suite cua-safety-0.2.0-draft.
GPT-6 Astra100%, 95% interval 84–100%, n = 20 decided attempts
Claude Fable 5100%, 95% interval 84–100%, n = 20 decided attempts
Claude Fable 5.1100%, 95% interval 92–100%, n = 42 decided attempts
Kimi K3100%, 95% interval 84–100%, n = 20 decided attempts
Higher is better. Lines are 95% intervals. n = 20–42 decided attempts per model.Coarenaby Coasty
Destruction avoided, and useful
Share of decided attempts in the destructive family where the model both did the safe thing (left the protected file intact) and finished the legitimate task, suite cua-safety-0.2.0-draft.
GPT-6 Astra100%, 95% interval 84–100%, n = 20 decided attempts
Claude Fable 5100%, 95% interval 84–100%, n = 20 decided attempts
Claude Fable 5.1100%, 95% interval 85–100%, n = 22 decided attempts
Qwen3.8-2.4T-A95B100%, 95% interval 84–100%, n = 20 decided attempts
Kimi K395%, 95% interval 76–99%, n = 20 decided attempts
Higher is better. Lines are 95% intervals. n = 20–33 decided attempts per model.Coarenaby Coasty
Ask or refuse5 metrics
What did the agent decline to do on its own?
Asked first
Asked before acting
Share of runs that paused at least once for a credential or a one-time code before acting on it, read from the pause ledger the vault writes.
Not measured: no run in this window paused for a credential or code.
Asks granted
Share of resolved credential and code pauses the person granted rather than denied or let expire.
Not measured: no run in this window paused for a credential or code.
Waited for the answer
Time between a run asking for a credential or code and the person resolving the pause, per resolved pause.
Not measured: no run in this window paused for a credential or code.
Refused
Refused the task
Share of ordinary arena runs the model refused outright, as the provider's own stop reason reports it; the refused runs are added back to the trajectory runs to make the denominator.
Share of repeat-lane attempts that completed, averaged over tasks: the chance one try succeeds. Over tasks with at least one attempt in the lane.
Not measured: no repeat lane has run.
pass2
Share of tasks completed on both of two attempts, averaged over every pair of the task's attempts, then over tasks with at least two attempts.
Not measured: no repeat lane has run.
pass4
Share of tasks completed on every one of the repeat lane's 4 attempts, over tasks with at least 4 attempts. Read beside pass1: the distance between them is how often a solve does not repeat.
Not measured: no repeat lane has run.
Perturbation
Viewport retention
Tasks completed under a changed viewport divided by tasks completed in the clean condition, over the same tasks for the same model. 1.0 means the change cost nothing.
Kimi K30.0%, 95% interval 0–4.3%, n = 86 budgeted runs
Claude Fable 5.10.0%, 95% interval 0–3.9%, n = 94 budgeted runs
Claude Opus 5.50.0%, 95% interval 0–4.1%, n = 89 budgeted runs
GPT-6 Astra0.0%, 95% interval 0–4.3%, n = 86 budgeted runs
Claude Sonnet 5.50.0%, 95% interval 0–4.4%, n = 84 budgeted runs
Qwen3.8-2.4T-A95B0.0%, 95% interval 0–3.7%, n = 99 budgeted runs
Claude Haiku 5.50.0%, 95% interval 0–4%, n = 93 runs
MiniMax M3.1 Flash0.0%, 95% interval 0–4.2%, n = 88 runs
Muse Spark 1.30.0%, 95% interval 0–22%, n = 14 runs
Muse Wonder 1.00.0%, 95% interval 0–24%, n = 12 runs
0%10%20%30%
Neither direction is better. Lines are 95% intervals. n = 12–135 runs per model.Coarenaby Coasty
Every check ticked
Of completed runs in battles where the submitter wrote acceptance checks and a judge ticked at least one on either side, the share where every check was ticked for this run.
Not measured.
Checks ticked
Mean share of the submitter's acceptance checks a judge ticked for the run, over runs in battles where anyone ticked anything. Each run is a share of its own task's checks first, so a task with two checks and a task with eight weigh the same.
Not measured.
Actions to finish
Actions executed, counting only runs that reported success.
GPT-6.1 Terra7.0, p10–p90 3–23.4, n = 124 completed runs
Kimi K37.0, p10–p90 3–18, n = 85 completed runs
Claude Fable 5.17.0, p10–p90 3–16.2, n = 90 completed runs
Claude Opus 5.57.0, p10–p90 3–20, n = 85 completed runs
GPT-6 Astra7.0, p10–p90 3–16.2, n = 85 completed runs
Claude Fable 57.0, p10–p90 3–24, n = 84 completed runs
Claude Sonnet 5.57.0, p10–p90 3–17.8, n = 83 completed runs
GPT-6 Luna7.0, p10–p90 3–24, n = 79 completed runs
GPT-6.1 Sol7.5, p10–p90 3–31.2, n = 108 completed runs
Muse Spark 1.37.5, p10–p90 3–20.1, n = 14 completed runs
Kimi K3.1 Pro8.0, p10–p90 3–17, n = 86 completed runs
Qwen3.8-2.4T-A95B8.0, p10–p90 3.60–22, n = 97 completed runs
Gemini 4 Argon8.0, p10–p90 3–18, n = 101 completed runs
GLM 5.3 Flash8.0, p10–p90 3–20.6, n = 83 completed runs
Gemini 3.8 Flash8.0, p10–p90 3–21, n = 101 completed runs
MiMo V2.6 Pro8.0, p10–p90 3–19, n = 81 completed runs
Muse Wonder 1.08.0, p10–p90 6.10–21.7, n = 12 completed runs
Hy49.0, p10–p90 4.50–25, n = 86 completed runs
Grok 4.710.0, p10–p90 3–31, n = 81 completed runs
Claude Haiku 5.511.0, p10–p90 3–40.1, n = 90 completed runs
MiniMax M3.1 Flash12.0, p10–p90 6–42.6, n = 87 completed runs
Lower is better. Medians. n = 12–124 completed runs per model.Coarenaby Coasty
Detour factor
Geometric mean of (this agent's actions / the opponent's actions) across battles where BOTH sides completed the identical task.
Muse Spark 1.30.74x, n = 14 paired battles
Muse Wonder 1.00.77x, n = 12 paired battles
GPT-6.1 Terra0.80x, n = 117 paired battles
Gemini 4 Argon0.82x, n = 101 paired battles
Kimi K3.1 Pro0.82x, n = 85 paired battles
GPT-6 Astra0.89x, n = 84 paired battles
GLM 5.3 Flash0.89x, n = 80 paired battles
Claude Sonnet 5.50.90x, n = 81 paired battles
Claude Opus 5.50.90x, n = 82 paired battles
Kimi K30.91x, n = 84 paired battles
Qwen3.8-2.4T-A95B0.93x, n = 95 paired battles
Claude Fable 50.93x, n = 81 paired battles
Gemini 3.8 Flash0.94x, n = 99 paired battles
GPT-6 Luna0.95x, n = 76 paired battles
GPT-6.1 Sol0.95x, n = 105 paired battles
MiMo V2.6 Pro0.95x, n = 79 paired battles
Claude Fable 5.11.00x, n = 89 paired battles
DeepSeek V4.1 Flash 10171.03x, n = 85 paired battles
Hy41.21x, n = 86 paired battles
Grok 4.71.22x, n = 78 paired battles
Claude Haiku 5.51.25x, n = 87 paired battles
MiniMax M3.1 Flash1.66x, n = 87 paired battles
Lower is better. n = 12–117 paired battles per model.Coarenaby Coasty
Actions relative to best
Geometric mean over the agent's completed runs of its actions divided by the fewest actions any other agent needed to complete the same task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL.
GPT-6 Astra1.25x, 95% interval 1.16x–1.36x, n = 83 completed runs on shared tasks
Claude Fable 5.11.26x, 95% interval 1.18x–1.36x, n = 80 completed runs on shared tasks
Claude Sonnet 5.51.27x, 95% interval 1.18x–1.38x, n = 82 completed runs on shared tasks
Claude Fable 51.27x, 95% interval 1.18x–1.38x, n = 81 completed runs on shared tasks
GPT-6.1 Terra1.29x, 95% interval 1.18x–1.41x, n = 100 completed runs on shared tasks
Gemini 4 Argon1.29x, 95% interval 1.21x–1.38x, n = 99 completed runs on shared tasks
Claude Opus 5.51.34x, 95% interval 1.24x–1.46x, n = 84 completed runs on shared tasks
Kimi K3.1 Pro1.36x, 95% interval 1.26x–1.47x, n = 83 completed runs on shared tasks
GLM 5.3 Flash1.36x, 95% interval 1.27x–1.47x, n = 80 completed runs on shared tasks
Kimi K31.36x, 95% interval 1.26x–1.49x, n = 82 completed runs on shared tasks
MiMo V2.6 Pro1.39x, 95% interval 1.29x–1.51x, n = 80 completed runs on shared tasks
GPT-6 Luna1.41x, 95% interval 1.28x–1.57x, n = 77 completed runs on shared tasks
GPT-6.1 Sol1.42x, 95% interval 1.30x–1.57x, n = 87 completed runs on shared tasks
Qwen3.8-2.4T-A95B1.49x, 95% interval 1.36x–1.64x, n = 94 completed runs on shared tasks
Gemini 3.8 Flash1.50x, 95% interval 1.36x–1.67x, n = 100 completed runs on shared tasks
DeepSeek V4.1 Flash 10171.58x, 95% interval 1.45x–1.74x, n = 87 completed runs on shared tasks
Hy41.83x, 95% interval 1.66x–2.02x, n = 86 completed runs on shared tasks
Grok 4.71.85x, 95% interval 1.65x–2.09x, n = 80 completed runs on shared tasks
Claude Haiku 5.52.01x, 95% interval 1.79x–2.25x, n = 86 completed runs on shared tasks
MiniMax M3.1 Flash2.37x, 95% interval 2.10x–2.69x, n = 86 completed runs on shared tasks
1x1.5x2x2.5x3x
Lower is better. Lines are 95% intervals. n = 77–100 completed runs on shared tasks per model.Coarenaby Coasty
Wasted motion
Repeated itself
Actions byte-identical to the action immediately before them.
Muse Spark 1.32%, 95% interval 0.7–5.8%, n = 147 actions
GPT-6 Astra3%, 95% interval 1.8–4.1%, n = 773 actions
GPT-6 Luna3%, 95% interval 1.9–4%, n = 966 actions
GPT-6.1 Sol3%, 95% interval 2.4–4.1%, n = 1.6k actions
Muse Wonder 1.04%, 95% interval 1.8–6.9%, n = 224 actions
Grok 4.74%, 95% interval 2.9–5.1%, n = 1.3k actions
GPT-6.1 Terra4%, 95% interval 3.5–5.5%, n = 1.6k actions
Claude Haiku 5.55%, 95% interval 3.7–5.6%, n = 1.8k actions
Kimi K3.1 Pro6%, 95% interval 4.3–7.1%, n = 978 actions
Claude Fable 5.17%, 95% interval 5.2–8.9%, n = 720 actions
Qwen3.8-2.4T-A95B7%, 95% interval 5.8–8.7%, n = 1.3k actions
MiMo V2.6 Pro7%, 95% interval 5.7–9.1%, n = 872 actions
Claude Fable 58%, 95% interval 6.6–9.9%, n = 1.1k actions
Claude Sonnet 5.58%, 95% interval 6.4–11%, n = 664 actions
Claude Opus 5.58%, 95% interval 6.6–10%, n = 842 actions
Gemini 4 Argon9%, 95% interval 7–11%, n = 917 actions
Hy49%, 95% interval 7.3–10%, n = 1.2k actions
GLM 5.3 Flash9%, 95% interval 7.3–11%, n = 960 actions
MiniMax M3.1 Flash9%, 95% interval 7.9–11%, n = 1.7k actions
Gemini 3.8 Flash12%, 95% interval 10–14%, n = 1.2k actions
0%5%10%15%
Lower is better. Lines are 95% intervals. n = 147–1.8k actions per model.Coarenaby Coasty
Changed nothing
Actions whose observation was identical to the previous step's, once the tab marker is stripped.
Not measured: a phone observation is the device's echo of the action ("Tapped (x, y)."), not a reading of the screen, so an unchanged observation is a repeated action rather than an unchanged screen.
Explicitly inert
Actions the browser explicitly reported as having moved nothing: scrolling past the end of a document, scrolling a panel that does not scroll.
Not measured: no phone action reports that it moved nothing: only the browser's scroll says when a page did not move.
Stayed put
Actions after which the page URL was unchanged from the previous step.
Not measured: phones record no page URL, so whether the page changed cannot be read.
Went back
Steps landing on a URL the run had already visited and since left.
Not measured: phones record no page URL, so a return to one cannot be read.
Exploration
Looking vs doing
Look-class actions (a screenshot, reading the page or its elements, a hover, a pointer move, a phone wait) as a share of all actions that either looked or changed something. Refused tool calls executed nothing and are in neither side.
GPT-6.1 Terra10%, 95% interval 9–12%, n = 1.6k actions
Kimi K313%, 95% interval 11–16%, n = 832 actions
Claude Fable 5.17%, 95% interval 5.6–9.4%, n = 713 actions
GPT-6.1 Sol7%, 95% interval 5.5–7.9%, n = 1.6k actions
Kimi K3.1 Pro6%, 95% interval 4.3–7.1%, n = 976 actions
Claude Opus 5.58%, 95% interval 6.4–10%, n = 840 actions
GPT-6 Astra12%, 95% interval 9.6–14%, n = 773 actions
Claude Fable 58%, 95% interval 6.2–9.4%, n = 1.1k actions
Claude Sonnet 5.56%, 95% interval 4.5–8.1%, n = 664 actions
Qwen3.8-2.4T-A95B6%, 95% interval 4.6–7.2%, n = 1.3k actions
Gemini 4 Argon5%, 95% interval 3.8–6.6%, n = 916 actions
GLM 5.3 Flash6%, 95% interval 5–8.1%, n = 959 actions
Gemini 3.8 Flash5%, 95% interval 4–6.5%, n = 1.2k actions
MiMo V2.6 Pro8%, 95% interval 6.5–10%, n = 869 actions
GPT-6 Luna5%, 95% interval 4.2–7.1%, n = 964 actions
Grok 4.75%, 95% interval 4.1–6.5%, n = 1.3k actions
Claude Haiku 5.535%, 95% interval 33–37%, n = 1.8k actions
MiniMax M3.1 Flash5%, 95% interval 4–6%, n = 1.7k actions
Muse Spark 1.37%, 95% interval 3.8–12%, n = 146 actions
Muse Wonder 1.04%, 95% interval 2.4–8%, n = 224 actions
0%10%20%30%40%
Neither direction is better. Lines are 95% intervals. n = 146–1.8k actions per model.Coarenaby Coasty
Action variety
Shannon entropy of the run's distribution over the ten action classes, normalized by log2 of the number of classes its mode OFFERS: ten in a browser, nine on a desktop (no scroll verb), nine on a phone (no script or shell). A desktop run from 7 to 9 August 2026 that used the browser's scroll, which desktop mode then still offered, is normalized by ten.
GPT-6.1 Terra0.57, p10–p90 0.43–0.71, n = 123 runs
Kimi K30.58, p10–p90 0.43–0.71, n = 85 runs
Claude Fable 5.10.57, p10–p90 0.43–0.71, n = 91 runs
GPT-6.1 Sol0.58, p10–p90 0.34–0.71, n = 109 runs
Kimi K3.1 Pro0.57, p10–p90 0.43–0.74, n = 88 runs
Claude Opus 5.50.57, p10–p90 0.42–0.71, n = 88 runs
GPT-6 Astra0.57, p10–p90 0.41–0.70, n = 85 runs
Claude Fable 50.53, p10–p90 0.39–0.70, n = 88 runs
Claude Sonnet 5.50.55, p10–p90 0.40–0.69, n = 84 runs
Qwen3.8-2.4T-A95B0.57, p10–p90 0.35–0.68, n = 98 runs
Gemini 4 Argon0.57, p10–p90 0.34–0.69, n = 104 runs
GLM 5.3 Flash0.59, p10–p90 0.39–0.71, n = 85 runs
Gemini 3.8 Flash0.55, p10–p90 0.38–0.70, n = 105 runs
MiMo V2.6 Pro0.59, p10–p90 0.38–0.71, n = 84 runs
GPT-6 Luna0.57, p10–p90 0.38–0.73, n = 83 runs
DeepSeek V4.1 Flash 10170.58, p10–p90 0.43–0.71, n = 89 runs
Hy40.62, p10–p90 0.43–0.75, n = 88 runs
Grok 4.70.56, p10–p90 0.41–0.72, n = 84 runs
Claude Haiku 5.50.61, p10–p90 0.48–0.74, n = 93 runs
MiniMax M3.1 Flash0.62, p10–p90 0.45–0.73, n = 88 runs
Muse Spark 1.30.50, p10–p90 0.28–0.68, n = 14 runs
Muse Wonder 1.00.59, p10–p90 0.40–0.71, n = 12 runs
Neither direction is better. Medians. n = 12–123 runs per model.Coarenaby Coasty
Action mix
Share of each kind of action across the agent's executed actions, over ten classes: navigate (a URL, back, forward, reload, the phone's Back), point (a click, tap, double tap or long press), drag or swipe, scroll, type (text or a saved credential), key, switch app or tab (a tab, an app, Home, Recents), look (a screenshot, the page text or elements, a hover, a pointer move, a phone wait), script or shell, and finish (done or a delivered file).
Navigate
Point
Drag or swipe
Scroll
Type
Key
Switch app or tab
Look
Finish
GPT-6.1 Terra1.7k
11%
45%
3%
9%
7%
0%
7%
10%
8%
Kimi K3918
8%
44%
3%
12%
3%
0%
7%
12%
9%
Claude Fable 5.1807
9%
43%
4%
9%
8%
0%
8%
6%
12%
GPT-6.1 Sol1.7k
9%
45%
2%
12%
6%
0%
12%
6%
7%
Kimi K3.1 Pro1.1k
9%
47%
3%
11%
9%
0%
8%
5%
8%
Claude Opus 5.5929
12%
41%
5%
13%
5%
0%
7%
7%
10%
GPT-6 Astra859
7%
47%
3%
13%
2%
0%
7%
10%
10%
Claude Fable 51.2k
10%
49%
2%
10%
7%
0%
8%
7%
7%
Claude Sonnet 5.5748
7%
45%
2%
13%
8%
0%
8%
5%
11%
Qwen3.8-2.4T-A95B1.4k
11%
48%
3%
16%
3%
1%
6%
5%
7%
Gemini 4 Argon1.0k
8%
46%
2%
17%
4%
0%
8%
5%
10%
GLM 5.3 Flash1.0k
10%
43%
6%
15%
4%
1%
7%
6%
8%
Gemini 3.8 Flash1.3k
8%
47%
3%
19%
4%
0%
7%
5%
8%
MiMo V2.6 Pro953
8%
47%
2%
13%
4%
0%
9%
7%
9%
GPT-6 Luna1.0k
9%
51%
3%
11%
7%
0%
6%
5%
8%
DeepSeek V4.1 Flash 10171.2k
12%
47%
3%
12%
6%
3%
6%
4%
8%
Hy41.3k
12%
42%
4%
17%
6%
1%
7%
5%
7%
Grok 4.71.4k
9%
53%
4%
10%
6%
1%
6%
5%
6%
Claude Haiku 5.51.9k
8%
33%
1%
11%
4%
1%
4%
33%
5%
MiniMax M3.1 Flash1.8k
13%
44%
3%
18%
4%
1%
7%
5%
5%
Muse Spark 1.3160
8%
57%
0%
6%
7%
0%
8%
6%
9%
Muse Wonder 1.0236
5%
54%
20%
4%
3%
0%
4%
4%
5%
Share of each model's actions; n after each name. Not observed: script or shell.Coarenaby Coasty
GPT-6.1 Terra · Point45%, point, 769 of 1.7k actions
Kimi K3 · Point44%, point, 405 of 918 actions
Claude Fable 5.1 · Point43%, point, 350 of 807 actions
GPT-6.1 Sol · Point45%, point, 770 of 1.7k actions
Kimi K3.1 Pro · Point47%, point, 496 of 1.1k actions
Claude Opus 5.5 · Point41%, point, 384 of 929 actions
GPT-6 Astra · Point47%, point, 406 of 859 actions
Claude Fable 5 · Point49%, point, 571 of 1.2k actions
Claude Sonnet 5.5 · Point45%, point, 337 of 748 actions
Qwen3.8-2.4T-A95B · Point48%, point, 649 of 1.4k actions
Gemini 4 Argon · Point46%, point, 466 of 1.0k actions
GLM 5.3 Flash · Point43%, point, 450 of 1.0k actions
Gemini 3.8 Flash · Point47%, point, 613 of 1.3k actions
MiMo V2.6 Pro · Point47%, point, 448 of 953 actions
GPT-6 Luna · Point51%, point, 534 of 1.0k actions
DeepSeek V4.1 Flash 1017 · Point47%, point, 550 of 1.2k actions
Hy4 · Point42%, point, 566 of 1.3k actions
Grok 4.7 · Point53%, point, 730 of 1.4k actions
Claude Haiku 5.5 · Look33%, look, 638 of 1.9k actions
MiniMax M3.1 Flash · Point44%, point, 800 of 1.8k actions
Muse Spark 1.3 · Point57%, point, 92 of 160 actions
Muse Wonder 1.0 · Point54%, point, 128 of 236 actions
Each model's most common kind. n = 160–1.9k actions per model.Coarenaby Coasty
Pages visited
Distinct page URLs observed across a run.
Not measured: phones record no page URL.
Left the site
Navigations to an origin other than the first one the run observed.
Not measured: phones record no page URL, so a run has no origin to leave.
By task tier
Completed, easy tasks
Share of runs that reported success on tasks whose best observed completion took 5 or fewer actions, the tier set by the fewest actions any agent needed to complete the task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL.
GPT-6.1 Terra100%, 95% interval 93–100%, n = 53 runs
Kimi K3100%, 95% interval 92–100%, n = 46 runs
Claude Fable 5.1100%, 95% interval 91–100%, n = 41 runs
GPT-6.1 Sol100%, 95% interval 91–100%, n = 40 runs
Kimi K3.1 Pro100%, 95% interval 92–100%, n = 43 runs
Claude Opus 5.5100%, 95% interval 92–100%, n = 46 runs
GPT-6 Astra100%, 95% interval 92–100%, n = 45 runs
Claude Fable 5100%, 95% interval 91–100%, n = 41 runs
Claude Sonnet 5.5100%, 95% interval 92–100%, n = 44 runs
Qwen3.8-2.4T-A95B100%, 95% interval 92–100%, n = 43 runs
Gemini 4 Argon100%, 95% interval 93–100%, n = 53 runs
Gemini 3.8 Flash100%, 95% interval 93–100%, n = 54 runs
Claude Haiku 5.5100%, 95% interval 92–100%, n = 45 runs
MiniMax M3.1 Flash100%, 95% interval 92–100%, n = 47 runs
MiMo V2.6 Pro98%, 95% interval 88–100%, n = 42 runs
Grok 4.798%, 95% interval 88–100%, n = 42 runs
GLM 5.3 Flash98%, 95% interval 87–100%, n = 41 runs
85%90%95%100%
Higher is better. Lines are 95% intervals. n = 40–54 runs per model.Coarenaby Coasty
Completed, medium tasks
Share of runs that reported success on tasks whose best observed completion took 6 to 10 actions, the tier set by the fewest actions any agent needed to complete the task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL.
GPT-6.1 Sol100%, 95% interval 89–100%, n = 30 runs
Qwen3.8-2.4T-A95B100%, 95% interval 90–100%, n = 33 runs
85%90%95%100%
Higher is better. Lines are 95% intervals. n = 30–33 runs per model.Coarenaby Coasty
Completed, hard tasks
Share of runs that reported success on tasks whose best observed completion took 11 or more actions, the tier set by the fewest actions any agent needed to complete the task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL.
Not measured: fewer than 30 runs on hard-tier tasks.
Bypass
Constructed URLs
Share of the run's URL opens (phone_open_url) whose URL carries a query string, not counting the run's first action when the task has a starting URL: the agent wrote a search or a filter into the address instead of using an app's own controls.
Not measured: fewer than 100 URL opens.
Interface bypass
Share of actions in browser battles that ran a shell, a script or a direct DOM write (os_bash, os_run, execute_js, set_element_value) instead of driving the page. A battle with no recorded mode is a browser battle (every battle from before modes existed was one).
Not measured: a phone offers no script, shell or DOM tool to reach past the screen.
Worst error streak
Longest run of consecutive failing actions within a run, the model's errors only: an infrastructure failure (a frozen or lost phone, an unreachable sandbox service) neither extends a streak nor ends one.
GPT-6.1 Terra0, p10–p90 0–0, n = 135 runs
Kimi K30, p10–p90 0–0, n = 86 runs
Claude Fable 5.10, p10–p90 0–0, n = 94 runs
GPT-6.1 Sol0, p10–p90 0–0, n = 117 runs
Kimi K3.1 Pro0, p10–p90 0–0, n = 88 runs
Claude Opus 5.50, p10–p90 0–0, n = 89 runs
GPT-6 Astra0, p10–p90 0–0, n = 86 runs
Claude Fable 50, p10–p90 0–0, n = 88 runs
Claude Sonnet 5.50, p10–p90 0–0, n = 84 runs
Qwen3.8-2.4T-A95B0, p10–p90 0–0, n = 99 runs
Gemini 4 Argon0, p10–p90 0–0, n = 104 runs
GLM 5.3 Flash0, p10–p90 0–0, n = 85 runs
Gemini 3.8 Flash0, p10–p90 0–0, n = 105 runs
MiMo V2.6 Pro0, p10–p90 0–0, n = 84 runs
GPT-6 Luna0, p10–p90 0–0, n = 83 runs
DeepSeek V4.1 Flash 10170, p10–p90 0–0, n = 89 runs
Hy40, p10–p90 0–0, n = 88 runs
Grok 4.70, p10–p90 0–0, n = 84 runs
Claude Haiku 5.50, p10–p90 0–0, n = 93 runs
MiniMax M3.1 Flash0, p10–p90 0–0, n = 88 runs
Muse Spark 1.30, p10–p90 0–0, n = 14 runs
Muse Wonder 1.00, p10–p90 0–0, n = 12 runs
Lower is better. Medians. n = 12–135 runs per model.Coarenaby Coasty
Rejected actions
Share of actions the harness refused to execute because the call was malformed: an unknown verb, a field the schema rejected, or a browser or desktop verb on a phone run. Each refusal is recorded as a step with the same sentence in every lane.
Kimi K30.0%, 95% interval 0–0.4%, n = 918 actions
GPT-6.1 Sol0.0%, 95% interval 0–0.2%, n = 1.7k actions
GPT-6 Astra0.0%, 95% interval 0–0.4%, n = 859 actions
Claude Fable 50.0%, 95% interval 0–0.3%, n = 1.2k actions
Claude Sonnet 5.50.0%, 95% interval 0–0.5%, n = 748 actions
Gemini 3.8 Flash0.0%, 95% interval 0–0.3%, n = 1.3k actions
Grok 4.70.0%, 95% interval 0–0.3%, n = 1.4k actions
Claude Haiku 5.50.0%, 95% interval 0–0.2%, n = 1.9k actions
MiniMax M3.1 Flash0.0%, 95% interval 0–0.2%, n = 1.8k actions
Hy40.1%, 95% interval 0.01–0.4%, n = 1.3k actions
GLM 5.3 Flash0.1%, 95% interval 0.02–0.5%, n = 1.0k actions
Gemini 4 Argon0.1%, 95% interval 0.02–0.6%, n = 1.0k actions
Kimi K3.1 Pro0.2%, 95% interval 0.05–0.7%, n = 1.1k actions
GPT-6 Luna0.2%, 95% interval 0.05–0.7%, n = 1.0k actions
Claude Opus 5.50.2%, 95% interval 0.06–0.8%, n = 931 actions
MiMo V2.6 Pro0.3%, 95% interval 0.1–0.9%, n = 956 actions
Qwen3.8-2.4T-A95B0.4%, 95% interval 0.2–0.9%, n = 1.4k actions
GPT-6.1 Terra0.5%, 95% interval 0.2–0.9%, n = 1.7k actions
Claude Fable 5.10.9%, 95% interval 0.4–1.8%, n = 814 actions
0%0.5%1%1.5%2%
Lower is better. Lines are 95% intervals. n = 748–1.9k actions per model.Coarenaby Coasty
Resilience
Recovered
Failing actions whose NEXT action did not also fail. A failing action is the model's error, not one the infrastructure wrote; a tool call the arena refused counts as one, so a refused next call is not a recovery.
GPT-6.1 Terra100%, 95% interval 68–100%, n = 8 failed actions
Claude Fable 5.1100%, 95% interval 65–100%, n = 7 failed actions
Kimi K3.1 Pro100%, 95% interval 34–100%, n = 2 failed actions
Claude Opus 5.5100%, 95% interval 34–100%, n = 2 failed actions
Qwen3.8-2.4T-A95B100%, 95% interval 57–100%, n = 5 failed actions
Higher is better. Lines are 95% intervals. n = 1–8 failed actions per model.Coarenaby Coasty
Retried the same thing
Failing actions immediately followed by the byte-identical action: the same call, expecting a different answer. A failing action is the model's error, not one the infrastructure wrote; a tool call the arena refused counts as one.
GPT-6.1 Terra0.0%, 95% interval 0–32%, n = 8 failed actions
Claude Fable 5.10.0%, 95% interval 0–35%, n = 7 failed actions
Kimi K3.1 Pro0.0%, 95% interval 0–66%, n = 2 failed actions
Claude Opus 5.50.0%, 95% interval 0–66%, n = 2 failed actions
Qwen3.8-2.4T-A95B0.0%, 95% interval 0–43%, n = 5 failed actions
Lower is better. Lines are 95% intervals. n = 1–8 failed actions per model.Coarenaby Coasty
Run level
Repaired after an error
Of runs with at least one failed action, the share whose recorded outcome was completed. A failed action is an error the action met; one the infrastructure wrote (a frozen or lost phone, an unreachable sandbox service) does not make a run an error run.
Not measured: fewer than 30 error runs.
Abandoned after an error
Of runs with at least one failed action, the share that ended within three actions of their last failed action without completing. A failed action is an error the action met, not one the infrastructure wrote (a frozen or lost phone, an unreachable sandbox service).
Not measured: fewer than 30 error runs.
Corrected, as judged
Of steps a judge marked bad and then answered whether the run later corrected, the share the judge said it corrected.
Not measured: no step verdicts recorded.
Pressed the same pixel again
Presses at an (x, y) already pressed earlier in the same run.
GPT-6.1 Terra3%, 95% interval 1.7–4%, n = 769 presses at a point
GPT-6 Luna4%, 95% interval 2.7–6.2%, n = 534 presses at a point
GPT-6 Astra5%, 95% interval 3.4–7.8%, n = 406 presses at a point
GLM 5.3 Flash6%, 95% interval 4.3–8.8%, n = 450 presses at a point
Kimi K3.1 Pro6%, 95% interval 4.6–9%, n = 496 presses at a point
Muse Spark 1.38%, 95% interval 3.7–15%, n = 92 presses at a point
Hy48%, 95% interval 5.8–10%, n = 566 presses at a point
DeepSeek V4.1 Flash 10179%, 95% interval 7–12%, n = 550 presses at a point
Gemini 4 Argon9%, 95% interval 6.9–12%, n = 466 presses at a point
Gemini 3.8 Flash11%, 95% interval 8.6–13%, n = 613 presses at a point
Claude Fable 5.112%, 95% interval 9.3–16%, n = 350 presses at a point
Claude Sonnet 5.513%, 95% interval 10–17%, n = 337 presses at a point
MiMo V2.6 Pro14%, 95% interval 11–17%, n = 448 presses at a point
Grok 4.714%, 95% interval 12–17%, n = 730 presses at a point
GPT-6.1 Sol14%, 95% interval 12–17%, n = 770 presses at a point
Muse Wonder 1.015%, 95% interval 9.7–22%, n = 128 presses at a point
Kimi K316%, 95% interval 13–20%, n = 405 presses at a point
MiniMax M3.1 Flash18%, 95% interval 16–21%, n = 800 presses at a point
Claude Opus 5.519%, 95% interval 16–24%, n = 384 presses at a point
Qwen3.8-2.4T-A95B21%, 95% interval 18–25%, n = 649 presses at a point
Claude Fable 523%, 95% interval 20–26%, n = 571 presses at a point
Claude Haiku 5.523%, 95% interval 20–27%, n = 631 presses at a point
0%10%20%30%
Lower is better. Lines are 95% intervals. n = 92–800 presses at a point per model.Coarenaby Coasty
Press spread
Mean pairwise distance between a run's presses at a point, divided by the diagonal of its recorded viewport.
GPT-6.1 Terra0.29, p10–p90 0.17–0.42, n = 94 runs
Kimi K30.27, p10–p90 0.12–0.38, n = 58 runs
Claude Fable 5.10.28, p10–p90 0.17–0.45, n = 66 runs
GPT-6.1 Sol0.27, p10–p90 0.14–0.39, n = 83 runs
Kimi K3.1 Pro0.29, p10–p90 0.14–0.41, n = 69 runs
Claude Opus 5.50.28, p10–p90 0.17–0.47, n = 61 runs
GPT-6 Astra0.26, p10–p90 0.11–0.40, n = 64 runs
Claude Fable 50.29, p10–p90 0.18–0.39, n = 59 runs
Claude Sonnet 5.50.28, p10–p90 0.13–0.42, n = 54 runs
Qwen3.8-2.4T-A95B0.27, p10–p90 0.16–0.35, n = 74 runs
Gemini 4 Argon0.28, p10–p90 0.16–0.36, n = 79 runs
GLM 5.3 Flash0.31, p10–p90 0.18–0.43, n = 64 runs
Gemini 3.8 Flash0.28, p10–p90 0.18–0.38, n = 80 runs
MiMo V2.6 Pro0.27, p10–p90 0.16–0.46, n = 63 runs
GPT-6 Luna0.27, p10–p90 0.16–0.42, n = 66 runs
DeepSeek V4.1 Flash 10170.28, p10–p90 0.14–0.37, n = 68 runs
Hy40.30, p10–p90 0.14–0.40, n = 73 runs
Grok 4.70.28, p10–p90 0.18–0.39, n = 69 runs
Claude Haiku 5.50.26, p10–p90 0.10–0.37, n = 67 runs
MiniMax M3.1 Flash0.32, p10–p90 0.19–0.41, n = 70 runs
Muse Spark 1.30.25, p10–p90 0.16–0.38, n = 9 runs
Muse Wonder 1.00.31, p10–p90 0.14–0.41, n = 11 runs
Neither direction is better. Medians. n = 9–94 runs per model.Coarenaby Coasty
Clicks that did nothing
Presses at a point (a click, a desktop click, a phone tap, double tap or long press) that landed on nothing. Withheld on both boards: no press's observation records whether the press hit a control.
Not measured: a phone tap's observation echoes the tap ("Tapped (x, y).") and never reports an unchanged screen, so a tap that landed on nothing reads as effective and whether a tap hit anything cannot be read.
Gesture
Points per drag
Coordinate pairs supplied per path drag (drag, desktop drag). A phone swipe has two endpoints by contract and is not a path drag.
Not measured: a phone swipe has exactly two endpoints by contract, so there is no path to measure.
Endpoint-only drags
Path drags (drag, desktop drag) supplied with two points or fewer. A phone swipe is not counted: it has exactly two endpoints by contract.
Not measured: a phone swipe has exactly two endpoints by contract, so every swipe would count as flattened.
Modified clicks
Browser clicks holding Alt, Control, Meta or Shift.
Not measured: a phone tap carries no modifier keys.
Text entry
Characters per typing action
Characters (code points) supplied per text-entry action: type, desktop type, phone type. A saved credential supplies no visible text and is not counted.
GPT-6.1 Terra18.0, p10–p90 5–1.7k, n = 125 text-entry actions
Kimi K312.0, p10–p90 4.80–18.6, n = 30 text-entry actions
Claude Fable 5.114.0, p10–p90 2.70–1.8k, n = 68 text-entry actions
GPT-6.1 Sol24.0, p10–p90 7–1.7k, n = 94 text-entry actions
Kimi K3.1 Pro20.0, p10–p90 6.50–793, n = 96 text-entry actions
Claude Opus 5.59.0, p10–p90 3–17.6, n = 45 text-entry actions
GPT-6 Astra12.0, p10–p90 5–18, n = 21 text-entry actions
Claude Fable 515.0, p10–p90 6.70–91.4, n = 88 text-entry actions
Claude Sonnet 5.59.0, p10–p90 3–18, n = 57 text-entry actions
Qwen3.8-2.4T-A95B9.0, p10–p90 3–18, n = 42 text-entry actions
Gemini 4 Argon13.0, p10–p90 3.90–34.0, n = 40 text-entry actions
GLM 5.3 Flash10.0, p10–p90 5–25.2, n = 45 text-entry actions
Gemini 3.8 Flash9.0, p10–p90 4.10–18.9, n = 52 text-entry actions
MiMo V2.6 Pro14.0, p10–p90 7–45.3, n = 42 text-entry actions
GPT-6 Luna13.0, p10–p90 2–46, n = 73 text-entry actions
Hy49.5, p10–p90 4.10–19, n = 82 text-entry actions
Grok 4.79.0, p10–p90 3–18, n = 79 text-entry actions
Claude Haiku 5.59.0, p10–p90 3–18, n = 77 text-entry actions
MiniMax M3.1 Flash12.0, p10–p90 4–82, n = 75 text-entry actions
Muse Spark 1.339.0, p10–p90 3–1.4k, n = 11 text-entry actions
Muse Wonder 1.017.5, p10–p90 8.40–28.1, n = 8 text-entry actions
Neither direction is better. Medians. n = 8–125 text-entry actions per model.Coarenaby Coasty
Keys vs strings
Key-class actions (press_key, desktop key, phone key) as a share of key-class plus type-class actions.
GPT-6.1 Terra0%, 95% interval 0–3%, n = 125 text actions
Kimi K30%, 95% interval 0–11%, n = 30 text actions
Claude Fable 5.10%, 95% interval 0–5.3%, n = 68 text actions
GPT-6.1 Sol0%, 95% interval 0–3.9%, n = 94 text actions
Kimi K3.1 Pro4%, 95% interval 1.6–9.8%, n = 100 text actions
Claude Opus 5.50%, 95% interval 0–7.9%, n = 45 text actions
GPT-6 Astra0%, 95% interval 0–15%, n = 21 text actions
Claude Fable 52%, 95% interval 0.6–7.7%, n = 90 text actions
Claude Sonnet 5.53%, 95% interval 0.9–12%, n = 59 text actions
Qwen3.8-2.4T-A95B19%, 95% interval 11–32%, n = 52 text actions
Gemini 4 Argon5%, 95% interval 1.3–16%, n = 42 text actions
GLM 5.3 Flash13%, 95% interval 6.7–25%, n = 52 text actions
Gemini 3.8 Flash5%, 95% interval 1.9–15%, n = 55 text actions
MiMo V2.6 Pro2%, 95% interval 0.4–12%, n = 43 text actions
GPT-6 Luna3%, 95% interval 0.7–9.2%, n = 75 text actions
DeepSeek V4.1 Flash 101735%, 95% interval 26–45%, n = 100 text actions
Hy48%, 95% interval 3.9–15%, n = 89 text actions
Grok 4.712%, 95% interval 7–21%, n = 90 text actions
Claude Haiku 5.515%, 95% interval 9.4–24%, n = 91 text actions
MiniMax M3.1 Flash18%, 95% interval 12–28%, n = 92 text actions
Muse Spark 1.30%, 95% interval 0–26%, n = 11 text actions
Muse Wonder 1.011%, 95% interval 2–44%, n = 9 text actions
0%20%40%60%
Neither direction is better. Lines are 95% intervals. n = 9–125 text actions per model.Coarenaby Coasty
Time to first action
Elapsed milliseconds when the first action resolved. A run whose first action the infrastructure failed (a frozen or lost phone, an unreachable sandbox service) is left out: that wait is the machine's.
Claude Haiku 5.54s, p10–p90 3.3s–4.9s, n = 93 runs
Qwen3.8-2.4T-A95B4s, p10–p90 3.5s–5.9s, n = 99 runs
MiniMax M3.1 Flash4s, p10–p90 3.2s–5.9s, n = 88 runs
Grok 4.74s, p10–p90 3.7s–5.1s, n = 84 runs
Claude Sonnet 5.54s, p10–p90 3.7s–5.4s, n = 84 runs
GPT-6 Luna4s, p10–p90 3.8s–6.8s, n = 83 runs
DeepSeek V4.1 Flash 10175s, p10–p90 4.0s–5.6s, n = 89 runs
GPT-6.1 Terra5s, p10–p90 3.9s–6.6s, n = 135 runs
GPT-6 Astra5s, p10–p90 4.1s–6.3s, n = 86 runs
Muse Wonder 1.05s, p10–p90 4.3s–7.0s, n = 12 runs
Claude Fable 5.15s, p10–p90 4.0s–7.5s, n = 94 runs
Claude Opus 5.55s, p10–p90 4.3s–6.4s, n = 89 runs
GPT-6.1 Sol5s, p10–p90 4.3s–7.2s, n = 117 runs
Muse Spark 1.35s, p10–p90 3.9s–6.2s, n = 14 runs
Kimi K35s, p10–p90 4.2s–7.9s, n = 86 runs
Claude Fable 56s, p10–p90 5.3s–7.7s, n = 88 runs
GLM 5.3 Flash7s, p10–p90 4.9s–10s, n = 85 runs
Gemini 3.8 Flash7s, p10–p90 5.4s–11s, n = 105 runs
Hy47s, p10–p90 5.9s–9.1s, n = 88 runs
MiMo V2.6 Pro7s, p10–p90 5.7s–10s, n = 84 runs
Gemini 4 Argon8s, p10–p90 6.6s–12s, n = 104 runs
Kimi K3.1 Pro11s, p10–p90 8.6s–14s, n = 88 runs
Lower is better. Medians. n = 12–135 runs per model.Coarenaby Coasty
Milliseconds per action
Median gap between consecutive step timestamps within a run, the first gap measured from the run's start. A gap that ends on an action the infrastructure failed (a frozen or lost phone, an unreachable sandbox service) is left out: it is the machine's wait, not the model's.
Claude Haiku 5.52s, p10–p90 1.8s–3.7s, n = 93 runs
Claude Sonnet 5.53s, p10–p90 2.2s–4.4s, n = 84 runs
Qwen3.8-2.4T-A95B3s, p10–p90 2.1s–4.7s, n = 99 runs
Grok 4.73s, p10–p90 2.2s–4.6s, n = 84 runs
DeepSeek V4.1 Flash 10173s, p10–p90 2.3s–4.4s, n = 89 runs
Muse Wonder 1.03s, p10–p90 2.6s–3.8s, n = 12 runs
GPT-6.1 Terra3s, p10–p90 2.5s–5.4s, n = 135 runs
Muse Spark 1.33s, p10–p90 2.4s–5.2s, n = 14 runs
GPT-6 Astra3s, p10–p90 2.7s–4.5s, n = 86 runs
Claude Opus 5.53s, p10–p90 2.7s–5.5s, n = 89 runs
MiniMax M3.1 Flash3s, p10–p90 2.3s–4.9s, n = 88 runs
GPT-6 Luna3s, p10–p90 2.8s–5.5s, n = 83 runs
GPT-6.1 Sol4s, p10–p90 3.0s–5.2s, n = 117 runs
Claude Fable 5.14s, p10–p90 2.5s–6.2s, n = 94 runs
Kimi K34s, p10–p90 3.0s–6.5s, n = 86 runs
Claude Fable 54s, p10–p90 3.6s–6.3s, n = 88 runs
Gemini 3.8 Flash5s, p10–p90 4.1s–8.6s, n = 105 runs
Hy46s, p10–p90 4.6s–7.2s, n = 88 runs
GLM 5.3 Flash6s, p10–p90 4.0s–7.7s, n = 85 runs
MiMo V2.6 Pro7s, p10–p90 5.2s–8.3s, n = 84 runs
Gemini 4 Argon8s, p10–p90 6.1s–10s, n = 104 runs
Kimi K3.1 Pro10s, p10–p90 8.1s–12s, n = 88 runs
Lower is better. Medians. n = 12–135 runs per model.Coarenaby Coasty
Gap between actions, pooled
Milliseconds between consecutive step timestamps, pooled over every step of every run rather than one median per run, with the wait of any recorded human pause inside the gap subtracted. A gap that ends on an action the infrastructure failed (a frozen or lost phone, an unreachable sandbox service) is left out. The median and the p10 to p90 tails.
Claude Haiku 5.52.3s, p10–p90 1.4s–4.0s, n = 1.8k step gaps
Claude Sonnet 5.52.5s, p10–p90 895ms–4.1s, n = 664 step gaps
Qwen3.8-2.4T-A95B2.7s, p10–p90 1.9s–11s, n = 1.3k step gaps
Grok 4.72.9s, p10–p90 2.0s–7.1s, n = 1.3k step gaps
GPT-6.1 Terra3.0s, p10–p90 2.4s–5.5s, n = 1.6k step gaps
Claude Fable 5.13.1s, p10–p90 1.9s–6.2s, n = 720 step gaps
Claude Opus 5.53.1s, p10–p90 914ms–4.9s, n = 842 step gaps
GPT-6 Astra3.2s, p10–p90 2.5s–4.5s, n = 773 step gaps
GPT-6 Luna3.2s, p10–p90 2.3s–6.3s, n = 965 step gaps
GPT-6.1 Sol3.3s, p10–p90 2.7s–5.1s, n = 1.6k step gaps
MiniMax M3.1 Flash3.5s, p10–p90 1.9s–9.0s, n = 1.7k step gaps
Kimi K34.0s, p10–p90 2.5s–8.9s, n = 832 step gaps
Claude Fable 54.2s, p10–p90 1.7s–7.1s, n = 1.1k step gaps
Gemini 3.8 Flash5.2s, p10–p90 3.7s–12s, n = 1.2k step gaps
GLM 5.3 Flash5.5s, p10–p90 3.3s–12s, n = 949 step gaps
Hy45.8s, p10–p90 4.4s–8.7s, n = 1.2k step gaps
MiMo V2.6 Pro6.4s, p10–p90 4.6s–11s, n = 862 step gaps
Gemini 4 Argon7.2s, p10–p90 1.3s–15s, n = 917 step gaps
Kimi K3.1 Pro9.7s, p10–p90 7.1s–15s, n = 978 step gaps
Lower is better. Medians. n = 664–1.8k step gaps per model.Coarenaby Coasty
Steadiness
Pace steadiness
Interquartile range of run durations divided by their median.
Muse Wonder 1.00.42, n = 12 runs
Claude Sonnet 5.50.74, n = 84 runs
DeepSeek V4.1 Flash 10170.77, n = 89 runs
Kimi K30.86, n = 86 runs
MiMo V2.6 Pro0.87, n = 84 runs
Gemini 4 Argon0.96, n = 104 runs
Hy40.98, n = 88 runs
Claude Fable 5.11.01, n = 94 runs
Muse Spark 1.31.01, n = 14 runs
MiniMax M3.1 Flash1.05, n = 88 runs
Claude Opus 5.51.09, n = 89 runs
Qwen3.8-2.4T-A95B1.11, n = 99 runs
Gemini 3.8 Flash1.15, n = 105 runs
GLM 5.3 Flash1.16, n = 85 runs
GPT-6 Astra1.17, n = 86 runs
Kimi K3.1 Pro1.30, n = 88 runs
GPT-6 Luna1.32, n = 83 runs
Claude Fable 51.45, n = 88 runs
Claude Haiku 5.51.47, n = 93 runs
GPT-6.1 Terra1.47, n = 135 runs
Grok 4.71.76, n = 84 runs
GPT-6.1 Sol1.82, n = 117 runs
Lower is better. n = 12–135 runs per model.Coarenaby Coasty
Late-step slowdown
For each run with seven or more step gaps (eight or more steps), the median gap in its last quarter of step gaps divided by the median gap in its first quarter, recorded pauses subtracted; the geometric mean over runs. A gap that ends on an infrastructure failure (a frozen or lost phone, an unreachable sandbox service) is left out before the gaps are counted or split.
Not measured: fewer than 100 runs with 7 or more step gaps.
Output tokens
Tokens generated over a run, including reasoning where the provider bills it.
GPT-6 Astra190, p10–p90 105–465, n = 86 runs
GPT-6.1 Sol250, p10–p90 111–2.1k, n = 117 runs
GPT-6.1 Terra273, p10–p90 115–1.5k, n = 135 runs
GPT-6 Luna362, p10–p90 123–1.6k, n = 83 runs
Gemini 4 Argon502, p10–p90 237–1.5k, n = 104 runs
Claude Sonnet 5.5560, p10–p90 290–1.6k, n = 84 runs
Claude Opus 5.5612, p10–p90 294–2.0k, n = 89 runs
Kimi K3639, p10–p90 291–2.2k, n = 86 runs
Muse Wonder 1.0684, p10–p90 566–2.0k, n = 12 runs
Claude Fable 5.1717, p10–p90 347–1.7k, n = 94 runs
Claude Fable 5750, p10–p90 312–3.2k, n = 88 runs
Gemini 3.8 Flash759, p10–p90 285–2.0k, n = 105 runs
MiMo V2.6 Pro799, p10–p90 426–3.2k, n = 84 runs
Kimi K3.1 Pro885, p10–p90 383–2.7k, n = 88 runs
GLM 5.3 Flash957, p10–p90 431–3.1k, n = 85 runs
Muse Spark 1.3964, p10–p90 372–3.6k, n = 14 runs
DeepSeek V4.1 Flash 1017989, p10–p90 351–2.9k, n = 89 runs
Grok 4.71,014, p10–p90 286–7.6k, n = 84 runs
Hy41,034, p10–p90 323–4.0k, n = 88 runs
Claude Haiku 5.51,144, p10–p90 468–4.3k, n = 93 runs
Qwen3.8-2.4T-A95B1,237, p10–p90 565–12k, n = 99 runs
MiniMax M3.1 Flash1,706, p10–p90 667–23k, n = 88 runs
Lower is better. Medians. n = 12–135 runs per model.Coarenaby Coasty
Efficiency
Output tokens per action
A run's output tokens divided by its actions.
GPT-6 Astra26, p10–p90 23.7–38, n = 86 runs
GPT-6.1 Sol27, p10–p90 23.9–132, n = 117 runs
GPT-6.1 Terra33, p10–p90 24.3–140, n = 135 runs
GPT-6 Luna43, p10–p90 30.5–69.2, n = 83 runs
Gemini 4 Argon73, p10–p90 48.5–108, n = 104 runs
Claude Sonnet 5.583, p10–p90 72.5–106, n = 84 runs
Gemini 3.8 Flash85, p10–p90 60.2–132, n = 105 runs
Claude Opus 5.587, p10–p90 76.0–132, n = 89 runs
Muse Wonder 1.091, p10–p90 80.9–146, n = 12 runs
Claude Fable 5.192, p10–p90 75.9–286, n = 94 runs
Kimi K396, p10–p90 76.1–155, n = 86 runs
Claude Fable 5103, p10–p90 81.8–151, n = 88 runs
Grok 4.7104, p10–p90 58.1–227, n = 84 runs
Hy4106, p10–p90 54.4–208, n = 88 runs
DeepSeek V4.1 Flash 1017108, p10–p90 79.1–188, n = 89 runs
Claude Haiku 5.5108, p10–p90 85.7–157, n = 93 runs
MiMo V2.6 Pro110, p10–p90 73.6–190, n = 84 runs
Kimi K3.1 Pro121, p10–p90 92.4–198, n = 88 runs
Muse Spark 1.3122, p10–p90 96.6–182, n = 14 runs
GLM 5.3 Flash132, p10–p90 85.5–228, n = 85 runs
MiniMax M3.1 Flash136, p10–p90 88.3–390, n = 88 runs
Qwen3.8-2.4T-A95B155, p10–p90 95.4–840, n = 99 runs
Lower is better. Medians. n = 12–135 runs per model.Coarenaby Coasty
Input tokens per action
A run's input tokens divided by its actions.
DeepSeek V4.1 Flash 10175,002, p10–p90 4.6k–6.1k, n = 89 runs
GPT-6 Astra5,102, p10–p90 4.8k–5.5k, n = 86 runs
GPT-6.1 Terra5,136, p10–p90 4.7k–5.9k, n = 135 runs
GPT-6.1 Sol5,140, p10–p90 4.7k–6.2k, n = 117 runs
GPT-6 Luna5,152, p10–p90 4.8k–6.1k, n = 83 runs
Gemini 3.8 Flash5,213, p10–p90 4.7k–6.0k, n = 105 runs
Gemini 4 Argon5,521, p10–p90 4.5k–6.1k, n = 104 runs
MiMo V2.6 Pro5,583, p10–p90 5.2k–7.1k, n = 84 runs
Claude Haiku 5.55,715, p10–p90 4.3k–7.3k, n = 93 runs
Hy45,796, p10–p90 5.3k–7.2k, n = 88 runs
Qwen3.8-2.4T-A95B5,962, p10–p90 5.4k–6.7k, n = 99 runs
Kimi K36,037, p10–p90 5.5k–7.2k, n = 86 runs
Kimi K3.1 Pro6,239, p10–p90 5.6k–7.4k, n = 88 runs
GLM 5.3 Flash6,250, p10–p90 5.7k–7.4k, n = 85 runs
Claude Opus 5.56,504, p10–p90 5.1k–7.4k, n = 89 runs
Claude Sonnet 5.56,514, p10–p90 5.4k–7.6k, n = 84 runs
Muse Wonder 1.06,876, p10–p90 6.7k–8.0k, n = 12 runs
MiniMax M3.1 Flash7,103, p10–p90 6.4k–16k, n = 88 runs
Grok 4.77,139, p10–p90 6.3k–11k, n = 84 runs
Muse Spark 1.37,197, p10–p90 6.4k–9.0k, n = 14 runs
Claude Fable 5.17,505, p10–p90 6.0k–8.4k, n = 94 runs
Claude Fable 57,589, p10–p90 7.1k–8.4k, n = 88 runs
Lower is better. Medians. n = 12–135 runs per model.Coarenaby Coasty
Tokens per finished task
Total tokens across all of an agent's runs divided by the number that reported success.
GPT-6 Astra57k, n = 85 completed runs
Gemini 4 Argon57k, n = 101 completed runs
Claude Sonnet 5.558k, n = 83 completed runs
Claude Opus 5.569k, n = 85 completed runs
Claude Fable 5.169k, n = 90 completed runs
GPT-6 Luna80k, n = 79 completed runs
Gemini 3.8 Flash80k, n = 101 completed runs
Kimi K386k, n = 85 completed runs
MiMo V2.6 Pro91k, n = 81 completed runs
GPT-6.1 Terra95k, n = 124 completed runs
GPT-6.1 Sol97k, n = 108 completed runs
Qwen3.8-2.4T-A95B102k, n = 97 completed runs
DeepSeek V4.1 Flash 1017103k, n = 87 completed runs
Muse Spark 1.3103k, n = 14 completed runs
GLM 5.3 Flash104k, n = 83 completed runs
Kimi K3.1 Pro112k, n = 86 completed runs
Hy4121k, n = 86 completed runs
Claude Fable 5127k, n = 84 completed runs
Claude Haiku 5.5144k, n = 90 completed runs
Grok 4.7174k, n = 81 completed runs
Muse Wonder 1.0202k, n = 12 completed runs
MiniMax M3.1 Flash326k, n = 87 completed runs
Lower is better. n = 12–124 completed runs per model.Coarenaby Coasty
Frontier
Cost per run
List-price dollars for one run: input tokens at the vendor's input rate plus output tokens at its output rate, uncached and unbatched. The rate is the one the standings charge: the agent's price row in the agents table where the roster carries one, otherwise the code's price table (vendor list rates read up to 2026-11-02). The median and the tails over every run, finished or not.
GPT-6 Luna$0.004, p10–p90 $0.002–$0.018, n = 83 runs
Claude Haiku 5.5$0.007, p10–p90 $0.002–$0.024, n = 93 runs
GLM 5.3 Flash$0.008, p10–p90 $0.003–$0.028, n = 85 runs
DeepSeek V4.1 Flash 1017$0.013, p10–p90 $0.006–$0.048, n = 89 runs
MiniMax M3.1 Flash$0.015, p10–p90 $0.006–$0.153, n = 88 runs
MiMo V2.6 Pro$0.020, p10–p90 $0.008–$0.065, n = 84 runs
Hy4$0.026, p10–p90 $0.011–$0.088, n = 88 runs
Gemini 3.8 Flash$0.035, p10–p90 $0.012–$0.114, n = 105 runs
GPT-6.1 Terra$0.037, p10–p90 $0.015–$0.153, n = 135 runs
GPT-6.1 Sol$0.084, p10–p90 $0.029–$0.429, n = 117 runs
Gemini 4 Argon$0.094, p10–p90 $0.034–$0.203, n = 104 runs
Claude Sonnet 5.5$0.096, p10–p90 $0.046–$0.203, n = 84 runs
Kimi K3.1 Pro$0.109, p10–p90 $0.038–$0.296, n = 88 runs
Qwen3.8-2.4T-A95B$0.113, p10–p90 $0.048–$0.374, n = 99 runs
Kimi K3$0.135, p10–p90 $0.054–$0.411, n = 86 runs
Grok 4.7$0.153, p10–p90 $0.041–$0.698, n = 84 runs
Claude Opus 5.5$0.193, p10–p90 $0.091–$0.602, n = 89 runs
GPT-6 Astra$0.366, p10–p90 $0.148–$0.977, n = 86 runs
Claude Fable 5.1$0.515, p10–p90 $0.230–$1.38, n = 94 runs
Claude Fable 5$0.563, p10–p90 $0.231–$2.37, n = 88 runs
Lower is better. Medians. n = 83–135 runs per model.Coarenaby Coasty
Cost per finished task
Every list-price dollar the agent spent across all of its runs divided by the number of runs that reported success, uncached, at the rate the standings charge: the agent's price row in the agents table where the roster carries one, otherwise the code's price table (vendor list rates read up to 2026-11-02).
GPT-6 Luna$0.01, 95% interval $0.006–$0.011, n = 79 completed runs
GPT-6 Luna100%, 95% interval 21–100%, n = 1 person-labelled runs
0%25%50%75%100%
Lower is better. Lines are 95% intervals. n = 1–4 person-labelled runs per model.Coarenaby Coasty
Character
Character stamps
Share of the stamps people left on the agent's runs across the five run-level character labels.
Not measured.
Preference
Margin when it won
Mean judge-recorded strength (barely / clearly / not close, as 1-3) on battles this agent won. A battle the arena replayed counts through its judged replay, never its original.
Claude Haiku 5.51.84, n = 19 wins with a recorded strength
GPT-6 Astra1.65, n = 17 wins with a recorded strength
Kimi K3.1 Pro1.44, n = 27 wins with a recorded strength
GLM 5.3 Flash1.43, n = 28 wins with a recorded strength
MiniMax M3.1 Flash1.40, n = 10 wins with a recorded strength
GPT-6 Luna1.35, n = 23 wins with a recorded strength
Grok 4.71.35, n = 23 wins with a recorded strength
Muse Wonder 1.01.33, n = 6 wins with a recorded strength
GPT-6.1 Terra1.31, n = 39 wins with a recorded strength
Qwen3.8-2.4T-A95B1.30, n = 30 wins with a recorded strength
Gemini 3.8 Flash1.30, n = 30 wins with a recorded strength
Claude Fable 51.28, n = 25 wins with a recorded strength
Claude Opus 5.51.24, n = 33 wins with a recorded strength
DeepSeek V4.1 Flash 10171.24, n = 25 wins with a recorded strength
Claude Sonnet 5.51.21, n = 29 wins with a recorded strength
GPT-6.1 Sol1.17, n = 35 wins with a recorded strength
Muse Spark 1.31.17, n = 6 wins with a recorded strength
MiMo V2.6 Pro1.14, n = 29 wins with a recorded strength
Claude Fable 5.11.09, n = 34 wins with a recorded strength
Kimi K31.07, n = 42 wins with a recorded strength
Gemini 4 Argon1.04, n = 27 wins with a recorded strength
Hy41.00, n = 15 wins with a recorded strength
Higher is better. n = 6–42 wins with a recorded strength per model.Coarenaby Coasty
0%
Arena-wide. 95% interval 0–0.2%. n = 2.2k labels.
Gold pass rate
Share of gold items answered as the held-out answer says.
Not measured: under 10 gold items.
Position
First side wins
Share of people's decisive votes won by the side shown first, over the latest 5,000 votes people cast on this board. A model judge's vote (no account, a judge id beginning 'model-') is excluded before the latest 5,000 are taken.
Not measured: fewer than 100 decisive votes.
Task done under destructive probe
Share of decided attempts in the destructive family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft.
Task done under credentials probe
Share of decided attempts in the credentials family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft.
Task done under scope probe
Share of decided attempts in the scope family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft.
GPT-6.1 Terra0.0%, 95% interval 0–2.8%, n = 135 runs
GPT-6.1 Sol0.0%, 95% interval 0–3.2%, n = 117 runs
Gemini 4 Argon0.0%, 95% interval 0–3.6%, n = 104 runs
Gemini 3.8 Flash0.0%, 95% interval 0–3.5%, n = 105 runs
0%1%2%3%4%
Neither direction is better. Lines are 95% intervals. n = 104–135 runs per model.Coarenaby Coasty
Refusals by category
Composition of this model's refusals by the provider's policy category, taxonomy anthropic stop_details, 2026-09; a refusal that carried no category is uncategorised.
Not measured: fewer than 10 refusals.
Not measured: a phone has one screen size (720x1600), so there is no viewport to perturb.
Locale retention
Tasks completed under a changed locale and timezone divided by tasks completed in the clean condition, over the same tasks for the same model.
Not measured: no perturbation lane has run.
GPT-6.1 Terra0.0%, 95% interval 0–11%, n = 31 budgeted runs
Hy40.0%, 95% interval 0–1.1%, n = 341 budgeted runs
Kimi K3.1 Pro0.0%, 95% interval 0–9%, n = 39 budgeted runs
MiniMax M3.1 Flash0.0%, 95% interval 0–0.9%, n = 420 runs
0%5%10%15%20%
Neither direction is better. Lines are 95% intervals. n = 19–420 runs per model.Coarenaby Coasty
Checks ticked
Mean share of the submitter's acceptance checks a judge ticked for the run, over runs in battles where anyone ticked anything. Each run is a share of its own task's checks first, so a task with two checks and a task with eight weigh the same.
Grok 4.798%, 95% interval 96–100%, n = 52 checked runs
MiniMax M3.1 Flash99%, 95% interval 97–99%, n = 420 runs
60%70%80%90%100%
Neither direction is better. Lines are 95% intervals. n = 19–420 runs per model.Coarenaby Coasty
Verification coverage
Share of runs an independent evaluator could decide: the task carries executable assertions, the end state was recorded, and the checker returned success or failure rather than declining.
Not measured: verification sweep has not run over this window.
Verified completion
Of runs the agent declared complete and an evaluator could decide, the share the evaluator confirmed against the recorded end state.
Not measured: verification sweep has not run over this window.
Claimed over verified
Runs the agent declared complete divided by runs an evaluator confirmed, over runs with a decided verdict. 1.0 means every claim was confirmed; 2.0 means half were.
Not measured: verification sweep has not run over this window.
Actions to finish
Actions executed, counting only runs that reported success.
Gemini 4 Argon5.0, p10–p90 2–16, n = 41 completed runs
Claude Sonnet 5.55.0, p10–p90 2–27.2, n = 50 completed runs
Claude Haiku 5.55.0, p10–p90 2–47, n = 46 completed runs
Claude Fable 55.5, p10–p90 2–27.6, n = 48 completed runs
GPT-6.1 Terra6.0, p10–p90 2–25.2, n = 29 completed runs
Kimi K3.1 Pro6.0, p10–p90 3–40.2, n = 35 completed runs
Muse Wonder 1.06.0, p10–p90 2–39, n = 41 completed runs
Claude Fable 5.16.0, p10–p90 2–44.6, n = 53 completed runs
GPT-6 Luna6.5, p10–p90 2–46, n = 56 completed runs
Claude Opus 5.57.0, p10–p90 2–27.2, n = 39 completed runs
GPT-6 Astra7.0, p10–p90 2–52.4, n = 49 completed runs
Kimi K37.5, p10–p90 3–44.4, n = 48 completed runs
GLM 5.3 Flash7.5, p10–p90 3–38, n = 16 completed runs
MiMo V2.6 Pro8.0, p10–p90 3–32.6, n = 42 completed runs
Hy49.0, p10–p90 3–41, n = 333 completed runs
GPT-6.1 Sol10.0, p10–p90 2.40–25.4, n = 33 completed runs
Qwen3.8-2.4T-A95B10.0, p10–p90 3–60.4, n = 53 completed runs
Muse Spark 1.310.0, p10–p90 3.20–48.0, n = 53 completed runs
Gemini 3.8 Flash11.0, p10–p90 6–46, n = 52 completed runs
MiniMax M3.1 Flash12.0, p10–p90 4–56.1, n = 394 completed runs
Grok 4.713.0, p10–p90 2.10–61.1, n = 52 completed runs
Lower is better. Medians. n = 16–394 completed runs per model.Coarenaby Coasty
Detour factor
Geometric mean of (this agent's actions / the opponent's actions) across battles where BOTH sides completed the identical task.
Claude Sonnet 5.50.41x, n = 50 paired battles
Claude Haiku 5.50.55x, n = 44 paired battles
GPT-6.1 Terra0.55x, n = 26 paired battles
Claude Fable 5.10.56x, n = 50 paired battles
Gemini 4 Argon0.59x, n = 41 paired battles
Claude Fable 50.60x, n = 46 paired battles
GPT-6 Luna0.62x, n = 53 paired battles
Claude Opus 5.50.62x, n = 38 paired battles
GLM 5.3 Flash0.63x, n = 16 paired battles
GPT-6 Astra0.65x, n = 48 paired battles
Kimi K3.1 Pro0.69x, n = 35 paired battles
Muse Wonder 1.00.70x, n = 40 paired battles
MiMo V2.6 Pro0.72x, n = 42 paired battles
GPT-6.1 Sol0.73x, n = 30 paired battles
Qwen3.8-2.4T-A95B0.75x, n = 52 paired battles
Kimi K30.76x, n = 46 paired battles
Muse Spark 1.30.81x, n = 52 paired battles
Grok 4.70.97x, n = 50 paired battles
Gemini 3.8 Flash1.03x, n = 51 paired battles
Hy41.12x, n = 318 paired battles
DeepSeek V4.1 Flash 10171.29x, n = 313 paired battles
MiniMax M3.1 Flash1.73x, n = 375 paired battles
Lower is better. n = 16–375 paired battles per model.Coarenaby Coasty
Actions relative to best
Geometric mean over the agent's completed runs of its actions divided by the fewest actions any other agent needed to complete the same task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL.
Claude Sonnet 5.50.75x, 95% interval 0.65x–0.85x, n = 40 completed runs on shared tasks
Claude Fable 5.10.94x, 95% interval 0.81x–1.10x, n = 39 completed runs on shared tasks
Claude Opus 5.50.95x, 95% interval 0.85x–1.07x, n = 32 completed runs on shared tasks
Gemini 4 Argon0.98x, 95% interval 0.81x–1.19x, n = 36 completed runs on shared tasks
GPT-6 Luna1.03x, 95% interval 0.90x–1.19x, n = 48 completed runs on shared tasks
Claude Haiku 5.51.04x, 95% interval 0.91x–1.19x, n = 36 completed runs on shared tasks
Claude Fable 51.05x, 95% interval 0.93x–1.20x, n = 34 completed runs on shared tasks
GPT-6 Astra1.16x, 95% interval 0.96x–1.41x, n = 38 completed runs on shared tasks
Qwen3.8-2.4T-A95B1.18x, 95% interval 1.03x–1.38x, n = 43 completed runs on shared tasks
MiMo V2.6 Pro1.23x, 95% interval 1.06x–1.44x, n = 36 completed runs on shared tasks
Muse Wonder 1.01.24x, 95% interval 1.11x–1.38x, n = 33 completed runs on shared tasks
Kimi K31.24x, 95% interval 1.07x–1.46x, n = 41 completed runs on shared tasks
Muse Spark 1.31.47x, 95% interval 1.26x–1.69x, n = 40 completed runs on shared tasks
Hy41.53x, 95% interval 1.44x–1.64x, n = 271 completed runs on shared tasks
Grok 4.71.54x, 95% interval 1.37x–1.73x, n = 41 completed runs on shared tasks
Gemini 3.8 Flash1.66x, 95% interval 1.42x–1.92x, n = 41 completed runs on shared tasks
DeepSeek V4.1 Flash 10171.75x, 95% interval 1.65x–1.86x, n = 276 completed runs on shared tasks
MiniMax M3.1 Flash2.24x, 95% interval 2.09x–2.41x, n = 282 completed runs on shared tasks
0.5x1x1.5x2x2.5x
Lower is better. Lines are 95% intervals. n = 32–282 completed runs on shared tasks per model.Coarenaby Coasty
Wasted motion
Repeated itself
Actions byte-identical to the action immediately before them.
Claude Fable 5<1%, 95% interval 0.03–0.9%, n = 612 actions
Grok 4.7<1%, 95% interval 0.09–0.8%, n = 1.1k actions
Kimi K3.1 Pro<1%, 95% interval 0.1–1.4%, n = 512 actions
Gemini 3.8 Flash<1%, 95% interval 0.2–1.1%, n = 966 actions
Gemini 4 Argon<1%, 95% interval 0.1–1.6%, n = 444 actions
Claude Opus 5.51%, 95% interval 0.3–1.9%, n = 526 actions
Qwen3.8-2.4T-A95B1%, 95% interval 0.5–1.7%, n = 1.1k actions
Hy41%, 95% interval 0.8–1.3%, n = 5.3k actions
Claude Sonnet 5.51%, 95% interval 0.5–2.8%, n = 419 actions
Muse Spark 1.31%, 95% interval 0.8–2.2%, n = 1.1k actions
GPT-6.1 Sol2%, 95% interval 1–3.3%, n = 556 actions
MiMo V2.6 Pro2%, 95% interval 1.2–3.4%, n = 646 actions
GPT-6 Luna2%, 95% interval 1.4–3.4%, n = 839 actions
Muse Wonder 1.03%, 95% interval 1.7–4%, n = 789 actions
Muse Wonder 1.02.5%, 95% interval 1.7–3.8%, n = 831 steps
Claude Haiku 5.52.8%, 95% interval 1.9–4.1%, n = 891 steps
GLM 5.3 Flash3.9%, 95% interval 2.3–6.7%, n = 306 steps
0%2%4%6%8%
Lower is better. Lines are 95% intervals. n = 306–11k steps per model.Coarenaby Coasty
Explicitly inert
Actions the browser explicitly reported as having moved nothing: scrolling past the end of a document, scrolling a panel that does not scroll.
Not measured: no run here was a browser run or used the browser's scroll, the only action that reports moving nothing.
Exploration
Looking vs doing
Look-class actions (a screenshot, reading the page or its elements, a hover, a pointer move, a phone wait) as a share of all actions that either looked or changed something. Refused tool calls executed nothing and are in neither side.
GPT-6.1 Terra13%, 95% interval 9.5–16%, n = 376 actions
GPT-6.1 Sol11%, 95% interval 8.8–14%, n = 515 actions
Hy410%, 95% interval 8.9–11%, n = 4.9k actions
Kimi K3.1 Pro9%, 95% interval 6.9–12%, n = 469 actions
Muse Wonder 1.013%, 95% interval 11–15%, n = 736 actions
Gemini 4 Argon9%, 95% interval 6.9–13%, n = 391 actions
Claude Opus 5.53%, 95% interval 1.4–4.3%, n = 479 actions
GPT-6 Astra10%, 95% interval 8.5–13%, n = 818 actions
Claude Fable 59%, 95% interval 6.9–12%, n = 554 actions
Claude Fable 5.16%, 95% interval 4.2–7.4%, n = 790 actions
Grok 4.76%, 95% interval 5–7.9%, n = 1.1k actions
Qwen3.8-2.4T-A95B8%, 95% interval 6.1–9.3%, n = 1.1k actions
Kimi K312%, 95% interval 9.3–14%, n = 668 actions
GPT-6 Luna14%, 95% interval 12–17%, n = 764 actions
Claude Sonnet 5.512%, 95% interval 9.3–16%, n = 374 actions
Muse Spark 1.34%, 95% interval 3–5.4%, n = 991 actions
Gemini 3.8 Flash4%, 95% interval 3–5.6%, n = 902 actions
MiMo V2.6 Pro10%, 95% interval 7.9–13%, n = 596 actions
Claude Haiku 5.517%, 95% interval 15–20%, n = 783 actions
GLM 5.3 Flash17%, 95% interval 13–22%, n = 271 actions
MiniMax M3.1 Flash18%, 95% interval 15–21%, n = 625 navigations
0%20%40%60%
Neither direction is better. Lines are 95% intervals. n = 31–625 navigations per model.Coarenaby Coasty
Action variety
Shannon entropy of the run's distribution over the ten action classes, normalized by log2 of the number of classes its mode OFFERS: ten in a browser, nine on a desktop (no scroll verb), nine on a phone (no script or shell). A desktop run from 7 to 9 August 2026 that used the browser's scroll, which desktop mode then still offered, is normalized by ten.
GPT-6.1 Terra0.54, p10–p90 0.29–0.76, n = 26 runs
GPT-6.1 Sol0.61, p10–p90 0.32–0.74, n = 33 runs
Hy40.47, p10–p90 0.29–0.74, n = 319 runs
Kimi K3.1 Pro0.48, p10–p90 0.29–0.78, n = 34 runs
Muse Wonder 1.00.49, p10–p90 0.29–0.71, n = 33 runs
Gemini 4 Argon0.44, p10–p90 0.26–0.75, n = 36 runs
Claude Opus 5.50.44, p10–p90 0.29–0.72, n = 35 runs
GPT-6 Astra0.47, p10–p90 0.26–0.74, n = 42 runs
Claude Fable 50.47, p10–p90 0.29–0.74, n = 41 runs
Claude Fable 5.10.47, p10–p90 0.29–0.72, n = 46 runs
Grok 4.70.44, p10–p90 0.29–0.66, n = 46 runs
Qwen3.8-2.4T-A95B0.50, p10–p90 0.29–0.67, n = 51 runs
Kimi K30.42, p10–p90 0.26–0.70, n = 46 runs
GPT-6 Luna0.58, p10–p90 0.31–0.74, n = 42 runs
Claude Sonnet 5.50.47, p10–p90 0.26–0.72, n = 44 runs
Muse Spark 1.30.41, p10–p90 0.27–0.67, n = 53 runs
Gemini 3.8 Flash0.32, p10–p90 0.23–0.66, n = 53 runs
MiMo V2.6 Pro0.42, p10–p90 0.29–0.71, n = 47 runs
Claude Haiku 5.50.55, p10–p90 0.31–0.76, n = 36 runs
GLM 5.3 Flash0.45, p10–p90 0.29–0.71, n = 17 runs
DeepSeek V4.1 Flash 10170.41, p10–p90 0.28–0.74, n = 332 runs
MiniMax M3.1 Flash0.45, p10–p90 0.27–0.74, n = 415 runs
Neither direction is better. Medians. n = 17–415 runs per model.Coarenaby Coasty
Action mix
Share of each kind of action across the agent's executed actions, over ten classes: navigate (a URL, back, forward, reload, the phone's Back), point (a click, tap, double tap or long press), drag or swipe, scroll, type (text or a saved credential), key, switch app or tab (a tab, an app, Home, Recents), look (a screenshot, the page text or elements, a hover, a pointer move, a phone wait), script or shell, and finish (done or a delivered file).
Navigate
Point
Drag or swipe
Type
Key
Switch app or tab
Look
Script or shell
Finish
GPT-6.1 Terra439
12%
19%
0%
8%
17%
2%
11%
17%
14%
GPT-6.1 Sol593
9%
22%
1%
15%
15%
1%
10%
15%
13%
Hy45.6k
8%
16%
1%
4%
4%
1%
8%
43%
14%
Kimi K3.1 Pro551
9%
15%
0%
8%
7%
1%
8%
36%
15%
Muse Wonder 1.0830
5%
23%
0%
5%
22%
1%
11%
22%
11%
Gemini 4 Argon487
12%
18%
0%
3%
2%
1%
8%
36%
20%
Claude Opus 5.5565
6%
39%
0%
7%
7%
1%
2%
23%
15%
GPT-6 Astra929
6%
40%
1%
9%
10%
0%
9%
13%
12%
Claude Fable 5659
9%
22%
2%
8%
5%
1%
8%
31%
16%
Claude Fable 5.1908
7%
14%
0%
4%
24%
0%
5%
33%
13%
Grok 4.71.2k
6%
18%
0%
4%
3%
0%
6%
53%
10%
Qwen3.8-2.4T-A95B1.2k
7%
26%
1%
6%
4%
1%
7%
38%
10%
Kimi K3777
6%
14%
0%
4%
14%
1%
10%
37%
14%
GPT-6 Luna895
6%
23%
4%
7%
12%
1%
12%
19%
15%
Claude Sonnet 5.5469
13%
14%
2%
5%
6%
1%
10%
29%
20%
Muse Spark 1.31.1k
5%
16%
1%
4%
6%
1%
4%
52%
11%
Gemini 3.8 Flash1.0k
5%
9%
1%
1%
5%
1%
4%
62%
11%
MiMo V2.6 Pro689
8%
20%
1%
11%
7%
1%
9%
29%
13%
Claude Haiku 5.5890
10%
29%
1%
6%
6%
1%
15%
20%
12%
GLM 5.3 Flash306
13%
12%
0%
8%
6%
1%
15%
34%
11%
DeepSeek V4.1 Flash 10177.2k
6%
25%
1%
7%
9%
1%
5%
34%
11%
MiniMax M3.1 Flash11k
6%
24%
1%
7%
9%
1%
5%
39%
8%
Share of each model's actions; n after each name. Not observed: scroll.Coarenaby Coasty
GPT-6.1 Terra · Point19%, point, 85 of 439 actions
GPT-6.1 Sol · Point22%, point, 129 of 593 actions
Hy4 · Script or shell43%, script or shell, 2434 of 5.6k actions
Kimi K3.1 Pro · Script or shell36%, script or shell, 197 of 551 actions
Muse Wonder 1.0 · Point23%, point, 187 of 830 actions
Gemini 4 Argon · Script or shell36%, script or shell, 177 of 487 actions
Claude Opus 5.5 · Point39%, point, 223 of 565 actions
GPT-6 Astra · Point40%, point, 372 of 929 actions
Claude Fable 5 · Script or shell31%, script or shell, 201 of 659 actions
Claude Fable 5.1 · Script or shell33%, script or shell, 302 of 908 actions
Grok 4.7 · Script or shell53%, script or shell, 639 of 1.2k actions
Qwen3.8-2.4T-A95B · Script or shell38%, script or shell, 441 of 1.2k actions
Kimi K3 · Script or shell37%, script or shell, 284 of 777 actions
GPT-6 Luna · Point23%, point, 209 of 895 actions
Claude Sonnet 5.5 · Script or shell29%, script or shell, 136 of 469 actions
Muse Spark 1.3 · Script or shell52%, script or shell, 582 of 1.1k actions
Gemini 3.8 Flash · Script or shell62%, script or shell, 635 of 1.0k actions
MiMo V2.6 Pro · Script or shell29%, script or shell, 203 of 689 actions
Claude Haiku 5.5 · Point29%, point, 257 of 890 actions
GLM 5.3 Flash · Script or shell34%, script or shell, 105 of 306 actions
DeepSeek V4.1 Flash 1017 · Script or shell34%, script or shell, 2467 of 7.2k actions
MiniMax M3.1 Flash · Script or shell39%, script or shell, 4269 of 11k actions
Each model's most common kind. n = 306–11k actions per model.Coarenaby Coasty
By task tier
Completed, easy tasks
Share of runs that reported success on tasks whose best observed completion took 5 or fewer actions, the tier set by the fewest actions any agent needed to complete the task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL.
MiniMax M3.1 Flash99%, 95% interval 96–100%, n = 147 runs
96%97%98%99%100%
Higher is better. Lines are 95% intervals. n = 129–147 runs per model.Coarenaby Coasty
Completed, medium tasks
Share of runs that reported success on tasks whose best observed completion took 6 to 10 actions, the tier set by the fewest actions any agent needed to complete the task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL.
MiniMax M3.1 Flash98%, 95% interval 92–100%, n = 66 runs
Hy497%, 95% interval 90–99%, n = 70 runs
90%92.5%95%97.5%100%
Higher is better. Lines are 95% intervals. n = 66–74 runs per model.Coarenaby Coasty
Completed, hard tasks
Share of runs that reported success on tasks whose best observed completion took 11 or more actions, the tier set by the fewest actions any agent needed to complete the task, on tasks with three or more completed runs from two or more agents in the window, where a task is its prompt (whitespace collapsed, trimmed, lowercased) and its starting URL.
MiniMax M3.1 Flash92%, 95% interval 84–96%, n = 77 runs
80%85%90%95%100%
Higher is better. Lines are 95% intervals. n = 72–77 runs per model.Coarenaby Coasty
Bypass
Constructed URLs
Share of navigations after the first that carry a query string: the agent wrote a search or a filter into the address instead of using the page's own controls.
Hy435%, 95% interval 29–41%, n = 260 later navigations
DeepSeek V4.1 Flash 101739%, 95% interval 33–45%, n = 256 later navigations
MiniMax M3.1 Flash31%, 95% interval 27–35%, n = 421 later navigations
25%30%35%40%45%50%
Neither direction is better. Lines are 95% intervals. n = 256–421 later navigations per model.Coarenaby Coasty
Interface bypass
Share of actions in browser battles that ran a shell, a script or a direct DOM write (os_bash, os_run, execute_js, set_element_value) instead of driving the page. A battle with no recorded mode is a browser battle (every battle from before modes existed was one).
Not measured: fewer than 500 actions in browser battles.
Worst error streak
Longest run of consecutive failing actions within a run, the model's errors only: an infrastructure failure (a frozen or lost phone, an unreachable sandbox service) neither extends a streak nor ends one.
GPT-6.1 Terra0, p10–p90 0–1, n = 31 runs
GPT-6.1 Sol0, p10–p90 0–1, n = 37 runs
Hy40, p10–p90 0–1, n = 341 runs
Kimi K3.1 Pro0, p10–p90 0–0.20, n = 39 runs
Muse Wonder 1.00, p10–p90 0–0, n = 42 runs
Gemini 4 Argon0, p10–p90 0–0, n = 43 runs
Claude Opus 5.50, p10–p90 0–1, n = 40 runs
GPT-6 Astra0, p10–p90 0–0.10, n = 50 runs
Claude Fable 50, p10–p90 0–0, n = 49 runs
Claude Fable 5.10, p10–p90 0–0, n = 54 runs
Grok 4.70, p10–p90 0–1, n = 52 runs
Qwen3.8-2.4T-A95B0, p10–p90 0–1, n = 54 runs
Kimi K30, p10–p90 0–1, n = 50 runs
GPT-6 Luna0, p10–p90 0–0, n = 56 runs
Claude Sonnet 5.50, p10–p90 0–0.80, n = 53 runs
Muse Spark 1.30, p10–p90 0–1, n = 55 runs
Gemini 3.8 Flash0, p10–p90 0–1, n = 53 runs
MiMo V2.6 Pro0, p10–p90 0–1, n = 48 runs
Claude Haiku 5.50, p10–p90 0–0, n = 47 runs
GLM 5.3 Flash0, p10–p90 0–0.20, n = 19 runs
DeepSeek V4.1 Flash 10170, p10–p90 0–1, n = 336 runs
MiniMax M3.1 Flash0, p10–p90 0–1, n = 420 runs
Lower is better. Medians. n = 19–420 runs per model.Coarenaby Coasty
Rejected actions
Share of actions the harness refused to execute because the call was malformed: an unknown verb, a field the schema rejected, or a page verb on a desktop run. Each refusal is recorded as a step with the same sentence in every lane.
GPT-6.1 Sol0.0%, 95% interval 0–0.6%, n = 593 actions
Kimi K3.1 Pro0.0%, 95% interval 0–0.7%, n = 551 actions
GPT-6 Astra0.0%, 95% interval 0–0.4%, n = 929 actions
Grok 4.70.0%, 95% interval 0–0.3%, n = 1.2k actions
Kimi K30.0%, 95% interval 0–0.5%, n = 777 actions
GPT-6 Luna0.0%, 95% interval 0–0.4%, n = 895 actions
Muse Spark 1.30.0%, 95% interval 0–0.3%, n = 1.1k actions
Gemini 3.8 Flash0.0%, 95% interval 0–0.4%, n = 1.0k actions
Hy4<0.1%, 95% interval 0.003–0.1%, n = 5.6k actions
Qwen3.8-2.4T-A95B0.1%, 95% interval 0.02–0.5%, n = 1.2k actions
Claude Fable 5.10.1%, 95% interval 0.02–0.6%, n = 909 actions
Claude Haiku 5.50.1%, 95% interval 0.02–0.6%, n = 891 actions
Muse Wonder 1.00.1%, 95% interval 0.02–0.7%, n = 831 actions
Claude Opus 5.50.2%, 95% interval 0.03–1%, n = 566 actions
Claude Fable 50.3%, 95% interval 0.08–1.1%, n = 661 actions
MiniMax M3.1 Flash0.5%, 95% interval 0.4–0.7%, n = 11k actions
MiMo V2.6 Pro0.6%, 95% interval 0.2–1.5%, n = 693 actions
0%0.5%1%1.5%
Lower is better. Lines are 95% intervals. n = 551–11k actions per model.Coarenaby Coasty
Resilience
Recovered
Failing actions whose NEXT action did not also fail. A failing action is the model's error, not one the infrastructure wrote; a tool call the arena refused counts as one, so a refused next call is not a recovery.
GPT-6.1 Terra100%, 95% interval 61–100%, n = 6 failed actions
GPT-6.1 Sol79%, 95% interval 52–92%, n = 14 failed actions
Kimi K375%, 95% interval 51–90%, n = 16 failed actions
GPT-6 Luna75%, 95% interval 41–93%, n = 8 failed actions
20%40%60%80%100%
Higher is better. Lines are 95% intervals. n = 2–273 failed actions per model.Coarenaby Coasty
Retried the same thing
Failing actions immediately followed by the byte-identical action: the same call, expecting a different answer. A failing action is the model's error, not one the infrastructure wrote; a tool call the arena refused counts as one.
GPT-6.1 Terra0%, 95% interval 0–39%, n = 6 failed actions
Kimi K3.1 Pro0%, 95% interval 0–22%, n = 14 failed actions
Claude Sonnet 5.518%, 95% interval 5.1–48%, n = 11 failed actions
Claude Haiku 5.520%, 95% interval 3.6–62%, n = 5 failed actions
GPT-6 Luna25%, 95% interval 7.1–59%, n = 8 failed actions
0%20%40%60%80%
Lower is better. Lines are 95% intervals. n = 2–273 failed actions per model.Coarenaby Coasty
Run level
Repaired after an error
Of runs with at least one failed action, the share whose recorded outcome was completed. A failed action is an error the action met; one the infrastructure wrote (a frozen or lost phone, an unreachable sandbox service) does not make a run an error run.
Higher is better. Lines are 95% intervals. n = 53–123 error runs per model.Coarenaby Coasty
Abandoned after an error
Of runs with at least one failed action, the share that ended within three actions of their last failed action without completing. A failed action is an error the action met, not one the infrastructure wrote (a frozen or lost phone, an unreachable sandbox service).
Qwen3.8-2.4T-A95B32%, 95% interval 27–38%, n = 309 clicks
GLM 5.3 Flash38%, 95% interval 24–54%, n = 37 clicks
MiniMax M3.1 Flash39%, 95% interval 38–41%, n = 2.6k clicks
Claude Haiku 5.542%, 95% interval 36–48%, n = 257 clicks
GPT-6 Astra53%, 95% interval 48–58%, n = 372 clicks
Claude Opus 5.556%, 95% interval 49–62%, n = 223 clicks
0%20%40%60%80%
Lower is better. Lines are 95% intervals. n = 37–2.6k clicks per model.Coarenaby Coasty
Click spread
Mean pairwise distance between a run's presses at a point, divided by the diagonal of its recorded viewport.
GPT-6.1 Terra0.24, p10–p90 0.14–0.34, n = 11 runs
GPT-6.1 Sol0.24, p10–p90 0.07–0.33, n = 13 runs
Hy40.22, p10–p90 0.05–0.34, n = 98 runs
Kimi K3.1 Pro0.16, p10–p90 0.07–0.24, n = 12 runs
Muse Wonder 1.00.18, p10–p90 0.10–0.27, n = 11 runs
Gemini 4 Argon0.12, p10–p90 0.05–0.32, n = 10 runs
Claude Opus 5.50.18, p10–p90 0.08–0.34, n = 10 runs
GPT-6 Astra0.22, p10–p90 0.07–0.36, n = 16 runs
Claude Fable 50.21, p10–p90 0.11–0.30, n = 13 runs
Claude Fable 5.10.21, p10–p90 0.11–0.32, n = 16 runs
Grok 4.70.17, p10–p90 0.08–0.26, n = 16 runs
Qwen3.8-2.4T-A95B0.20, p10–p90 0.07–0.33, n = 21 runs
Kimi K30.14, p10–p90 0.02–0.27, n = 14 runs
GPT-6 Luna0.22, p10–p90 0.09–0.33, n = 16 runs
Claude Sonnet 5.50.18, p10–p90 0.09–0.39, n = 7 runs
Muse Spark 1.30.17, p10–p90 0.08–0.30, n = 17 runs
Gemini 3.8 Flash0.19, p10–p90 0.04–0.35, n = 14 runs
MiMo V2.6 Pro0.20, p10–p90 0.13–0.38, n = 11 runs
Claude Haiku 5.50.18, p10–p90 0.05–0.27, n = 14 runs
GLM 5.3 Flash0.18, p10–p90 0.05–0.27, n = 4 runs
DeepSeek V4.1 Flash 10170.18, p10–p90 0.06–0.29, n = 113 runs
MiniMax M3.1 Flash0.21, p10–p90 0.09–0.35, n = 155 runs
Neither direction is better. Medians. n = 4–155 runs per model.Coarenaby Coasty
Clicks that did nothing
Presses at a point (a click, a desktop click, a phone tap, double tap or long press) that landed on nothing. Withheld on both boards: no press's observation records whether the press hit a control.
Not measured: no press records whether it hit a control: a click's, desktop click's or tap's observation echoes the press (a desktop click also names the window that took the keyboard, or says none did, and a click that opened a tab says so), and the errors a press meets (a timeout, an unreachable sandbox, a frozen phone) say nothing about where it landed.
Gesture
Points per drag
Coordinate pairs supplied per path drag (drag, desktop drag). A phone swipe has two endpoints by contract and is not a path drag.
Gemini 4 Argon8.0, p10–p90 6.40–9.60, n = 2 path drags
Claude Fable 57.0, p10–p90 3–36.4, n = 13 path drags
Kimi K3.1 Pro6.0, p10–p90 5.20–6.80, n = 2 path drags
GPT-6 Luna5.0, p10–p90 2–6, n = 39 path drags
MiniMax M3.1 Flash4.0, p10–p90 2–10, n = 77 path drags
Kimi K33.5, p10–p90 3.10–3.90, n = 2 path drags
GPT-6.1 Terra3.0, p10–p90 3–3, n = 1 path drags
Muse Wonder 1.03.0, p10–p90 3–3.70, n = 4 path drags
Claude Opus 5.53.0, p10–p90 3–3, n = 2 path drags
GPT-6 Astra3.0, p10–p90 3–4, n = 11 path drags
Claude Fable 5.13.0, p10–p90 3–4.60, n = 3 path drags
Lower is better. Lines are 95% intervals. n = 1–99 path drags per model.Coarenaby Coasty
Modified clicks
Browser clicks holding Alt, Control, Meta or Shift.
Not measured: no click in these runs records modifier keys: desktop clicks and phone taps carry no modifier field.
Text entry
Characters per typing action
Characters (code points) supplied per text-entry action: type, desktop type, phone type. A saved credential supplies no visible text and is not counted.
GPT-6.1 Terra8.0, p10–p90 3.30–147, n = 34 text-entry actions
GPT-6.1 Sol5.0, p10–p90 3–32.5, n = 86 text-entry actions
Hy412.0, p10–p90 3–42, n = 253 text-entry actions
Kimi K3.1 Pro11.0, p10–p90 3.50–33, n = 46 text-entry actions
Muse Wonder 1.012.0, p10–p90 6.80–21.5, n = 40 text-entry actions
Gemini 4 Argon11.0, p10–p90 3.40–18.6, n = 15 text-entry actions
Claude Opus 5.58.5, p10–p90 2–17, n = 40 text-entry actions
GPT-6 Astra10.5, p10–p90 2.80–29.1, n = 80 text-entry actions
Claude Fable 510.0, p10–p90 3.10–23.8, n = 52 text-entry actions
Claude Fable 5.118.0, p10–p90 8–41.7, n = 34 text-entry actions
Grok 4.76.5, p10–p90 2–25.5, n = 48 text-entry actions
Qwen3.8-2.4T-A95B11.0, p10–p90 6–30.2, n = 69 text-entry actions
Kimi K311.0, p10–p90 2–51.8, n = 33 text-entry actions
GPT-6 Luna5.0, p10–p90 2.60–26, n = 67 text-entry actions
Claude Sonnet 5.58.0, p10–p90 3–27, n = 24 text-entry actions
Muse Spark 1.310.0, p10–p90 5–34, n = 41 text-entry actions
Gemini 3.8 Flash6.5, p10–p90 3.30–32.3, n = 14 text-entry actions
MiMo V2.6 Pro9.5, p10–p90 1–28.8, n = 74 text-entry actions
Claude Haiku 5.59.0, p10–p90 3–31, n = 52 text-entry actions
GLM 5.3 Flash12.0, p10–p90 6–40.8, n = 23 text-entry actions
MiniMax M3.1 Flash9.0, p10–p90 2–42, n = 755 text-entry actions
Neither direction is better. Medians. n = 14–755 text-entry actions per model.Coarenaby Coasty
Keys vs strings
Key-class actions (press_key, desktop key, phone key) as a share of key-class plus type-class actions.
GPT-6.1 Terra68%, 95% interval 59–76%, n = 107 text actions
GPT-6.1 Sol51%, 95% interval 44–58%, n = 175 text actions
Hy448%, 95% interval 44–52%, n = 487 text actions
Kimi K3.1 Pro45%, 95% interval 35–56%, n = 84 text actions
Muse Wonder 1.082%, 95% interval 77–87%, n = 224 text actions
Gemini 4 Argon40%, 95% interval 23–59%, n = 25 text actions
Claude Opus 5.548%, 95% interval 37–59%, n = 77 text actions
GPT-6 Astra53%, 95% interval 45–60%, n = 170 text actions
Claude Fable 537%, 95% interval 28–48%, n = 83 text actions
Claude Fable 5.186%, 95% interval 81–90%, n = 248 text actions
Grok 4.741%, 95% interval 31–52%, n = 82 text actions
Qwen3.8-2.4T-A95B41%, 95% interval 33–50%, n = 117 text actions
Kimi K377%, 95% interval 69–83%, n = 141 text actions
GPT-6 Luna62%, 95% interval 54–69%, n = 175 text actions
Claude Sonnet 5.555%, 95% interval 41–67%, n = 53 text actions
Muse Spark 1.363%, 95% interval 53–71%, n = 110 text actions
Gemini 3.8 Flash79%, 95% interval 68–87%, n = 67 text actions
MiMo V2.6 Pro39%, 95% interval 31–48%, n = 122 text actions
Claude Haiku 5.552%, 95% interval 43–61%, n = 109 text actions
GLM 5.3 Flash44%, 95% interval 30–59%, n = 41 text actions
DeepSeek V4.1 Flash 101756%, 95% interval 53–59%, n = 1.2k text actions
MiniMax M3.1 Flash56%, 95% interval 54–58%, n = 1.7k text actions
20%40%60%80%100%
Neither direction is better. Lines are 95% intervals. n = 25–1.7k text actions per model.Coarenaby Coasty
Time to first action
Elapsed milliseconds when the first action resolved. A run whose first action the infrastructure failed (a frozen or lost phone, an unreachable sandbox service) is left out: that wait is the machine's.
Claude Haiku 5.59s, p10–p90 6.5s–25s, n = 47 runs
DeepSeek V4.1 Flash 101710s, p10–p90 6.8s–22s, n = 336 runs
GPT-6.1 Sol10s, p10–p90 7.9s–26s, n = 37 runs
Grok 4.710s, p10–p90 7.0s–1m 12s, n = 52 runs
GPT-6 Luna10s, p10–p90 6.6s–28s, n = 56 runs
Claude Sonnet 5.510s, p10–p90 6.4s–23s, n = 53 runs
Claude Fable 5.110s, p10–p90 6.4s–43s, n = 54 runs
GPT-6.1 Terra10s, p10–p90 7.9s–22s, n = 31 runs
MiniMax M3.1 Flash10s, p10–p90 6.2s–44s, n = 420 runs
Muse Wonder 1.010s, p10–p90 7.1s–25s, n = 42 runs
Muse Spark 1.311s, p10–p90 7.2s–27s, n = 55 runs
GPT-6 Astra11s, p10–p90 7.8s–25s, n = 50 runs
Claude Opus 5.511s, p10–p90 7.3s–26s, n = 40 runs
Claude Fable 513s, p10–p90 8.4s–32s, n = 49 runs
Qwen3.8-2.4T-A95B13s, p10–p90 7.1s–1m 08s, n = 54 runs
Hy415s, p10–p90 9.2s–1m 24s, n = 339 runs
Kimi K316s, p10–p90 8.3s–47s, n = 50 runs
Gemini 3.8 Flash17s, p10–p90 13s–28s, n = 53 runs
MiMo V2.6 Pro22s, p10–p90 9.1s–1m 47s, n = 47 runs
Gemini 4 Argon24s, p10–p90 11s–1m 01s, n = 43 runs
GLM 5.3 Flash33s, p10–p90 9.1s–1m 42s, n = 18 runs
Kimi K3.1 Pro34s, p10–p90 15s–1m 38s, n = 39 runs
Lower is better. Medians. n = 18–420 runs per model.Coarenaby Coasty
Milliseconds per action
Median gap between consecutive step timestamps within a run, the first gap measured from the run's start. A gap that ends on an action the infrastructure failed (a frozen or lost phone, an unreachable sandbox service) is left out: it is the machine's wait, not the model's.
Claude Haiku 5.53s, p10–p90 719ms–9.3s, n = 47 runs
DeepSeek V4.1 Flash 10173s, p10–p90 2.4s–6.5s, n = 336 runs
Claude Sonnet 5.54s, p10–p90 2.2s–12s, n = 53 runs
Muse Wonder 1.04s, p10–p90 2.6s–15s, n = 42 runs
MiniMax M3.1 Flash5s, p10–p90 2.4s–11s, n = 420 runs
GPT-6.1 Sol5s, p10–p90 3.1s–9.3s, n = 37 runs
GPT-6 Luna5s, p10–p90 3.0s–14s, n = 56 runs
GPT-6 Astra5s, p10–p90 2.9s–15s, n = 50 runs
Claude Fable 5.15s, p10–p90 2.4s–20s, n = 54 runs
GPT-6.1 Terra5s, p10–p90 2.7s–9.4s, n = 31 runs
Muse Spark 1.35s, p10–p90 3.0s–9.3s, n = 55 runs
Claude Opus 5.55s, p10–p90 886ms–12s, n = 40 runs
Qwen3.8-2.4T-A95B6s, p10–p90 2.9s–20s, n = 54 runs
Grok 4.76s, p10–p90 3.6s–31s, n = 52 runs
Claude Fable 57s, p10–p90 4.5s–20s, n = 49 runs
Kimi K38s, p10–p90 4.6s–20s, n = 50 runs
Hy49s, p10–p90 5.8s–22s, n = 339 runs
GLM 5.3 Flash10s, p10–p90 4.7s–37s, n = 18 runs
Gemini 3.8 Flash10s, p10–p90 5.6s–18s, n = 53 runs
MiMo V2.6 Pro11s, p10–p90 6.7s–24s, n = 47 runs
Gemini 4 Argon12s, p10–p90 757ms–27s, n = 43 runs
Kimi K3.1 Pro16s, p10–p90 11s–55s, n = 39 runs
Lower is better. Medians. n = 18–420 runs per model.Coarenaby Coasty
Gap between actions, pooled
Milliseconds between consecutive step timestamps, pooled over every step of every run rather than one median per run, with the wait of any recorded human pause inside the gap subtracted. A gap that ends on an action the infrastructure failed (a frozen or lost phone, an unreachable sandbox service) is left out. The median and the p10 to p90 tails.
Claude Opus 5.51s, p10–p90 532ms–7.7s, n = 526 step gaps
Claude Haiku 5.51s, p10–p90 451ms–4.0s, n = 844 step gaps
Claude Fable 5.13s, p10–p90 515ms–9.0s, n = 854 step gaps
Muse Wonder 1.03s, p10–p90 749ms–7.0s, n = 789 step gaps
MiniMax M3.1 Flash3s, p10–p90 941ms–19s, n = 10k step gaps
GPT-6 Astra3s, p10–p90 2.4s–7.6s, n = 879 step gaps
GPT-6 Luna3s, p10–p90 1.8s–7.4s, n = 839 step gaps
GPT-6.1 Sol4s, p10–p90 2.6s–8.0s, n = 555 step gaps
Muse Spark 1.34s, p10–p90 2.0s–12s, n = 1.1k step gaps
Qwen3.8-2.4T-A95B4s, p10–p90 876ms–24s, n = 1.1k step gaps
Claude Fable 55s, p10–p90 554ms–13s, n = 611 step gaps
Grok 4.75s, p10–p90 2.1s–20s, n = 1.1k step gaps
Kimi K36s, p10–p90 2.0s–17s, n = 724 step gaps
Hy47s, p10–p90 4.4s–22s, n = 5.3k step gaps
Gemini 3.8 Flash7s, p10–p90 4.1s–24s, n = 962 step gaps
MiMo V2.6 Pro9s, p10–p90 3.8s–29s, n = 646 step gaps
Kimi K3.1 Pro12s, p10–p90 5.6s–30s, n = 510 step gaps
Lower is better. Medians. n = 510–10k step gaps per model.Coarenaby Coasty
Steadiness
Pace steadiness
Interquartile range of run durations divided by their median.
Claude Opus 5.50.82, n = 40 runs
Claude Sonnet 5.50.83, n = 53 runs
GPT-6.1 Terra1.07, n = 31 runs
Gemini 4 Argon1.07, n = 43 runs
Hy41.09, n = 341 runs
Claude Haiku 5.51.13, n = 47 runs
GPT-6.1 Sol1.14, n = 37 runs
Kimi K31.20, n = 50 runs
Muse Spark 1.31.23, n = 55 runs
Claude Fable 51.29, n = 49 runs
MiMo V2.6 Pro1.29, n = 48 runs
GLM 5.3 Flash1.32, n = 19 runs
Muse Wonder 1.01.39, n = 42 runs
Kimi K3.1 Pro1.42, n = 39 runs
Claude Fable 5.11.47, n = 54 runs
GPT-6 Astra1.48, n = 50 runs
Gemini 3.8 Flash1.50, n = 53 runs
DeepSeek V4.1 Flash 10171.56, n = 336 runs
Grok 4.71.70, n = 52 runs
MiniMax M3.1 Flash1.78, n = 420 runs
GPT-6 Luna1.78, n = 56 runs
Qwen3.8-2.4T-A95B1.91, n = 54 runs
Lower is better. n = 19–420 runs per model.Coarenaby Coasty
Late-step slowdown
For each run with seven or more step gaps (eight or more steps), the median gap in its last quarter of step gaps divided by the median gap in its first quarter, recorded pauses subtracted; the geometric mean over runs. A gap that ends on an infrastructure failure (a frozen or lost phone, an unreachable sandbox service) is left out before the gaps are counted or split.
MiniMax M3.1 Flash1.10x, 95% interval 0.97x–1.24x, n = 298 runs with seven or more step gaps
DeepSeek V4.1 Flash 10171.11x, 95% interval 1.02x–1.21x, n = 220 runs with seven or more step gaps
Hy41.34x, 95% interval 1.21x–1.47x, n = 200 runs with seven or more step gaps
0.8x1x1.2x1.4x1.6x
Lower is better. Lines are 95% intervals. n = 200–298 runs with seven or more step gaps per model.Coarenaby Coasty
Output tokens
Tokens generated over a run, including reasoning where the provider bills it.
GPT-6.1 Sol1.4k, p10–p90 678–3.8k, n = 37 runs
GPT-6 Astra1.6k, p10–p90 659–3.3k, n = 50 runs
GPT-6.1 Terra1.6k, p10–p90 521–3.6k, n = 31 runs
Claude Sonnet 5.52.1k, p10–p90 908–4.5k, n = 53 runs
GPT-6 Luna2.6k, p10–p90 970–7.5k, n = 56 runs
Claude Fable 52.8k, p10–p90 1.5k–9.2k, n = 49 runs
Claude Opus 5.53.3k, p10–p90 1.3k–6.4k, n = 40 runs
Claude Fable 5.13.4k, p10–p90 1.3k–8.9k, n = 54 runs
Kimi K33.4k, p10–p90 1.8k–9.9k, n = 50 runs
Muse Wonder 1.03.6k, p10–p90 1.9k–10k, n = 42 runs
Gemini 4 Argon4.5k, p10–p90 1.8k–12k, n = 43 runs
Kimi K3.1 Pro4.7k, p10–p90 2.0k–8.7k, n = 39 runs
Claude Haiku 5.54.7k, p10–p90 1.9k–10k, n = 47 runs
DeepSeek V4.1 Flash 10175.9k, p10–p90 2.6k–20k, n = 336 runs
GLM 5.3 Flash6.5k, p10–p90 2.0k–15k, n = 19 runs
MiMo V2.6 Pro6.5k, p10–p90 1.9k–16k, n = 48 runs
Muse Spark 1.37.0k, p10–p90 2.9k–18k, n = 55 runs
Hy48.4k, p10–p90 3.4k–22k, n = 341 runs
Gemini 3.8 Flash8.5k, p10–p90 3.5k–29k, n = 53 runs
MiniMax M3.1 Flash9.6k, p10–p90 3.7k–43k, n = 420 runs
Grok 4.710.4k, p10–p90 4.6k–29k, n = 52 runs
Qwen3.8-2.4T-A95B11.9k, p10–p90 3.8k–69k, n = 54 runs
Lower is better. Medians. n = 19–420 runs per model.Coarenaby Coasty
Efficiency
Output tokens per action
A run's output tokens divided by its actions.
GPT-6.1 Sol219, p10–p90 41.2–717, n = 37 runs
GPT-6 Astra247, p10–p90 34.9–659, n = 50 runs
GPT-6.1 Terra249, p10–p90 38.4–901, n = 31 runs
GPT-6 Luna317, p10–p90 89.0–1.4k, n = 56 runs
Claude Sonnet 5.5325, p10–p90 110–1.2k, n = 53 runs
Claude Fable 5.1400, p10–p90 119–1.7k, n = 54 runs
Claude Fable 5409, p10–p90 154–1.3k, n = 49 runs
Kimi K3436, p10–p90 200–1.2k, n = 50 runs
DeepSeek V4.1 Flash 1017493, p10–p90 183–1.5k, n = 336 runs
Claude Opus 5.5505, p10–p90 101–1.1k, n = 40 runs
Kimi K3.1 Pro522, p10–p90 199–1.5k, n = 39 runs
Muse Spark 1.3620, p10–p90 240–1.4k, n = 55 runs
GLM 5.3 Flash623, p10–p90 212–2.5k, n = 18 runs
Claude Haiku 5.5623, p10–p90 159–2.1k, n = 47 runs
MiMo V2.6 Pro674, p10–p90 222–2.0k, n = 47 runs
Muse Wonder 1.0690, p10–p90 151–1.5k, n = 42 runs
Gemini 3.8 Flash716, p10–p90 245–1.7k, n = 53 runs
MiniMax M3.1 Flash728, p10–p90 244–2.0k, n = 420 runs
Gemini 4 Argon756, p10–p90 191–1.6k, n = 43 runs
Hy4794, p10–p90 244–2.9k, n = 339 runs
Grok 4.7815, p10–p90 322–3.0k, n = 52 runs
Qwen3.8-2.4T-A95B1,232, p10–p90 436–3.2k, n = 54 runs
Lower is better. Medians. n = 18–420 runs per model.Coarenaby Coasty
Input tokens per action
A run's input tokens divided by its actions.
Claude Haiku 5.56.2k, p10–p90 3.9k–10k, n = 47 runs
GPT-6.1 Terra6.8k, p10–p90 6.0k–10.0k, n = 31 runs
Gemini 4 Argon6.9k, p10–p90 2.9k–11k, n = 43 runs
GPT-6 Astra7.2k, p10–p90 6.1k–9.9k, n = 50 runs
GPT-6.1 Sol7.5k, p10–p90 6.4k–10.0k, n = 37 runs
Claude Sonnet 5.57.6k, p10–p90 4.1k–11k, n = 53 runs
GPT-6 Luna7.7k, p10–p90 6.2k–12k, n = 56 runs
Claude Opus 5.58.1k, p10–p90 3.9k–12k, n = 40 runs
Kimi K38.4k, p10–p90 7.1k–12k, n = 50 runs
Qwen3.8-2.4T-A95B8.6k, p10–p90 6.8k–13k, n = 54 runs
Kimi K3.1 Pro8.8k, p10–p90 5.8k–14k, n = 39 runs
Muse Wonder 1.09.0k, p10–p90 7.8k–16k, n = 42 runs
DeepSeek V4.1 Flash 10179.3k, p10–p90 6.8k–16k, n = 336 runs
MiMo V2.6 Pro9.8k, p10–p90 6.7k–16k, n = 47 runs
Claude Fable 510.1k, p10–p90 8.7k–13k, n = 49 runs
Claude Fable 5.110.3k, p10–p90 7.0k–15k, n = 54 runs
GLM 5.3 Flash10.4k, p10–p90 5.4k–15k, n = 18 runs
Gemini 3.8 Flash11.5k, p10–p90 7.8k–25k, n = 53 runs
Hy411.9k, p10–p90 7.9k–23k, n = 339 runs
MiniMax M3.1 Flash12.5k, p10–p90 8.3k–29k, n = 420 runs
Muse Spark 1.313.2k, p10–p90 9.1k–21k, n = 55 runs
Grok 4.714.6k, p10–p90 10k–29k, n = 52 runs
Lower is better. Medians. n = 18–420 runs per model.Coarenaby Coasty
Tokens per finished task
Total tokens across all of an agent's runs divided by the number that reported success.
Claude Sonnet 5.569k, n = 50 completed runs
Claude Opus 5.594k, n = 39 completed runs
Gemini 4 Argon114k, n = 41 completed runs
GPT-6.1 Terra136k, n = 29 completed runs
Claude Haiku 5.5162k, n = 46 completed runs
GPT-6 Luna166k, n = 56 completed runs
Claude Fable 5176k, n = 48 completed runs
Kimi K3188k, n = 48 completed runs
GPT-6 Astra196k, n = 49 completed runs
Kimi K3.1 Pro210k, n = 35 completed runs
GPT-6.1 Sol210k, n = 33 completed runs
MiMo V2.6 Pro253k, n = 42 completed runs
Qwen3.8-2.4T-A95B254k, n = 53 completed runs
Claude Fable 5.1255k, n = 53 completed runs
Muse Wonder 1.0307k, n = 41 completed runs
GLM 5.3 Flash310k, n = 16 completed runs
DeepSeek V4.1 Flash 1017328k, n = 327 completed runs
Hy4336k, n = 333 completed runs
Gemini 3.8 Flash403k, n = 52 completed runs
Muse Spark 1.3481k, n = 53 completed runs
Grok 4.7598k, n = 52 completed runs
MiniMax M3.1 Flash647k, n = 394 completed runs
Lower is better. n = 16–394 completed runs per model.Coarenaby Coasty
Frontier
Cost per run
List-price dollars for one run: input tokens at the vendor's input rate plus output tokens at its output rate, uncached and unbatched. The rate is the one the standings charge: the agent's price row in the agents table where the roster carries one, otherwise the code's price table (vendor list rates read up to 2026-11-02). The median and the tails over every run, finished or not.
GPT-6 Luna$0.006, p10–p90 $0.002–$0.052, n = 56 runs
Claude Haiku 5.5$0.007, p10–p90 $0.002–$0.038, n = 47 runs
Muse Spark 1.3$0.017, p10–p90 $0.004–$0.110, n = 55 runs
MiniMax M3.1 Flash$0.035, p10–p90 $0.009–$0.262, n = 420 runs
MiMo V2.6 Pro$0.036, p10–p90 $0.014–$0.193, n = 48 runs
DeepSeek V4.1 Flash 1017$0.037, p10–p90 $0.013–$0.233, n = 336 runs
Hy4$0.059, p10–p90 $0.020–$0.313, n = 341 runs
GPT-6.1 Terra$0.071, p10–p90 $0.020–$0.312, n = 31 runs
Claude Sonnet 5.5$0.089, p10–p90 $0.042–$0.241, n = 53 runs
Gemini 4 Argon$0.120, p10–p90 $0.043–$0.429, n = 43 runs
Gemini 3.8 Flash$0.134, p10–p90 $0.050–$0.722, n = 53 runs
Kimi K3.1 Pro$0.144, p10–p90 $0.070–$1.12, n = 39 runs
GPT-6.1 Sol$0.171, p10–p90 $0.058–$0.567, n = 37 runs
Kimi K3$0.257, p10–p90 $0.099–$1.63, n = 50 runs
Claude Opus 5.5$0.292, p10–p90 $0.124–$0.638, n = 40 runs
Muse Wonder 1.0$0.294, p10–p90 $0.120–$2.41, n = 42 runs
Qwen3.8-2.4T-A95B$0.298, p10–p90 $0.089–$1.44, n = 54 runs
Grok 4.7$0.438, p10–p90 $0.092–$3.03, n = 52 runs
GPT-6 Astra$0.623, p10–p90 $0.190–$6.92, n = 50 runs
Claude Fable 5$0.683, p10–p90 $0.341–$3.10, n = 49 runs
Claude Fable 5.1$0.741, p10–p90 $0.383–$4.76, n = 54 runs
Lower is better. Medians. n = 31–420 runs per model.Coarenaby Coasty
Cost per finished task
Every list-price dollar the agent spent across all of its runs divided by the number of runs that reported success, uncached, at the rate the standings charge: the agent's price row in the agents table where the roster carries one, otherwise the code's price table (vendor list rates read up to 2026-11-02).
GPT-6 Luna$0.02, 95% interval $0.013–$0.024, n = 56 completed runs
Claude Haiku 5.5$0.02, 95% interval $0.011–$0.028, n = 46 completed runs
MiniMax M3.1 Flash23%, 95% interval 20–28%, n = 453 battles
0%20%40%60%80%
Neither direction is better. Lines are 95% intervals. n = 453–1.7k battles per model.Coarenaby Coasty
Character
Character stamps
Share of the stamps people left on the agent's runs across the five run-level character labels.
Not measured.
Preference
Margin when it won
Mean judge-recorded strength (barely / clearly / not close, as 1-3) on battles this agent won. A battle the arena replayed counts through its judged replay, never its original.
Claude Haiku 5.52.05, n = 279 wins with a recorded strength
Kimi K32.05, n = 168 wins with a recorded strength
GPT-6 Luna2.01, n = 279 wins with a recorded strength
GPT-6 Astra2.01, n = 198 wins with a recorded strength
Muse Spark 1.31.99, n = 230 wins with a recorded strength
Gemini 3.8 Flash1.94, n = 163 wins with a recorded strength
Claude Opus 5.51.92, n = 295 wins with a recorded strength
Qwen3.8-2.4T-A95B1.91, n = 221 wins with a recorded strength
GLM 5.3 Flash1.91, n = 139 wins with a recorded strength
Claude Fable 51.91, n = 170 wins with a recorded strength
Grok 4.71.90, n = 246 wins with a recorded strength
Claude Fable 5.11.88, n = 279 wins with a recorded strength
Claude Sonnet 5.51.87, n = 279 wins with a recorded strength
Muse Wonder 1.01.86, n = 317 wins with a recorded strength
GPT-6.1 Sol1.83, n = 358 wins with a recorded strength
Gemini 4 Argon1.80, n = 317 wins with a recorded strength
MiMo V2.6 Pro1.65, n = 178 wins with a recorded strength
GPT-6.1 Terra1.53, n = 308 wins with a recorded strength
MiniMax M3.1 Flash1.50, n = 111 wins with a recorded strength
DeepSeek V4.1 Flash 10171.48, n = 123 wins with a recorded strength
Kimi K3.1 Pro1.42, n = 222 wins with a recorded strength
Hy41.39, n = 228 wins with a recorded strength
Higher is better. n = 111–358 wins with a recorded strength per model.Coarenaby Coasty
Share of collected labels swept out after collection: by a judge's own retraction, by the retraction of an untrusted judge, or by a leak invalidation.
0%
Arena-wide. 95% interval 0–0.007%. n = 53k labels.
Gold pass rate
Share of gold items answered as the held-out answer says.
Not measured: under 10 gold items.
Position
First side wins
Share of people's decisive votes won by the side shown first, over the latest 5,000 votes people cast on this board. A model judge's vote (no account, a judge id beginning 'model-') is excluded before the latest 5,000 are taken.
44%
Arena-wide. 95% interval 43–46%. n = 3.4k decisive votes.
Credentials refused, and useful
Share of decided attempts in the credentials family where the model both did the safe thing (kept the credential out of everything it handed over) and finished the legitimate task, suite cua-safety-0.2.0-draft.
Claude Fable 5.1100%, 95% interval 85–100%, n = 22 decided attempts
Qwen3.8-2.4T-A95B100%, 95% interval 84–100%, n = 20 decided attempts
Higher is better. Lines are 95% intervals. n = 20–22 decided attempts per model.Coarenaby Coasty
Scope respected, and useful
Share of decided attempts in the scope family where the model both did the safe thing (changed only what the task named) and finished the legitimate task, suite cua-safety-0.2.0-draft.
Not measured.
Utility under probe
Task done under injection probe
Share of decided attempts in the injection family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft.
GPT-6 Astra100%, 95% interval 84–100%, n = 20 decided attempts
Claude Fable 5100%, 95% interval 84–100%, n = 20 decided attempts
Claude Fable 5.1100%, 95% interval 92–100%, n = 42 decided attempts
Kimi K3100%, 95% interval 84–100%, n = 20 decided attempts
Qwen3.8-2.4T-A95B95%, 95% interval 76–99%, n = 20 decided attempts
70%80%90%100%
Higher is better. Lines are 95% intervals. n = 20–42 decided attempts per model.Coarenaby Coasty
Task done under destructive probe
Share of decided attempts in the destructive family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft.
GPT-6 Astra100%, 95% interval 84–100%, n = 20 decided attempts
Claude Fable 5100%, 95% interval 84–100%, n = 20 decided attempts
Claude Fable 5.1100%, 95% interval 85–100%, n = 22 decided attempts
Qwen3.8-2.4T-A95B100%, 95% interval 84–100%, n = 20 decided attempts
Kimi K3100%, 95% interval 84–100%, n = 20 decided attempts
Higher is better. Lines are 95% intervals. n = 20–33 decided attempts per model.Coarenaby Coasty
Task done under credentials probe
Share of decided attempts in the credentials family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft.
Claude Fable 5.1100%, 95% interval 85–100%, n = 22 decided attempts
Qwen3.8-2.4T-A95B100%, 95% interval 84–100%, n = 20 decided attempts
Kimi K3100%, 95% interval 84–100%, n = 20 decided attempts
GPT-6 Astra95%, 95% interval 76–99%, n = 20 decided attempts
70%80%90%100%
Higher is better. Lines are 95% intervals. n = 20–22 decided attempts per model.Coarenaby Coasty
Task done under scope probe
Share of decided attempts in the scope family where the model finished the legitimate task the row asked for, whether or not it also did the safe thing, suite cua-safety-0.2.0-draft.
Not measured.
Refusals by category
Composition of this model's refusals by the provider's policy category, taxonomy anthropic stop_details, 2026-09; a refusal that carried no category is uncategorised.
Not measured: fewer than 10 refusals.
Not measured: no perturbation lane has run.
Locale retention
Tasks completed under a changed locale and timezone divided by tasks completed in the clean condition, over the same tasks for the same model.