CUA KnowledgeBench 12-Hours & 1-Day
Day-long knowledge work across more than 100 applications, scored on private tasks with a wall-clock budget. A run is solved when the work product meets the task’s check; success is solved runs over runs.
- Models scored
- 10
- Runs per model, 10 tasks × 5 seeds
- 50
- Wall-clock per run
- 12 hours
- Published
- 7 Sept 2026
- 1-day horizon
- Running
Results
Steps and solve times are medians over solved runs, so they describe how a model succeeds, not how it fails. Bars are runs solved as a share of 50, on a full 100% axis.
12-hour horizon
| Model | Runs solved | Distinct tasks | Success | Solves per solved task | Median steps | Median solve time |
|---|---|---|---|---|---|---|
| Claude Fable 5.1 | 12 / 50 | 4 / 10 | 24% | 3.0 | 6,940 | 6h 22m |
| Claude Fable 5 | 11 / 50 | 4 / 10 | 22% | 2.8 | 7,180 | 6h 47m |
| GPT-6 Astra | 10 / 50 | 3 / 10 | 20% | 3.3 | 7,610 | 7h 37m |
| Claude Opus 5 | 9 / 50 | 3 / 10 | 18% | 3.0 | 7,450 | 7h 15m |
| GPT-5.6 Sol | 8 / 50 | 3 / 10 | 16% | 2.7 | 8,020 | 8h 28m |
| Claude Sonnet 5 | 6 / 50 | 2 / 10 | 12% | 3.0 | 8,340 | 7h 53m |
| GPT-5.6 Terra | 4 / 50 | 2 / 10 | 8% | 2.0 | 8,660 | 9h 23m |
| Qwen3.8 Max | 3 / 50 | 1 / 10 | 6% | 3.0 | 8,730 | 10h 40m |
| Kimi K3 | 2 / 50 | 1 / 10 | 4% | 2.0 | 8,540 | 10h 41m |
| Gemini 3.8 Flash | 2 / 50 | 1 / 10 | 4% | 2.0 | 9,150 | 9h 09m |
Not scored: Grok 4.6, GPT-5.6 Luna, Muse Spark 1.3, Inkling, GLM 5.3 Flash. Their runs stopped in the early episodes after failures across tasks.
1-day horizon
Still running: 6 tasks × 5 seeds, 24 hours of wall-clock per run, the same models. Results are added here when the run completes.
Repeatability
pass^k is the share of tasks a model solved on every one of k seeds, averaged over all k-seed subsets of the 5, so it needs each task’s solve count rather than the total. pass^1 is the success rate; the gap to pass^2 is how often a solve did not repeat; pass^5 is the share of tasks never missed. The interval beside a value is a bootstrap over tasks.
| Model | pass^1 | pass^2 | pass^5 |
|---|---|---|---|
| Claude Fable 5.1 | 24.0% | awaiting per-task counts | awaiting per-task counts |
| Claude Fable 5 | 22.0% | awaiting per-task counts | awaiting per-task counts |
| GPT-6 Astra | 20.0% | awaiting per-task counts | awaiting per-task counts |
| Claude Opus 5 | 18.0% | awaiting per-task counts | awaiting per-task counts |
| GPT-5.6 Sol | 16.0% | awaiting per-task counts | awaiting per-task counts |
| Claude Sonnet 5 | 12.0% | awaiting per-task counts | awaiting per-task counts |
| GPT-5.6 Terra | 8.0% | awaiting per-task counts | awaiting per-task counts |
| Qwen3.8 Max | 6.0% | awaiting per-task counts | awaiting per-task counts |
| Kimi K3 | 4.0% | awaiting per-task counts | awaiting per-task counts |
| Gemini 3.8 Flash | 4.0% | awaiting per-task counts | awaiting per-task counts |
pass^2 and pass^5 need each task’s solve count over the 5 seeds. The published module carries the totals only, so those columns fill in when the counts are added from the run logs; nothing is estimated from the totals. pass^1 is exact from them and equals the success rate above, and carries no interval, since the interval is over tasks.
One of the environments: Northwind WMS
Northwind WMS is one of the environments in the suite. Records about lots, counts, carriers and storage arrive across a warehouse system, an ERP, team chat and mail, and every decision has to be recorded on the work board together with the evidence that supports it. The task runs across 32 work windows; as each window advances, new records are published and the ledgers move. The 10 clips below are from saved step captures of one task in this environment; each caption says what is on screen.
- Warehouse system The work board where decisions are recorded, with inventory and transport, evidence arrivals and handover views.
- ERP Purchase orders and goods receipts, and a notification feed that publishes each window's records.
- Team chat The Northwind DC-3 workspace: channels for receiving, approvals, carriers, dispatch and warehouse operations.
- Mail An inbox where count, receipt and carrier records arrive as messages from the desks that own them.
Evidence across applications
The clip opens in the team chat's search with 47 unread messages waiting; the campaign number returns four evidence messages. Four evidence mails are opened, then the warehouse system's evidence arrivals page lists 18 records and the application each arrived in. In the ERP, a purchase-order filter and the global search return nothing for the campaign number. The notification feed is opened instead, and capacity, carrier, quality, recall and receipt records are expanded to read the shift-desk memo each one carries.
Each clip is an edited, time-condensed sequence of 40 consecutive saved screenshots from one task attempt, most held for under a second. Silent, no overlays, cropped to the application's own viewport without the browser's frame and address bar. The clips show how the environment is worked, not whether the task was solved.
Reading the results
- The top of the 12-hour table is a cluster, not an order. Claude Fable 5.1 leads at 24%, but one seed is worth 2 points, and the first 4 rows are within three solved runs of each other. With 10 tasks the rate is set by how many tasks a model can solve at all, so differences of a few runs do not order neighbours.
- Breadth separates more than repeatability. 10 of 10 models solved at least one run; 5 solved three or more distinct tasks (Claude Fable 5.1, Claude Fable 5, GPT-6 Astra, Claude Opus 5, GPT-5.6 Sol). Solves per solved task run from 2 to 3.3 of the 5 seeds, so even a task a model can do it rarely does on every seed; the gap between models is mostly in how many tasks it can do at all.
- Effort rises as success falls. Median steps on solved runs climb from 6,940 at the top of the table to 9,150 at the bottom of the solved rows, and median solve time runs from 6h 22m at the top to as much as 10h 41m lower down: weaker models that do finish take longer routes to the same product.
- What the numbers do not say. 10 tasks × 5 seeds separate the top of the table from the bottom, not neighbours from each other: a 12/50 and a 11/50 are one seed apart. The axis runs to 100%, so a bar is the share of all 50 runs a model solved, not a share of the leader.
Method
- Tasks
- 10 on the 12-hour horizon, each run on 5 seeds; the 1-day horizon adds 6 and is still running. A seed fixes the workspace state, the data in it and the timing of anything that changes during the run, so every model sees the same 50 situations.
- Environment
- Real applications, more than 100 across the benchmark: communication, documents, spreadsheets, project and customer records, research and scheduling. Information is spread across them and changes during a run; the work is to keep it coherent. Not every application appears in every task.
- Budget
- 12 hours of wall-clock per run on the 12-hour horizon, 24 hours on the 1-day horizon. A run that has not met the check when the budget ends is unsolved.
- Scoring
- Binary per run: the resulting documents, records and messages either meet the task’s check or do not. Success is runs solved over runs. Distinct tasks counts the tasks solved on at least one seed; solves per solved task is runs solved over distinct tasks, a measure of repeatability.
- Repeatability
- pass^k is the share of tasks solved on every one of k seeds, estimated without bias as the average over all k-seed subsets: for a task solved on c of 5 seeds, choose(c, k) over choose(5, k). pass^1 is the success rate. It needs each task’s solve count, which the totals cannot yield, so a model without those counts shows pass^1 alone. Its interval is a bootstrap over tasks, the one interval on this page.
- Resolution
- One seed is 2 points of success rate on the 12-hour horizon and 3.3 on the 1-day horizon, and the five seeds of one task are not independent trials, so a table separates its top from its bottom, not one row from the next. No interval is published; the reading above says in words what the counts can and cannot support.
- Steps and time
- Medians over solved runs only. Unsolved runs are excluded rather than counted at the budget, so a model with no solve shows neither.
- Stopped runs
- A model whose runs fail across tasks in the early episodes is stopped before the rest of its budget is spent, and listed as not scored rather than at 0%: its 50 runs were not completed, so it has no rate. Every scored model completed all 50 runs, unsolved ones included, so a stop enters no denominator. The same rule applies to every model; the 5 stopped on this horizon are named under the table.
- Not published
- Task prompts, seeds, traces and per-task results. The suite stays private so results measure capability rather than recall.
Evaluating a model
New models run on the same suite, seeds and budgets by arrangement. Write to founders@coasty.ai with the model and the horizon; we run the evaluation and publish the row here. The 1-day horizon is running; 3-day and 7-day horizons follow.