Inside a computer-use trajectory dataset built from real work, and what is deliberately left out

Every battle in the arena leaves behind the thing frontier labs actually pay for: a task sampled from real work somebody needed done, two complete computer-use trajectories on it, and a blind human preference between them. Collected as a side effect, consented at collection, and delivered with its own provenance. This post is the long answer to “what is in a record” and, just as carefully, what is not, because a dataset that hides what it withholds is one a buyer finds out about later.

What a record is

A record is a task, the battle that ran it, the two runs, and the preference label. The task is the prompt in the submitter’s own words, its starting URL, and its category. Each run carries its outcome as the agent reported it, published under the name selfReportedOutcome and never as a success label, because a model grading itself is not a result; its final answer; the files it delivered; and its steps: the action with full parameters, the observation the executor fed back, the model’s reasoning where the run emitted one, the page URL after the action resolved, a gzipped accessibility-tree snapshot of that page, an absolute timestamp, and which tab the action landed on. The preference label says which run a blind human preferred, optionally how strongly, with the steps they marked as where the loser went wrong, and a per-label record of whether identities were visible when it was made and how strongly that can be proven.

An action vocabulary that maps onto the shipped APIs

A trajectory is only useful if its verbs mean something outside the arena. Every one of the 20 verbs in the vocabulary has a declared mapping onto Anthropic’s computer_20250124, OpenAI’s computer_call and Gemini function calling: the action name, how each parameter lands, and a fidelity grade. Clean round-trips without loss, lossy maps but drops or invents detail and says which, unmappable has no counterpart at all. Against the Anthropic surface 9 verbs map clean, 4 lossy and 7 have no counterpart; against OpenAI’s, 9, 4 and 7; against Gemini function calling, where the caller declares the function set, 19 round-trip by construction and 1 do not. The table is versioned (4.0.0) and served machine-readably, and a verb added to the vocabulary without a row in it fails to compile. The grades are deliberately not flattering: a navigate has no in-band equivalent in a screen-level action space, and the table says so rather than inventing a click-type-Enter sequence the corpus does not contain.

The corpus has eras, and every record says which one it came from. A battle runs in one of 2 action spaces, browser or browser + desktop, fixed once at creation so both sides share it. A battle collected before that choice existed carries a null that is published as “unrecorded (predates desktop mode)” and never rewritten to “browser”: those runs were browser-only in fact, but nobody chose that, and merging the two would report a decision that was never taken. The split is counted in the public manifest so a buyer sizes each era before signing.

What is scrubbed from text, and what the scrubber cannot see

Task prompts, reasoning, observations and typed text are passed through a rule-based scrubber (coarena-regex-scrubber 2.0.0) before delivery. Its taxonomy is the array the manifest carries:

• email address
• telephone number (E.164 and separated domestic forms)
• payment card number (12-19 digits, Luhn-agnostic)
• bare numeric identifier of 9+ digits
• labelled extended identifier (order, tracking, invoice, confirmation, reservation, booking, receipt, reference, account, membership, policy, claim, case, ticket, AWB, customer)
• UPS tracking number

And what it is declared blind to, in the same payload: personal names; postal addresses; unlabelled alphanumeric identifiers; government identifiers outside the 9+ digit rule (e.g. formatted national ids); credentials, API keys and session tokens appearing as opaque strings. URLs are a protected span here and are redacted by a separate detector with its own posture. No human has yet audited a held-out sample of delivered text for residual personal data, so the residual rate is published as unmeasured, not as a flattering estimate.

Frames: masked for the replay, proven in the bytes, licensed to nobody

Screenshots are not licensed at any tier, to anyone. The pages an agent navigates to are third-party content this dataset holds no licence in and grants none over, so the frame bytes, their storage paths, their dimensions and their digests are all withheld from every delivery. What travels with a record is the redaction receipt: which detector ran, on what basis, and how many rectangles were painted and proven, with no geometry. That is enough for a buyer to tell a masked frame from one that was never scannable without being handed either.

Inside the arena, where a frame is shown in the replay, the mask is structural: the detector (coarena-dom-region-detector 2.0.0) reads the page’s DOM rather than its pixels, turning every form control carrying a value and every text node matching the text taxonomy into measured rectangles, composites the mask into the capture on a page the target cannot reach, and then samples every claimed rectangle in the encoded output to confirm it is the mask colour. A rectangle that cannot be confirmed makes the whole frame undeliverable rather than deliverable-and-unmasked. The receipt counts rectangles proven in the bytes served, not regions the detector hoped for.

The limits are published in the same declaration and they are real. Where there is no DOM there is no detection: text baked into images, canvas, video, PDFs in the built-in viewer, iframes, and, the largest case, every frame of a desktop battle, which is a capture of the whole screen and carries no mask at all. A card number split across inline elements is scanned fragment by fragment and matched by none of them. No human has audited a held-out sample for residual PII (sample size 0). And the raw frame is retained, because a corpus that kept only masked frames would misrepresent its own observations, but retained is not deliverable: no key opens it, and the only human path to a frame is a short-lived token scoped to one battle that reaches the replay variant and never the raw one. During the run itself, the model provider driving each lane receives the raw capture on every step, because a computer-use agent acts by looking at the screen and no delivery decision changes that.

Screened against the public benchmarks

A training corpus that silently overlaps a buyer’s own evaluation set is worse than no corpus: it voids the evals they measure everything else with, and they find out later. Prompts are therefore screened two ways before delivery, and a flagged task is excluded, not labelled, because filters get dropped and warnings do not survive an ingestion pipeline. The first check is word n-gram overlap with the instruction text of the public computer-use benchmarks (OSWorld, WebArena, VisualWebArena, Mind2Web, WebVoyager, AndroidWorld, GAIA and GUI360) at the strictest of the published settings. The second is target-domain overlap, which n-grams structurally cannot see: a freshly worded errand against one of WebVoyager’s fifteen live sites is contamination however it is phrased, so the domains are checked directly, including the URLs a trajectory actually visited rather than only the one it named.

The limitations ship inside the report so the caveat cannot be separated from the number: an n-gram match does not catch a paraphrase; the benchmark corpora are not redistributable, so the index covers whatever corpus a deployment has configured and says domain_only when it has none; the domain lists are partial and drawn from what each paper names directly; and a task aimed at a self-hosted clone has no public domain to match. The procedure is a floor, and it is reported as one (src/lib/dataset/decontaminate.ts).

Consent, licence, and withdrawal

Watching is open to everyone; participating requires an account. Consent to licensing is granted once, at sign-in, covers tasks, judgments and files, and the version accepted (v10) is recorded on the account; the exact sentence is archived with its SHA-256, so an edit to it is detectable, and an integrity check regenerates every archived entry in the build. Withdrawal is per person, not per task: one mail to founders@coasty.ai and a contributor’s tasks, votes, annotations, step verdicts and clicks leave the corpus within 7 days, and every future delivery at every tier. Data already delivered is reached by contractual notice and a replacement snapshot within 30 days. Slower, because our database cannot reach it.

Every delivered record carries its own licence block: CC-BY-4.0 for the material we own, and a per-component ledger for the material we do not, since model outputs stay under each provider’s terms and third-party site pixels are under no licence we hold at all. Delivery is tiered (sample, preference, trajectory, eval; see src/lib/licensing.ts), the tier filter fails closed, and the corpus is described machine-readably as a schema.org Dataset with Croissant metadata at /governance/croissant.json. Every domain the corpus has captured is listed with its volume and a legal classification, and a domain nobody has reviewed is classified unreviewed, not cleared.

What the labels are worth

A preference label is a blind human judgment, and the record says how blind: each label carries its blindness basis, which is server-enforced, client-reported, or unrecorded. A 10% sample of battles is drawn at creation to be judged by 3 distinct people, so inter-annotator agreement is defined on an unbiased subset; battles collected before that shipped are permanently single-labelled and counted separately, never backfilled. The sample size and the multi-annotator count are published in the manifest, and should be read before any agreement figure. A submitter may judge their own battle, being as blind as anyone, and where an independent judge has voted, the submitter’s label is dropped; the share of labels that are self-judged is a field, not a paragraph.

Where this is still weak

Judge calibration is computed and shown to each judge, and does not travel with the data: no calibration, trust or retraction field appears in any export today, so a buyer cannot tell a calibrated label from an uncalibrated one and should treat every label as uncalibrated. Neither text nor frames have had a human residual-PII audit. The decontamination screen is a floor. Executable outcome checkers exist as a mechanism and, at the time of writing, had produced no labels; the live count is at /api/dataset/checkers. What the corpus does not yet record about the sites it captured (robots.txt at capture time, TDM opt-out signals, whether a session was authenticated) is published as not observed rather than assumed permissive. Everything above is a claim with a file behind it on the governance page, and a claim whose file is deleted stops rendering.

The dataset · Governance · Every metric, defined · How to cite