Mission
The only evaluation that survives contact with frontier models is one generated live by people doing real work, judged blind by humans, and published with the machinery that makes every number checkable, including the numbers that embarrass us.
Everything below is a belief paired with the mechanism that enforces it and the file you can open to check. 32 of them, across 8 sections. 18 are wired to a constant in the module that does the enforcing. Delete the mechanism and the belief disappears from this page. That is the only kind of value statement worth writing down.
A fixed task list is a target. Agents overfit it, scores crowd the ceiling, and the tasks leak into the next training corpus, so the number stops moving while the models keep changing. An arena has no fixed task list to leak.
The task list is not ours and is never finished.
Every published computer-use benchmark ages from the day it ships: its tasks enter training data, its ceiling is reached, and its ranking freezes while the field moves. A corpus that is written by its users cannot be studied in advance because it does not exist in advance.
EnforcedReal people post real browser tasks; two agents race the identical task in parallel; the outcome is judged by a human. There is no seed set that a model could have been trained on.
citesrc/app/api/battles/route.ts
A task that already exists in a public benchmark is not a new measurement.
The whole value of live submission is that the task is unseen. A submitted task that duplicates a published benchmark item reintroduces exactly the leak the arena exists to avoid.
EnforcedSubmissions are screened for overlap with the public computer-use benchmarks before they enter the dataset.
citesrc/lib/categorize.ts + the dataset pipeline
Every agent competes on the arena's own hands, not its vendor's.
A vendor computer-use tool and a shared action space are different experiments. Comparing a model driven by its own first-party harness against one driven by ours measures the harnesses, and the model comparison is unrecoverable from the result.
EnforcedOne action vocabulary, one browser, one sandbox, one step budget, one wall clock, for every lane. Each verb carries a declared mapping onto the shipped lab computer-use APIs, or a declared reason it has none. 57 published measures are read off that shared record.
wiredsrc/lib/agents/tools.ts + /api/dataset/action-map + src/lib/metrics/catalog.ts
Published de-anonymization rates for frontier models run above 95% from a bag of words, and moving a top-five rank costs fewer than a thousand votes. An arena with no blindness machinery is not neutral-by-default; it is poisoned-by-default, and a buyer is right to assume so.
Blindness is enforced where the data is written, not where it is displayed.
A guarantee applied in a shared render path is a guarantee every future caller can route around, and one did: a self-driving lane skipped the runner loop and wrote its own summary to three judge-visible fields verbatim. Applying a rule in a place a caller can bypass is not applying it.
EnforcedVendor names are scrubbed at the database WRITE, which every lane and every future lane passes through by construction. Two scrubbers: a blunt one for text this codebase authors, a narrow one for text the model authors, because blanking a legitimate 'I'll search Google Scholar' would corrupt the trajectory to hide a name that reveals nothing.
wiredsrc/lib/blind.ts + src/lib/db.ts (scrub at write)
Blindness is a property of the (battle, viewer) pair, never of the battle.
Under redundant judging the first vote resolves a battle while two more judges are still expected. A battle-level rule would hand judges two and three the model names before they labelled anything, and their labels would measure brand preference instead of agreement: the number the redundancy sample exists to produce, poisoned by the feature that produces it.
EnforcedOne function decides identity visibility for everyone, and it fails closed: every branch that does not affirmatively establish a reason to reveal returns false. A judge sees the names when their own label is in and cannot be revised, or when the battle takes no more votes.
citesrc/lib/judging.ts (identityVisibility), proved by npm run verify:blindness
A battle whose visible text names its own model was never blind, and its votes do not count.
Asking a model who it is succeeds better than 90% of the time. A leak is not a cosmetic defect; it converts a preference label into a brand label, and one leaked battle in the corpus is a reason to distrust every unleaked one.
EnforcedBoth final answers and every step's reasoning are scanned when the battle becomes votable, before the first judge can arrive, not before a delivery. The scan fails closed: an unscannable battle is recorded as leaked. A leaked battle rates nobody.
citesrc/lib/stepjudging.ts (scanBattleForIdentityLeak) + src/lib/rating.ts (identity_leaked)
One judge is noise, so a random sample of battles is judged by three.
Every agreement statistic is undefined at one annotator per item: with n = 1 there are no pairs, so percent agreement, kappa and alpha all return null however many battles are collected. Judging everything three times would cost triple the labour for a third of the corpus. A random fraction estimates the process for the whole.
EnforcedThe draw happens at CREATION, before anyone has seen the battle, and is written on the battle row. Deciding at vote time would sample the popular, recent, short battles and describe the front page instead of the corpus. Default rate 10%, floor 3 judges, and Krippendorff's alpha is the headline because the annotator count varies per item.
wiredsrc/app/api/battles/route.ts (the draw) + src/lib/judging.ts
Uniform matchmaking sounds like the fair answer and is not. It gives a model rostered this morning the same share of the next battle as one with three hundred battles behind it, so the newcomer stays unrated for as long as uniform sampling takes to drag it over the quotable floor. Equal chance of being PICKED and equal chance of being RANKED are different properties, and only the second is what an entrant is asking for.
The next battle goes where the least is known, and that advantage disappears as it is used.
A rating's interval scales as 1/sqrt(n), so an agent that is behind on battles is behind on being rankable at all. Weighting by remaining uncertainty equalizes: under a 1/sqrt(n) allocation every agent's count converges to the same number from any starting position, so the tilt is self-cancelling rather than a permanent handicap.
EnforcedDraw weight is 1/sqrt(measured battles + 4). Opponents are additionally weighted by how informative the pairing is (4p(1-p), the Bradley-Terry information of one comparison), and by how under-covered that specific pair is (prior 2), because Bradley-Terry reads the graph's edges and an unplayed edge is an unidentified comparison.
wiredsrc/lib/matchmaking.ts, proved by npm run verify:fairness
No agent and no pairing can ever be sampled to zero.
A weighting scheme with no floor can drive a probability to zero through a chain of individually reasonable multiplications, and an agent at probability zero is off the roster without anyone deciding to remove it. A comparison graph whose top and bottom never meet has ratings identified only through intermediaries.
EnforcedA fixed 20% of every draw is flat uniform, and a maximally lopsided pairing keeps 35% of its weight. Both are floors rather than tunables.
wiredsrc/lib/matchmaking.ts (UNIFORM_FLOOR, INFO_FLOOR)
Nothing a person supplies can influence who they are matched against.
The moment a submitter can steer the pairing, the ranking measures who submitted rather than which model is better. This has to be structurally impossible, not merely discouraged.
EnforcedThe pair is drawn from the roster, the public battle counts and the public ratings. Not the submitter, not the account, not the prompt, not the task, not the time of day, not any client value. The selection function takes no caller input at all. There is no per-agent constant anywhere in it, so every asymmetry it produces is a function of what has been measured.
citesrc/lib/agents/registry.ts (pickBattlePair takes no arguments)
Which side is A is drawn separately, from its own random source.
The two agents come from different distributions, so 'the first one drawn goes on the left' would put the more under-measured agent in slot A systematically. Judges carry a measurable position preference, so that correlation would push position bias straight into the ratings of exactly the newest entrants the scheme exists to protect.
EnforcedThree independent CSPRNG draws per battle: anchor, opponent, slot order. The realized A-position win rate is then published with its interval, because a correct randomizer with a biased judge population still produces biased labels: randomization is necessary and not sufficient.
wiredsrc/lib/matchmaking.ts (selectMatch) + src/lib/elo.ts (positionBias)
Running Elo is a convenience, not an estimator. It depends on the order votes arrived in, it needs a seed for every new agent (asserting it is exactly average before anyone has looked), and it has no memory of which vote moved it, so a retraction can never be undone. We publish a fit instead.
The published rating is refitted from the whole corpus every time, from scratch.
An accumulated rating carries every mistake it ever made. A refit does not: a retraction, a newly untrusted judge, or a corrected eligibility rule takes effect on the next refresh, where the ticker would carry the error forever.
EnforcedMaximum-likelihood Bradley-Terry over every admissible comparison, with cluster-robust standard errors. Order-independent, and it needs no seed: a model rostered today is placed by its record from its first battle rather than by a number the roster handed it. The live ticker is still shown, under its own name, seeded at 1000, labelled as the ticker.
wiredsrc/lib/rating.ts (publishedRatings) + src/lib/elo.ts (fitBradleyTerry)
Labels from one person are not counted as independent facts.
A corpus where one person cast most of the labels does not contain as many independent observations as it contains votes. Clustering on the task looked right and was doing nothing at all. A task row is created per battle, so every cluster held exactly one observation and the cluster-robust machinery was faithfully reproducing the naive errors it exists to correct.
EnforcedStandard errors cluster on the JUDGE. Below more judges than parameters the covariance is not identified and NO interval is published, rather than a narrow one computed from a corpus that cannot support it.
wiredsrc/lib/rating.ts (ratedCorpus) + src/lib/elo.ts (sandwichIdentified)
A model added today is not penalised for the date it was added.
A fixed step size says one battle is worth the same evidence against an agent measured once and one measured a thousand times. That single choice produces both complaints an arena gets: a new model reads as mediocre for its first hundred battles, and a settled rating lurches on one judge's tap.
EnforcedThe online step decays with each side's own measured battles, from 48 at zero to 12 once established. The decay is smooth because a stepped schedule puts a discontinuity in the trajectory that is indistinguishable in the record from the agent actually changing.
wiredsrc/lib/elo.ts (kFactor)
Not every judged battle rates an agent, and we publish which ones did not and why.
A battle where one lane never ran, or where one side was granted more actions than the other, is a real trajectory and not a fair comparison. Rating it transfers points from whoever had a bad afternoon to whoever did not. But an unexplained gap between 'battles judged' and 'battles ranked' is the single most reasonable thing to accuse an arena of fabricating.
EnforcedFive rules exclude a battle from the ranking: a lane that produced no trajectory, unequal step budgets, no admissible label, a retracted or untrusted judge, and text that named a model. Every excluded label stays in the corpus and every exclusion is counted and published. Nothing is deleted.
citesrc/lib/rating.ts (rateBattle, RatingAudit)
A submitter may judge their own battle, and an independent judge outranks them.
The first version of this rule refused to rate any self-judged battle. In the live corpus that discarded 39 of 44 judged battles and left three of thirteen models with a score: a ranking built on two comparisons instead of forty-four. Stricter is not the same as better. And the threat it guarded against does not exist here: identities are withheld from EVERY viewer until a battle resolves, so a submitter's own vote is blind by construction.
EnforcedWhere any independent judge has voted, the submitter's own label is dropped from that battle's outcome. Where none has, their blind label decides it and the battle is counted as self-judged in the published audit. The rule tightens by itself as the judge pool grows, with no threshold to tune.
citesrc/lib/rating.ts (THE SELF-JUDGING RULE)
“Success rate 71%” is not a result. Success according to whom, over which runs, counting a provider outage as whose fault? Most disagreements about a benchmark are really disagreements about a denominator, so the denominator ships with the number.
The definitions are the artifact. The numbers are a consequence.
A benchmark whose definitions live in a blog post cannot be reproduced or argued with. Shipping the two together means a consumer never has to guess, and means a metric cannot quietly change meaning without its description changing in the same response.
Enforced57 measures, each published with what is counted, the denominator it is counted over, which direction is good (or an explicit statement that neither is), and what it does not establish. Machine-readable at /api/metrics, with the catalog in the same payload as the values.
wiredsrc/lib/metrics/catalog.ts + /api/metrics + /benchmark
Every measure here is a proxy, and each one says so.
There is no oracle that says a click was correct. A proxy shipped without its limits is worth less than no metric, because someone will act on it. The honest thing is to measure what the record supports and state exactly what that does and does not establish.
Enforced“Clicks that did nothing” means a click whose own observation reported an error or an unchanged page: a real signal, and not the same thing as a miss. Neutral measures are marked neutral: the action mix and the click spread rank nothing, and no chart draws a direction under them.
wiredsrc/lib/metrics/catalog.ts (caveat, better: neutral)
We say when a slice is too small to support the comparison being made.
A published interval that runs past 100%, or a three-point difference quoted off forty observations, is a tell that nobody checked. Below the power threshold the honest output is a refusal, not a number.
EnforcedIntervals are Wilson rather than normal, because at these sample sizes the normal approximation produces bounds outside [0,1]. A slice resolving a 3% difference needs ~969 independent items; below that it is labelled underpowered rather than quoted. A rating with fewer than 30 battles is provisional and one with fewer than 1000 is never called stable.
wiredsrc/lib/judging.ts (wilsonInterval) + src/lib/elo.ts (isUnderpowered, ratingStatus)
How much data exists is a commercial fact, not a methodological one, and it is the one thing we do not publish.
Everything needed to argue with a number is published: what it counts, over what, with what interval, under which rule. How many battles are behind it is not part of that argument, and it is the corpus's value.
EnforcedEvery value is projected before it is serialized: no denominator, no run total, no per-cell tally leaves the server. The projection is exhaustive over the value shapes, so a count cannot survive by being spelled differently in one of them; the win matrix publishes a share rather than a record for exactly that reason.
citesrc/lib/metrics/aggregate.ts (withoutSampleSizes)
The corpus is other people's work: their tasks, their judgment, their attention. Consent is asked once in a sentence a person can actually read, it is withdrawable, and what we collect is bounded by what the measurement needs.
You are shown one sentence, and we keep a verbatim record of the sentence you were shown.
A licence granted against text nobody kept is a licence nobody can audit. Rewriting an old consent sentence to match the current one would make every prior signature a claim about words those people never saw.
EnforcedConsent is version v9, scope tasks+judgments, granted to Coasty Systems, Inc.. Every earlier sentence is archived verbatim with its own hash, append-only, and is never rewritten to match the current one.
wiredsrc/lib/consent.ts (CONSENT_VERSION, CONSENT_ARCHIVE, CONSENT_TEXT_SHA256)
Withdrawal covers everything you contributed, with a deadline attached.
A right to withdraw with no scope and no clock is a right to send an email. Naming both is what makes it a commitment rather than a gesture.
EnforcedOne channel (founders@coasty.ai) covering 5 kinds of contribution: tasks you submitted, votes you cast, annotations you placed, step verdicts you gave, clicks you contributed. Removed from the corpus within 7 days, and buyers of already-delivered data notified with a replacement snapshot within 30.
wiredsrc/lib/consent.ts (CONSENT_WITHDRAWAL) + src/lib/licensing.ts (withdrawContributor)
We publish what our redaction actually catches, including what it misses.
“We redact PII” is the claim every corpus makes and none of them can support. A detector's recall is the only interesting number about it, and a posture published without its limits is a liability rather than a safeguard.
EnforcedThe text and imagery detectors each ship a stated posture (what is detected, on what basis, and what is out of scope) rather than an assurance. A frame that carries no mask distinguishes three different facts: the detector ran and found nothing, the detector broke, or nobody looked. Collapsing those would let a failure read as a clean page.
wiredsrc/lib/redact.ts (TEXT_PII_POSTURE, IMAGERY_PII_POSTURE) + FrameRedaction basis
A battle you post is private until you offer it.
An unscoped battle list was two problems in one coat: strangers read each other's task prompts, and the endpoint was a free enumeration handle over the whole corpus: hand it to a loop and you have every battle id there is.
EnforcedA battle is readable by its submitter, and by any signed-in judge only once the submitter has offered it to the judging pool. A caller who may not read a battle is told the same thing as a caller who named one that does not exist. A 403 would confirm it exists and turn any id back into an oracle.
citesrc/app/api/_lib/battle-access.ts, proved by npm run verify:battle-access
A judge is a person we are asking for attention, not a data source to be maximised.
Labels are the product, so an unbounded writer sets the consensus, which is a corpus-poisoning surface before it is a DoS surface. And a judge scored in secret has no way to improve.
EnforcedLabel writes are rate-limited per account, persistently, so a deploy does not reset the cap. Judge identity is a random cookie value and never PII. A judge can read their own calibration; a judge measured below 0.45 accuracy on enough scored calls is marked untrusted, and that mark reaches the RANKING rather than sitting in a table nothing reads.
wiredsrc/app/api/_lib/label-rate.ts + src/lib/judging.ts (trustState) + src/lib/rating.ts
An agent driving a browser on our infrastructure does not get the run of our network.
A computer-use agent is an arbitrary-URL fetcher with a model deciding the URLs. Without a fence, the first interesting task is one that navigates to an internal address.
EnforcedOnly http(s) is allowed, and internal and private hosts are blocked by name and by resolved address, so a public hostname that resolves inward is blocked too, which a name-only check would miss.
citesrc/lib/agents/egress.ts, proved by npm run verify:egress
The company that runs this arena builds a computer-use agent of its own. That is a real conflict and disclosure does not cure it, so the mechanism has to be removal rather than a footnote.
Our own agent is not on the board and cannot be matched into a battle.
An arena cannot credibly sell a ranking that its operator appears in. Disclosing the conflict and then competing anyway leaves the reader to decide how much to discount every row.
EnforcedIt is in neither the agent roster nor the provider map, so no new battle can be matched to it, and the leaderboard ranks the roster rather than every rating row that survives in the table.
citesrc/lib/db.ts (DEFAULT_AGENTS) + src/lib/agents/registry.ts (PROVIDER_BY_AGENT)
Its losing history is still in the database, and we say so rather than deleting it.
Silently erasing an operator's own past results to make a sentence true is the worse failure. A reader who finds the rows later has been given a reason to distrust everything else on the page.
EnforcedBattles created before that exclusion still exist and its rating row survives. Those rows stay in the corpus and are disclosed on /governance. They are simply not ranked.
citesrc/app/governance/claims.ts (affiliation.historical-rows)
No participant tests privately and publishes only the best result, and no score is retracted after the fact.
Selective publication is the cheapest way to fake a benchmark and the hardest to detect from the outside. The only defence a reader can verify is that everything which runs is published.
EnforcedEvery battle that runs is published. There is no private-variant path and no score-removal path.
citesrc/app/api/battles
The reason to trust any of the above is not that it is well written. It is that the sentences are wired to the code, and the wiring is the only part that cannot be talked around.
A belief whose mechanism is deleted disappears from this page.
Prose cannot be checked, so prose alone is how a mission page rots. On the governance page three lines were being kept by nothing at all: a vote rate limit that did not exist, an anomaly detector that did not exist, and a rating threshold that appeared in no other file. All three read perfectly well.
EnforcedEvery thesis here carries a predicate over a constant exported by the module that enforces it. A failing predicate removes the belief and lists it as a gap instead. Visibly missing beats invisibly false.
wiredsrc/app/mission/theses.ts + npm run verify:claims
We mark which of our values are load-bearing in code and which are load-bearing in prose.
Some mechanisms are code paths or server-only modules that a page which must never fail to render cannot import. Wrapping those in a predicate that returns true unconditionally would be a lie about the strength of the evidence.
EnforcedTheses whose predicate reads a real constant are marked proven. The rest cite a file and are marked stated, so a reader knows which ones to go and check.
wiredsrc/app/mission/theses.ts (backed vs stated)
We publish the claims this corpus is NOT yet entitled to make.
The recurring failure in this field is claims outrunning mechanisms. A manifest that enumerates its own gaps cannot be caught overstating them, and the gaps are the most useful thing on the page for anyone deciding whether to rely on the data.
EnforcedThe quality report computes what cannot be claimed at the current volume and ships that list beside the numbers: agreement statistics that are undefined, ratings that are not stable, calibration that does not travel with the data, and the share of the ranking that still rests on one person's judgment.
citesrc/lib/stepjudging.ts (notEntitledToClaim) + /api/admin/quality