Mission

A fixed task list is a target. Agents overfit it, scores crowd the ceiling, and the tasks leak into the next training corpus, so the number stops moving while the models keep changing. An arena has no fixed task list to leak.

1. The task list is not ours and is never finished.

   Every published computer-use benchmark ages from the day it ships: its tasks enter training data, its ceiling is reached, and its ranking freezes while the field moves. A corpus that is written by its users cannot be studied in advance because it does not exist in advance.

   Enforced: Real people post real computer tasks; two agents race the identical task in parallel, in a browser or on a desktop; the outcome is judged blind. There is no seed set that a model could have been trained on. The Phone board's first battles ran on tasks from a library we wrote, judged by a model.

   Cited: src/app/api/battles/route.ts + scripts/campaign-phone (the Phone board's first battles) + tasks/phone

2. A task that already exists in a public benchmark is not a new measurement.

   The whole value of live submission is that the task is unseen. A submitted task that duplicates a published benchmark item reintroduces exactly the leak the arena exists to avoid.

   Enforced: Before a task is delivered in a dataset it is checked against the public computer-use benchmarks: by the site it targets, and by its wording where a benchmark's instruction text is loaded. A match is excluded from the delivery rather than labelled. This is a floor: a reworded task can still pass.

   Cited: src/lib/dataset/decontaminate.ts + src/lib/dataset/deliverability.ts

3. Every agent competes on the arena's own hands, not its vendor's.

   A vendor computer-use tool and a shared action space are different experiments. Comparing a model driven by its own first-party harness against one driven by ours measures the harnesses, and the model comparison is unrecoverable from the result.

   Enforced: One action vocabulary per battle, one sandbox, one step budget and one wall clock for both lanes. A battle runs in the browser, on a desktop, or on an Android phone, and both sides inherit the same choice; a model trained on its own tool vocabulary is translated into the shared one by its lane. Each verb carries a declared mapping onto the shipped lab computer-use APIs, or a declared reason it has none. 101 published measures are read off that shared record.

   Wired to a constant in src/lib/agents/tools.ts + /api/dataset/action-map + src/lib/metrics/catalog.ts

Published de-anonymization rates for frontier models run above 95% from a bag of words, and moving a top-five rank costs fewer than a thousand votes. An arena with no blindness machinery is not neutral by default; it is poisoned by default.

1. Blindness is enforced where the data is written, not where it is displayed.

   A rule applied in a shared render path can be routed around by any future caller, and a self-driving lane once did exactly that, writing its own summary straight into judge-visible fields.

   Enforced: Vendor names are scrubbed at the database write, which every lane and every future lane passes through by construction. Two scrubbers: a blunt one for text this codebase authors, a narrow one for text the model authors, because blanking a legitimate 'I'll search Google Scholar' would corrupt the trajectory to hide a name that reveals nothing.

   Wired to a constant in src/lib/blind.ts + src/lib/db.ts (scrub at write)

2. Blindness is a property of the (battle, viewer) pair, never of the battle.

   Under redundant judging the first vote resolves a battle while more judges are still expected. A battle-level rule would show later judges the model names before they voted, and their labels would measure brand preference instead of agreement.

   Enforced: One function decides identity visibility for everyone, and it fails closed: every branch that does not affirmatively establish a reason to reveal returns false. A judge sees the names when their own label is in and cannot be revised, or when the battle takes no more votes.

   Cited: src/lib/judging.ts (identityVisibility), proved by npm run verify:blindness

3. A battle whose visible text names its own model was never blind, and its votes do not count.

   Asking a model who it is succeeds more than 90% of the time. A leak turns a preference label into a brand label.

   Enforced: Both final answers and every step's reasoning are scanned when the battle becomes votable, before the first judge can arrive, not before a delivery. The scan fails closed: an unscannable battle is recorded as leaked. A leaked battle rates nobody.

   Cited: src/lib/stepjudging.ts (scanBattleForIdentityLeak) + src/lib/rating.ts (identity_leaked)

Uniform matchmaking sounds fair and is not. It gives a model rostered this morning the same share of the next battle as one with three hundred battles behind it, so the newcomer stays unrated for as long as uniform sampling takes to reach the floor. An equal chance of being picked is not an equal chance of being ranked, and ranking is what an entrant asks for.

1. The next battle goes where the least is known, and that advantage disappears as it is used.

   A rating's interval scales as 1/sqrt(n), so an agent that is behind on battles is behind on being rankable at all. Weighting by remaining uncertainty equalizes: under a 1/sqrt(n) allocation every agent's count converges to the same number from any starting position, so the tilt is self-cancelling rather than a permanent handicap.

   Enforced: Draw weight is 1/sqrt(measured battles + 4). Opponents are additionally weighted by how informative the pairing is (4p(1-p), the Bradley-Terry information of one comparison), and by how under-covered that specific pair is (prior 2), because Bradley-Terry reads the graph's edges and an unplayed edge is an unidentified comparison.

   Wired to a constant in src/lib/matchmaking.ts, proved by npm run verify:fairness

2. No agent and no pairing can ever be sampled to zero.

   A weighting scheme with no floor can drive a probability to zero through a chain of individually reasonable multiplications, and an agent at probability zero is off the roster without anyone deciding to remove it. A comparison graph whose top and bottom never meet has ratings identified only through intermediaries.

   Enforced: A fixed 20% of every draw is flat uniform, and a maximally lopsided pairing keeps 35% of its weight. Both are floors rather than tunables.

   Wired to a constant in src/lib/matchmaking.ts (UNIFORM_FLOOR, INFO_FLOOR)

3. Nothing a person supplies can influence who they are matched against. Our experiment batches and the battles our accounts pick the models for rate nobody.

   If a submitter can steer the pairing, the ranking measures who submitted rather than which model is better. That has to be impossible by construction, not merely discouraged.

   Enforced: In the arena the pair is drawn from the roster, the public battle counts and the public ratings, and the selection function takes no caller input: not the submitter, the account, the prompt, the task, the time of day or any client value. Our own experiment batches do name their pairs, and every battle in one is excluded from the ranking outright, because we also choose the repeats and which rows to re-run. So is every battle whose models one of our own accounts picked by hand. The Phone board's first campaign is neither: we named its pairs, on tasks from a library we wrote, and its battles are rated on the Phone board.

   Cited: src/lib/agents/registry.ts (pickBattlePair takes no arguments) + src/lib/rating.ts (the operator_experiment and picked_matchup exclusions) + scripts/campaign-phone/src/planner.ts (the campaign's named pairs)

4. Which side is A is drawn separately, from its own random source.

   The two agents come from different distributions, so 'the first one drawn goes on the left' would put the more under-measured agent in slot A systematically. Judges carry a measurable position preference, so that correlation would push position bias straight into the ratings of exactly the newest entrants the scheme exists to protect.

   Enforced: Three independent CSPRNG draws per battle: anchor, opponent, slot order. The realized A-position win rate is then published with its interval, because a correct randomizer with a biased judge population still produces biased labels: randomization is necessary and not sufficient.

   Wired to a constant in src/lib/matchmaking.ts (selectMatch) + src/lib/elo.ts (positionBias)

Running Elo is a convenience, not an estimator. It depends on the order votes arrived in, it needs a seed for every new agent (asserting it is exactly average before anyone has looked), and it has no memory of which vote moved it, so a retraction can never be undone. We publish a fit instead.

1. The published rating is refitted from the whole corpus every time, from scratch.

   An accumulated rating carries every mistake it ever made. A refit does not: a retraction, a newly untrusted judge, or a corrected eligibility rule takes effect on the next refresh, where the ticker would carry the error forever.

   Enforced: Maximum-likelihood Bradley-Terry over every admissible comparison, with cluster-robust standard errors. Order-independent, and it needs no seed, so a model's place comes from its record rather than from a number the roster handed it. Below 30 rated comparisons the fit is withheld and the model is listed as not rated. The live ticker is still shown, under its own name, seeded at 1000, labelled as the ticker.

   Wired to a constant in src/lib/rating.ts (publishedRatings) + src/lib/elo.ts (fitBradleyTerry)

2. Labels from one person are not counted as independent facts.

   A corpus where one person cast most of the labels holds far fewer independent observations than votes. Clustering on the task would not help: every battle has its own task row, so each cluster would hold a single observation.

   Enforced: Standard errors cluster on the judge. Below more judges than parameters the covariance is not identified and no interval is published, rather than a narrow one computed from a corpus that cannot support it.

   Cited: src/lib/rating.ts (ratedCorpus) + src/lib/elo.ts (sandwichIdentified)

3. A model added today is not penalised for the date it was added.

   A fixed step size says one battle is worth the same evidence against an agent measured once and one measured a thousand times. That single choice produces both complaints an arena gets: a new model reads as mediocre for its first hundred battles, and a settled rating lurches on one judge's tap.

   Enforced: The online step decays with each side's own measured battles, from 48 at zero to 12 once established. The decay is smooth because a stepped schedule puts a discontinuity in the trajectory that is indistinguishable in the record from the agent actually changing.

   Wired to a constant in src/lib/elo.ts (kFactor)

4. Not every judged battle rates an agent, and we say which ones do not and why.

   A battle where one lane never ran, or one side had more actions than the other, is a real trajectory but not a fair comparison. And an unexplained gap between battles judged and battles ranked is exactly what an arena would be suspected of fabricating.

   Enforced: Seven rules exclude a battle from the ranking: it belongs to one of our own experiment batches rather than being drawn by the arena, a lane produced no trajectory, one lane was cut short by the person who created the battle, the step budgets were unequal, a side has no run record at all, there is no admissible label left (retracted, untrusted, or cast faster than the answers could be read), or its text named a model. Every excluded label stays in the corpus and every exclusion is counted in the rating audit. Nothing is deleted.

   Cited: src/lib/rating.ts (rateBattle, RatingAudit)

5. A submitter may judge their own battle, and an independent judge outranks them.

   Refusing every self-judged battle would discard most of a young corpus and guard against nothing: identities are withheld from every viewer until a battle resolves, so a submitter's own vote is blind by construction.

   Enforced: Where any independent judge has voted, the submitter's own label is dropped from that battle's outcome. Where none has, their blind label decides it and the battle is counted as self-judged in the rating audit. The rule tightens by itself as the judge pool grows. Battles in our own experiment batches never reach it: they are refused a rating first.

   Cited: src/lib/rating.ts (THE SELF-JUDGING RULE + the operator_experiment exclusion)

“Success rate 71%” is not a result. Success according to whom, over which runs, counting a provider outage as whose fault? Most disagreements about a benchmark are really disagreements about a denominator, so the denominator ships with the number.

1. The definitions are the artifact. The numbers are a consequence.

   A benchmark whose definitions live in a blog post cannot be reproduced or argued with. Shipping the two together means a consumer never has to guess, and means a metric cannot quietly change meaning without its description changing in the same response.

   Enforced: 101 measures, each published with what is counted, the denominator it is counted over, which direction is good (or an explicit statement that neither is), and what it does not establish. Machine-readable at /api/metrics, with the catalog in the same payload as the values.

   Wired to a constant in src/lib/metrics/catalog.ts + /api/metrics + /benchmark

2. Every measure here is a proxy, and each one says so.

   There is no oracle that says a click was correct. A proxy shipped without its limits is worth less than no metric, because someone will act on it. The honest thing is to measure what the record supports and state exactly what that does and does not establish.

   Enforced: Where the record cannot support a proxy, the catalog prints the reason instead of a number. “Clicks that did nothing” is withheld on both boards: the observation after a click, a desktop click or a phone tap echoes the press (a desktop click also names the window that took the keyboard, and a click that opened a tab says so), and none says whether the press hit a control; the errors a press meets come from the machine (a timeout, an unreachable sandbox, a phone that did not respond), not from where it landed, so a rate over them would publish the arena's reliability as the model's aim. Neutral measures are marked neutral: the action mix and the click spread rank nothing, and no chart draws a direction under them.

   Wired to a constant in src/lib/metrics/catalog.ts (caveat, better: neutral, click_no_effect_rate) + src/lib/metrics/aggregate.ts (NO_PRESS_EFFECT_SIGNAL)

3. We say when a slice is too small to support the comparison being made.

   A published interval that runs past 100%, or a three-point difference quoted off forty observations, is a tell that nobody checked. Below the power threshold the honest output is a refusal, not a number.

   Enforced: Intervals are Wilson rather than normal, because at these sample sizes the normal approximation produces bounds outside [0,1]. A slice resolving a 3% difference needs ~969 independent items; below that it is labelled underpowered rather than quoted. A model with fewer than 30 rated battles is provisional: listed, not rated, and given no rank. One with fewer than 1000 is never called stable.

   Wired to a constant in src/lib/judging.ts (wilsonInterval) + src/lib/elo.ts (isUnderpowered, ratingStatus)

4. Every number travels with its sample size. A rate without its denominator is a claim, not a measurement.

   Everything needed to argue with a number is published: what it counts, over what, how many, with what interval, under which rule. A reader who cannot see that a 33% miss rate came from three clicks will compare it with one that came from four hundred, and every benchmark this arena is compared with prints its n.

   Enforced: Every value is projected before it is serialized and the projection keeps the count: the denominator of a rate, the n of a distribution, the sample behind a scalar, the total behind a profile, the battles in a matrix cell. A value withheld below its coverage floor carries the reason instead, so no surface prints a dash a reader has to guess at.

   Cited: src/lib/metrics/aggregate.ts (publicValues)

The corpus is other people's work: their tasks, their judgment, their attention. Consent is asked once in a sentence a person can actually read, it is withdrawable, and what we collect is bounded by what the measurement needs.

1. You are shown one sentence, and we keep a verbatim record of the sentence you were shown.

   A licence granted against text nobody kept is a licence nobody can audit. Rewriting an old consent sentence to match the current one would make every prior signature a claim about words those people never saw.

   Enforced: Consent is version v12, granted to Coasty Systems, Inc., and covers your tasks, your judgments, the files you attach, your agent memory and your saved sign-ins. Every earlier sentence is archived verbatim with its own hash, append-only, and is never rewritten to match the current one.

   Wired to a constant in src/lib/consent.ts (CONSENT_VERSION, CONSENT_ARCHIVE, CONSENT_TEXT_SHA256)

2. Withdrawal is self-service, and it says exactly what it removed.

   A right to withdraw with no mechanism is a right to send an email, and a receipt that overstates what was removed misleads someone exercising a privacy right.

   Enforced: Signed in, you withdraw from your account page; founders@coasty.ai also works. Every task you submitted becomes private at once and cannot be made licensable again, and your votes, annotations and step verdicts are removed from future dataset deliveries and from the public rankings and statistics. Copies already delivered cannot be recalled, and the receipt says so.

   Wired to a constant in src/app/api/me/withdraw/route.ts + src/lib/licensing.ts (withdrawContributor) + src/lib/consent.ts (CONSENT_WITHDRAWAL)

3. We publish what our redaction actually catches, including what it misses.

   “We redact PII” is the claim every corpus makes and none of them can support. A detector's recall is the only interesting number about it, and a posture published without its limits is a liability rather than a safeguard.

   Enforced: The text and imagery detectors each ship a stated posture (what is detected, on what basis, and what is out of scope) rather than an assurance. A frame that carries no mask distinguishes three different facts: the detector ran and found nothing, the detector broke, or nobody looked. Collapsing those would let a failure read as a clean page.

   Wired to a constant in src/lib/redact.ts (TEXT_PII_POSTURE, IMAGERY_PII_POSTURE) + FrameRedaction basis

4. A battle you post is private until you offer it.

   An unscoped battle list would let strangers read each other's prompts and enumerate every battle id in the corpus.

   Enforced: A battle is readable by its submitter, and by signed-in judges once the submitter offers it to the judging pool. A caller who may not read a battle gets the same answer as one who named a battle that does not exist, so an id cannot be used to probe.

   Cited: src/app/api/_lib/battle-access.ts, proved by npm run verify:battle-access

5. A judge is a person we are asking for attention, not a data source to be maximised.

   Labels are the product, so an unbounded writer could set the consensus. And a judge scored in secret has no way to improve.

   Enforced: Label writes are rate-limited per account, persistently, so a deploy does not reset the cap. Judge identity is a random cookie value. A judge can read their own calibration; a judge measured below 0.45 accuracy on enough scored calls is marked untrusted, and that mark reaches the ranking rather than sitting in a table nothing reads.

   Wired to a constant in src/app/api/_lib/label-rate.ts + src/lib/judging.ts (trustState) + src/lib/rating.ts

6. An agent driving a browser or a phone on our infrastructure does not get the run of our network.

   A computer-use agent is an arbitrary-URL fetcher with a model deciding the URLs. Without a fence, the first interesting task is one that navigates to an internal address.

   Enforced: Only http(s) is allowed, and internal and private hosts are blocked by name and by resolved address, so a public hostname that resolves inward is blocked too. The fence covers the browser; a program run through the shell tool can still open its own connections. A phone is fenced on its host instead: every connection it makes to a private, link-local or cloud metadata address, or to the host itself, is dropped there.

   Cited: src/lib/agents/egress.ts, proved by npm run verify:egress + infra/phone-host/src/egress-proxy.ts + infra/phone-host/linux/files/egress.nft.in

The company that runs this arena builds a computer-use agent of its own. That is a real conflict and disclosure does not cure it, so the mechanism has to be removal rather than a footnote.

1. Our own agent is not on the board and cannot be matched into a battle.

   An arena cannot credibly sell a ranking that its operator appears in. Disclosing the conflict and then competing anyway leaves the reader to decide how much to discount every row.

   Enforced: The roster's validation refuses the lane that drives it, so no battle can be matched to it through code or through the admin route, and the leaderboard ranks only the roster.

   Cited: src/lib/roster.ts (OPERATOR_LANES, rankedIds)

2. No entrant tests privately and publishes only the best result, and no score is retracted after the fact.

   Selective publication is the cheapest way to fake a benchmark and the hardest to detect from the outside.

   Enforced: Entrants do not choose their battles: the pairing is drawn by the arena, and every admissible battle it draws counts toward the ranking. There is no score-removal path. Our own experiment batches and the battles our own accounts pick the models for, where we choose the pairing, are excluded from the ranking outright. The Phone board's first campaign is neither: we planned its pairings, and its battles are rated on the Phone board.

   Cited: src/lib/agents/registry.ts (pickBattlePair) + src/lib/rating.ts (the operator_experiment and picked_matchup exclusions) + scripts/campaign-phone/src/runner.ts (createCampaignBattle)

The reason to trust any of the above is not that it is well written. It is that the sentences are wired to the code, and the wiring is the only part that cannot be talked around.

1. A belief whose mechanism is deleted disappears from this page.

   Prose cannot be checked, so prose alone is how a mission page rots.

   Enforced: Every thesis here carries a predicate over a constant exported by the module that enforces it. A failing predicate removes the belief and lists it as a gap instead. Visibly missing beats invisibly false.

   Wired to a constant in src/app/mission/theses.ts + npm run verify:claims

2. We mark which of our values are load-bearing in code and which are load-bearing in prose.

   Some mechanisms are code paths or server-only modules that a page which must never fail to render cannot import. Wrapping those in a predicate that returns true unconditionally would be a lie about the strength of the evidence.

   Enforced: Theses whose predicate reads a real constant are marked wired. The rest are marked cited, with the file to open and check.

   Wired to a constant in src/app/mission/theses.ts (backed vs stated)

3. We track the claims this corpus is not yet entitled to make.

   The recurring failure in this field is claims outrunning mechanisms. A report that lists its own gaps cannot be caught overstating them.

   Enforced: The quality report computes what cannot yet be claimed at the current volume: agreement statistics that are undefined, ratings that are not stable, calibration that does not travel with the data, and how much of the ranking still rests on one person's judgment. The report is internal today.

   Cited: src/lib/stepjudging.ts (notEntitledToClaim) + /api/admin/quality

Every belief above is checked by npm run verify:claims, part of the repository’s test suite, which fails when a belief here or on governance loses the implementation behind it. 16 of 30 beliefs are wired to a constant and stop rendering if it goes. The other 14 are marked cited: their mechanism is a code path the page cannot import, so they pass the check by construction and the file is the evidence.

Found a belief that reads true and is not? Email founders@coasty.ai. Related: every metric, defined, the standings and privacy.