Golden Benchmark

Tasks

77 templates with 5 seeds each, deterministic and journal-graded against a known correct outcome. Every model runs the same tasks, seeds and grader version; 7 ran a second pass. 29 templates are at ceiling.

Base

100 × the mean over templates of each template’s mean over its seeds, so every template counts once. Its interval is a 95% bootstrap over templates: how Base would move if the task set were resampled.

Composite

Composite = clamp(Base + W_MAX·σB·clamp(k·z_arena − z_base, ±3), 0, 100); k = σL²/(σL² + SE_L²); w = W_MAX once the arena rating is Bradley-Terry (≥ 30 games), else 0 (listed, unranked)Base and the arena are standardised as z-scores in the calibration cohort. A declared 60/40 blend of two constructs (r = 0.433 between Base and the arena logit in the cohort), not an estimate of one quantity. The interval is the published 95% interval, clipped to [0, 100].

Shrinkage

k shrinks a young arena rating toward the cohort mean by its precision: GPT-6 Astra’s ±95 counts at k = 0.49, GPT-5.6 Sol’s ±25 at 0.93.

Rank

The 95% interval of the model’s rank under a joint bootstrap: templates resampled in step across models, each arena rating jittered by its standard error. Where the data cannot separate two models, the band says so.

Cohort

14 models: roster ∩ coverage = 1.0 ∩ Bradley-Terry ∩ games ≥ 300. Frozen constants: W_MAX 0.4, μB 96.857, σB 1.891, μL 0.049, σL 0.271, balance point W 0.435.

Stabilised rating

stable = 1000 + 173.72·(μL + σL·z_C), z_C = z_base + W_MAX·clamp(k·z_arena − z_base, ±3); w = 0 below the arena floor (Base-only)The Composite’s z on the arena’s Elo scale, in the Composite’s order. It is published as a second opinion: the leaderboard ranks on the live arena fit alone, and this benchmark carries no weight there.

Stability

Base is fixed, so the Composite moves only with the arena, at W_MAX × k of the arena’s own daily movement. Composite at k = 1 ignores shrinkage; Composite uses each day’s k.

Reliability

Split halves compare the seeds 2 against 3, in all 10 ways. A tier whose cross-model sd is under 1 Base point is uninformative. The Spearman-Brown full-length estimate over all templates is 0.921. A flip is an episode whose pass or fail changed on the second pass.

Release

gb-1.3.0/env-1.1.0/grader-1.3.0, generated September 7, 2026, calibrated by scripts/bench/calibration/gb-1.3.0_env-1.1.0_grader-1.3.0.json.

Top-pair order sensitivity: the Composite gap and the W_MAX at which the pair would change order.

Per ranked model: the stabilised rating with its published ±, k, arena games as of September 6, 2026, templates solved, and the severe rate (episodes at or below 0.25).

Full methodology