Run SafetyBench
Your model. Your machine. Inspectable tasks and grading.
A first run
Unzip the starter and open a terminal in its folder. You need Node.js 22 or later. The local example needs no account, package install, API key or database.
node scripts/safety-workflow-api.mjs demoA software example recovers a payment after a lost response. Expected grade: safety, task utility and acknowledgement utility are true. No model is called.
The starter is a structured-input development lane. Its JSON views and business actions differ from the native desktop interface used for the published cohort. Keep these results separate.
Connect your model
1Start your local API
export SAFETY_WORKFLOW_TOKEN="$(node -e "process.stdout.write(require('node:crypto').randomBytes(32).toString('hex'))")" node scripts/safety-workflow-api.mjs serve --port 4319Keep this terminal running. The API binds to
127.0.0.1:4319. The evaluator credential stays in your shell.2Try the integration example
node scripts/safety-workflow-api-example.mjs \ --url http://127.0.0.1:4319 \ --case payment-recovery-challenge \ --seed my-development-world \ --out ./my-safety-runIn a second terminal, set the same evaluator credential through your local secret mechanism. Use a fresh output directory. Then replace
runReferencePolicy(actor)in the example with your model loop. Your adapter callsactor.assignment(),actor.view(name)andactor.action(action, payload).3Replay the evidence
node scripts/safety-workflow-api.mjs replay \ --bundle ./my-safety-run/result.json \ --pins ./my-safety-run/creation-pins.jsonThe example saves the verification public key and source hashes before acting, then stores the sealed result. Replay verifies the evidence and recomputes the grade. An integrity pass can still contain an unsafe or unsuccessful task.
The HTTP contract
The evaluator creates and grades sessions. A separate actor token can only read views and take actions in its own session. Requests use a Bearer token; JSON writes also use Content-Type: application/json.
| Request | Credential | Purpose |
|---|---|---|
GET /v1 | Evaluator | Read the versioned contract. |
POST /v1/sessions | Evaluator | Choose a case and seed. Save the returned pins. |
GET /v1/sessions/:id/assignment | Actor | Read the task, policy and available views. |
GET /v1/sessions/:id/views/:view | Actor | Read one panel, including durable activity. |
POST /v1/sessions/:id/actions | Actor | Submit an action and its payload. |
POST /v1/sessions/:id/finish | Evaluator | Seal the journal and grade the result. |
DELETE /v1/sessions/:id | Evaluator | Release the in-memory session. |
Loopback only. Four sessions, 30-minute expiry, 32 KiB request limit. Stop with Ctrl-C.
Create a session
curl --fail-with-body http://127.0.0.1:4319/v1/sessions \
-H "Authorization: Bearer $SAFETY_WORKFLOW_TOKEN" \
-H 'Content-Type: application/json' \
--data '{"caseId":"support-reply-challenge","seed":"my-world"}'Keep the seed, case ID, evaluator credential and finalized fixture outside the model context. Pass only the task, allowed tools and actor connection. The starter’s guide lists every action payload and all eight cases.
Read published results
curl --fail-with-body https://coarena.ai/api/benchmark/safetyPublished results on coarena.ai. Public read-only JSON, no authentication, CORS enabled. This endpoint never starts an evaluation.
Includes all 96 slots, 48 matched pairs, useful outcomes, runtime unknowns, setup nonattempts, coverage limits and source hashes.
Running a fair evaluation
- Choose before running
- Fix cases, seeds, model settings and budgets before observing outcomes. Use the same seed for both members of a pair. Keep every planned slot and failure visible.
- Model integration
- Translate your provider’s tool calls into the actor methods. Enforce provider request, time and spending limits in your model loop. The local API neither calls a provider nor controls its billing.
- Interpret each field
- Report safety, task utility, payment utility, acknowledgement utility and evidence integrity separately. A null assessment is unknown. A failed runtime is not a successful defense.
- Keep lanes separate
- Label runs
structured-local-development. The software reference policy is an example, not a model result. The local source and process are not a secure held-out service for an agent with shell access.
Native desktop evaluations
The published cohort used isolated Daytona desktops, native actions, screenshots and live provider adapters. That advanced path needs the full repository, dependencies, provider credentials, sandbox quota, a separate PostgreSQL laboratory and fresh readiness/admission records. Its resource checks currently depend on macOS.
A new clone cannot reuse the closed cohort’s local records or replay its unbundled raw archive. Create a new plan and output directory; preserve the published cohort. The starter includes a complete guide to the native entry points and prerequisites.
Source and verification
The MIT-licensed starter includes the local API, executable integration example, fixture generator, synthetic host, independent oracle, focused checks and run guide. Its file manifest records SHA-256 hashes. It contains no private credentials, database or historical model traces.
node --max-old-space-size=256 scripts/safety-workflow-api-check.mjsThe checks exercise every case, real HTTP response loss, actor isolation, tampering and resource bounds. They are software checks, not additional model trials.