Ground truth is a real deploy
Most coding benchmarks run unit tests in a sandbox. Here the model's work is deployed to a real Cloudflare account and used — over HTTP, in a browser, or by checking the facts in the file it produced.
One run, end to end
The model writes its own Worker and its own wrangler.jsonc — wiring the bindings is part of what's tested. The harness owns deploy and teardown, and overrides only the worker name for isolation. The model has no Cloudflare token in its shell, so it can't deploy itself or touch anything else.
- Provision real bindings (KV / D1 / R2) for the task.
- The agent builds in a loop — write files, run a shell — inside an isolated Cloudflare container. Its code never touches the host.
- Deploy what it wrote, with a route-stripped config so a model can't hijack a live zone.
- Poll the live URL until the worker answers (cold starts are real).
- Grade behaviourally — hit the API, drive the page, or read the artifact.
- Tear down the worker and the bindings.
Three kinds of grading — all outcome-focused
Build a Worker/API
Deploy, then assert on real HTTP responses — status, JSON shape, the value that changed.
Build a site/feature
Deploy, then a headless browser checks the elements exist, the interaction works, and the console is clean.
Produce a doc from data
The facts are checked in code. Only the subjective layer — is it clear? — goes to an AI judge.
Deterministic facts, judge only for nuance
Numbers, config and behaviour are graded by code — never by a model. An AI judge is reserved for the one thing code can't read: genuine nuance, like whether an answer flagged a conflict or whether a steer was actually explained. And the judge is itself validated against good/bad reference cases, the same way every verifier is.
Verifiers are tested before models run
A regex that mistakes "no Postgres connection strings needed" (dismissing it) for "set up Postgres" (doing it) would punish the right answer. So every judgement task ships hand-written good and bad sample outputs plus the verdict each should reach, and a validator asserts the verifier agrees — before a single model is spent against it. This caught two false-positives that looked exactly like model failures.
The context tripod
Hold the model fixed and vary one leg at a time, so each result isolates one cause.
What it knows
Base model, plus optional hand-written skills or live docs. We measure how much closing the gap helps — and what the minimum useful skill is.
What it can do
Files and a shell, identical for every model, in an isolated container.
How it's asked
A crisp spec, a one-liner, a nervous beginner, or a spoken ramble. Most real prompts aren't tidy — grading adapts to the register.