A map, not a leaderboard

Why a diagnostic map

A leaderboard tells you which model is "best". It doesn't tell you that a frontier model will confidently guess on ambiguous data, that an open-weight model can be talked into following an instruction hidden in a document, or that a cheap self-hostable model handles judgement as well as the priciest one. Those are the things that decide whether you can hand a task to an agent or have to sit over its shoulder. So the product here is the diagnosis; the ranking is a byproduct.

Why Cloudflare

Cloudflare gives us ground truth most benchmarks can't: a real deploy and a real HTTP call. The platform's quirks — bindings versus API keys, nodejs_compat, D1 limits, current static-assets versus deprecated Workers Sites — cleanly separate strong models from weak ones. And because the API shapes are current, the benchmark stays meaningful as training data ages: a model that learned last year's Cloudflare gets caught.

What we look for

Failures, mostly — they're more informative than wins. Especially the differentiating ones (where models genuinely diverge) and the nuanced ones that aren't a clean pass or fail. We also test how models cope with how real people actually prompt: not tidy specs, but one-liners, unsure beginners, and spoken rambles. Benchmarks that assume an expert user miss the most common way agents get steered wrong.

A loop with our own rules

An accumulated set of coding rules is a crystallised archive of past failures — each rule exists because something once went wrong. Running FlareBench tells us which still bite (keep the rule) and which current models now handle (delete it). The benchmark feeds the guidance, and the guidance feeds the benchmark.

Inspiration

FlareBench takes its shape from DeepSWE (Datacurve): contamination-free hand-written tasks, behavioural verification, natural prompts, no model sabotage, and efficiency measured alongside correctness. Built and run on Cloudflare Workers by Jezweb — it's open source on GitHub.