The machine that finds the gaps also fills them
Every red cell on the map is a fact a model was missing. The same harness that measures the gap can derive the shortest fact that closes it — and prove the fact works before keeping it. Pointed at the docs for seven fast-moving platforms, it has produced 2,746 measured facts, packaged as seven installable, standards-compliant Agent Skills.
From a map to a library
A benchmark tells you a model gets D1 schema creation wrong. The interesting question is: what's the single sentence that fixes it? FlareBench answers that mechanically. Point it at a documentation page, and it walks out the other side with measured, current facts — each one stamped with how many models it flipped from wrong to right.
Doc page → facts
Split a page into micro-tasks, ask a naked from-memory model to implement each, judge it against the current doc, inject the dense fact, re-test. Keep it only if it flips ≥2 of 3 cost-tier models. That's the lift.
Against live docs
Lift means "models didn't know it" — not "it's correct". So every fact is re-checked against the live docs. This caught a model retired the day before scoring a perfect lift five times.
Collapse the noise
A regex can't tell that "get a stub by name" written eight ways is one skill. Agents can. They collapse semantic duplicates and drop facts that leaked from the wrong product.
The trap we walked into (and the law that fixes it)
The pipeline judges a fact against the documentation page. So when the page is stale, it manufactures a confident, high-scoring, wrong skill. The worst offenders are exactly what you'd expect: model ids, version numbers, dates — anything that rots monthly.
Seven libraries so far
The factory started on Cloudflare — 480 doc pages in, ~800 curated facts out across 20 products plus a cross-cutting core. Then the same machinery, with nothing new but a ~4-line descriptor and one enumeration function per site, walked six more platforms: five LLM APIs and two dev frameworks. The total is 2,746 measured, current facts, each loaded only when you're working in its platform.
| Skill | Pages | Facts | Sourced | Doc source · what it catches |
|---|---|---|---|---|
| cloudflare-current | 480 | 784 | 97% | developers.cloudflare.com · D1 exec() newline, current getByName |
| ai-sdk-current | 319 | 844 | 99% | ai-sdk.dev · the unified Output.* API |
| gemini-current | 134 | 225 | 99% | ai.google.dev · response_format + @google/genai |
| openai-current | 106 | 241 | 99% | developers.openai.com · the Responses API |
| anthropic-current | 64 | 155 | 92% | platform.claude.com · messages + tool use |
| tanstack-current | 406 | 452 | 77% | tanstack.com · Query v4→v5, Table v7→v8 |
| mcp-current | 34 | 45 | 100% | modelcontextprotocol.io · build a server |
| 7 libraries | 1,543 | 2,746 | 95% | 5 LLM APIs + 2 dev frameworks |
Inspected, not trusted
Audit agents read each library against live docs and find the staleness trap by hand. On Gemini, that caught one agent wrongly deleting twelve real Interactions-API facts — reading the output, not the aggregate, is what resolved it.
Where models lag
The audit catches most traps where the platform churns hardest — Cloudflare, Gemini, TanStack. It finds almost none where docs are current and models well-trained: Anthropic, OpenAI, MCP all audited near-clean.
Not an LLM-API thing
Two of the seven are dev frameworks, not model APIs — so this isn't a trick that only works on AI providers. It's a way to keep any fast-moving framework's facts current. Same law, no new machinery per platform.
Provable, not "over-spec vibes"
A normal rules file accretes lines nobody measured — you can't tell which earn their place. Here every fact is falsifiable on two axes: it traces to a documentation page (provenance) and it carries a measured lift (it flipped real models). The earned-place test stops being a judgement call and becomes mechanical.
0 invented
Every derived fact carries a source doc URL; 2,599 of 2,746 shipped facts (95%) trace back to their exact page — 77-100% per library. Nothing is asserted from a model's memory.
Every line flipped ≥2 models
A fact that changed no model's behaviour is never emitted. The lift on each line is its proof of work.
~225 bytes a fact
Pure load-bearing fact — no boilerplate slab a model could write itself. All seven libraries together are ~640 KB, and a skill only loads the platform you're working in.
Why this matters
A model's training data is always older than the platform's current API. The usual fix is "tell the agent to read the docs" — slow, and it still trusts its memory for the things it doesn't think to check. A measured skill library is the other half: the specific, current facts a model reliably gets wrong, handed over up front, with the volatile bits pointed at a live source so they can't quietly rot. The benchmark finds the gaps; the factory fills them; the law keeps it honest. It's open source on GitHub.