papers / benchmark-family

Agents Can Build Benchmarks. Approvals Missed What Probing Found: 22 Cases

Ryan Armani · orcid 0009-0005-9964-6841 · 2026-10-17 · preprint

abstract

Coding agents built the 22 benchmark packages this paper studies, 1204 tasks on one shared engine; the 22 cases of the title are the packages. Probing the packages found defects that needed a fix round in every one of them (median 3, maximum 9), across 13 defect classes, and at least 13 packages paid credit to a strategy that ignored the task. Re-running the logged exploits on the artifacts replicates 45 of 50 record clauses, 31 of 34 measured on the published tarballs and 14 of 16 on the unreleased commits the record cited, and the median attack score over the 16 paired wave-1 rows falls from 0.7500 before the fixes to 0.0841 after; 5 of the 52 post-fix clauses still fail at the shipped versions and 7 more hold only under the weak lower-than-before bound. The other layers did not stay blind in both directions: the diff-reading review flagged a published hidden seed before one merge, and the generic blind battery later found released credit on payment-delegation that the probing record never logged. Under the preregistered design the best of 8 generic blind policies scored higher before the fixes in only 2 of 12 packages (p = 0.50 on 3 non-tied packages), a null that bounds what the registered design can claim; at the released versions the best blind score on any package is 0.4000. At the released versions 7 probe cells on 5 graders pay content that never did the task, and 2 of 22 graders accept forged fields. The record documents 2 of the 94 approvals as approval-to-defect cases. We release the engine, the packages, the re-review protocol, the defect record and the scripts behind every number, and we condense the classes into five checks to run before an agent-built benchmark ships. No real model was run and no leaderboard is kept; the registered real-model arm is unfinished.

benchmark: benchmark-* family. submitted to: Research Square.

citation

Armani, Ryan (2026). Agents Can Build Benchmarks. Approvals Missed What Probing Found: 22 Cases. https://doi.org/10.5281/zenodo.23455125

bibtex

@misc{benchmark-family,
  author = {Armani, Ryan},
  title = {Agents Can Build Benchmarks. Approvals Missed What Probing Found: 22 Cases},
  year = {2026},
  doi = {10.5281/zenodo.23455125},
  url = {https://ryanarmani.com/papers/benchmark-family},
}

venues, counts and the software records live on papers.