build1 publisher
A 12-task suite leaves each holdout task worth 20 points of pass rate
Coding-agent pass rates get re-quoted after every prompt edit. A dev.to protocol freezes a seeded holdout split before any tuning, and at its minimum size the published number gets too coarse to show a 13-point gain.
Publishers:dev.to
Reality
- Evidence55
- Adoption
- Insufficient
- Hype gap+15
- Incentives20
- Confidence62