build1 publisher
RepoExec grades a passing solution on whether it called the helper in its own prompt
Across 18 models the best pass@1 on the benchmark's 355 Python problems is 42.57%, and the second score the paper adds counts how much of the repository's own code those passing solutions quietly rewrote.
Publishers:dev.to
Reality
- Evidence58
- Adoption
- Insufficient
- Hype gap+10
- Incentives40
- Confidence55