StarCoder2

RepoExec grades a passing solution on whether it called the helper in its own prompt

Across 18 models the best pass@1 on the benchmark's 355 Python problems is 42.57%, and the second score the paper adds counts how much of the repository's own code those passing solutions quietly rewrote.