Tests are what make any of this scoreable. Amazon discarded every repository with no unit or integration tests, calling them essential to validate migration correctness in a robust way [8], and that is the right instinct: preserving original functionality [10] is a claim you can only settle by executing something, and the framework shipped with the Java set validates Maven upgrades against exactly those suites [5].
The migration task as Amazon defines it includes test framework optimization alongside runtime upgrade, deprecated API replacement and syntax modernization [10], and that inclusion creates a wrinkle worth naming before anyone reports a number: an agent permitted to modernise a project's test framework is editing the instrument it is graded by. The harness has to decide which test artifacts are frozen, and a pass rate means different things depending on that answer.
Then the denominators, which are uneven. Add the three Poly-MigrationBench ecosystems together and you get 257 repositories, from 100 .NET Framework, 74 Node.js and 83 Python [6][1]. Divide 5,102 by 257 and the Java collection is roughly twenty times the size of everything else combined [2]. That reflects where the curation effort went, not a defect in the collection, and it changes how a cross-language table should be read: a Java result on 5,102 repositories is a measurement, while a Python result on 83 is closer to a pilot that happens to be expressible as a percentage.
The 300-repository subset is where most academic work will land, since it exists for groups without the compute to run the full set, and Amazon describes it as representative [4]. That is a testable claim, and whether subset results track full-set results is what would make two labs' numbers comparable at all.
One further property of the collection deserves naming. The Java pipeline kept repositories under MIT or Apache 2.0 licences with at least three GitHub stars [7], and Poly-MigrationBench used a similar curation process [11]. That curation approach makes sense for a public benchmark. It also roughly describes the code large models were trained on. A pass rate on permissively licensed, starred GitHub projects carries a contamination question the dataset cannot answer about itself, and whoever reports the score should be the one to address it.
What none of this measures is the price of the attempt. A pass rate over 5,102 repositories [4] says nothing about tokens spent per repository, how many retries the agent was allowed, or the hours an engineer spends reviewing a diff that compiles and passes. Those are the figures that decide whether a migration agent is worth deploying, and they sit outside the data. A leaderboard on public Maven projects and a win on an internal monolith with a hand-built dependency tree are separate measurements; the first one is what just became cheap for anyone to reproduce.
