A reconstructed postmortem on dev.to has a merge bot posting eval=pass from a markdown summary, and its proposed replacement scores git diff only, with a contract test that keys on four directory names.
Reality
- Evidence47
- Adoption
- Insufficient
- Hype gap+28
- Incentives30
- Confidence55
A dev.to procedure keeps invertible pairs and normalizers in an in-repo registry, runs them as one isolated required check, and scans the same diff for assertions that got weaker while the suite stayed green.
Reality
- Evidence48
- Adoption
- Insufficient
- Hype gap+10
- Incentives20
- Confidence55
A dev.to proposal sorts a repo's tests into human oracles, seeded property checks and quarantined flakes, then requires all three classes to agree before a merge. The classifier it ships checks paths and expiry dates.
Reality
- Evidence55
- Adoption
- Insufficient
- Hype gap+35
- Incentives
- Insufficient
- Confidence45
A dev.to proposal freezes each forbidden input as a hashed fixture next to the exception class or HTTP status it must still return, and keeps that file on a path the agent's own patch is not allowed to edit.
Reality
- Evidence35
- Adoption
- Insufficient
- Hype gap+10
- Incentives20
- Confidence50
The author of a now-public merge-gate core recounted his own suite: 22 tests in the evasion file, of which 17 are attacks and 5 exist to stop the gate being red all the time.
Reality
- Evidence44
- Adoption12
- Hype gap−14
- Incentives55
- Confidence42