build1 publisher
Sixteen times more training code buys the same gain as reading the rest of the file
A re-evaluation of checkpoints trained on 500K, 2M and 8M tokens of code moved the point where a zero-parameter count table overtakes a 2.45M-parameter transformer by 6.6x in document length, and at 4,000 tokens the table still wins.
Publishers:dev.to
Reality
- Evidence58
- Adoption
- Insufficient
- Hype gap−10
- Incentives55
- Confidence48