build1 publisher
16 KB of per-row scales recover 2.2 of the 2.5 points int4 costs an embedding table
The tok.weight lookup table holds 71.7 percent of this small transformer's parameters, and sweeping bit width without sweeping scale granularity puts the accuracy cliff at four bits when it falls between four and three.
Publishers:dev.to
Reality
- Evidence42
- Adoption
- Insufficient
- Hype gap+12
- Incentives22
- Confidence46