Build1 publisher2 min readPublished
16 KB of per-row scales recover 2.2 of the 2.5 points int4 costs an embedding table
The tok.weight lookup table holds 71.7 percent of this small transformer's parameters, and sweeping bit width without sweeping scale granularity puts the accuracy cliff at four bits when it falls between four and three.
The Engineer · Build desk
What happened
- In the small transformer under test the tok.weight embedding holds 1,024,000 parameters, 71.7 percent of the model, so most of what is called the network is a lookup table.
- The experiment swept embedding width at int8, int4, int3 and int2 while holding the transformer body at int8.
- At int4, one scale per token scored 0.347 network top-1 against 0.325 for a single tensor-wide scale, and the extra scales added 1.7 percent to the bytes.
- At int8 the two granularities were indistinguishable, both scoring 0.350.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision A width-only sweep supports a confident wrong answer: int4 reads as the break point when the break was a scaling choice, so the granularity axis has to be swept where levels are scarce.
- cost The accuracy recovery is paid entirely in bytes that ship with the weights, because nothing in the sweep was retrained and no fine-tuning compute was spent.
- constraint The fp32-matching mixture score assumes a roughly 25 MB count table sitting beside the quantised network, so the compression ratio does not describe a system that fits in a megabyte.
- capability A 1 MB budget that previously held only count statistics can now hold a network too, scoring 0.356 where the counts-only configuration reached 0.289.
Per-row scaling helps here because of what an embedding row is. tok.weight holds 1,024,000 parameters across 8,000 vocabulary rows, which makes each row 128 numbers [2]. Token embedding norms differ by orders of magnitude between frequent and rare words, so one scale for the whole tensor spends most of its levels on a handful of outlier rows and leaves the rest of the vocabulary on a coarse grid [6]. Per-token scaling gives every row its own step size for one fp16 value per row: 8,000 rows at two bytes each is 16,000 bytes [3].
The write-up puts that at under 2% of the table [5]. Which table it is measured against changes the number. Against an int8 table of 1,024,000 bytes, 16 KB is 1.6%; against the int4 table of 512,000 bytes it is 3.1% [6]. The 1.7% quoted for the int4 result is measured against the whole network, and the bytes reconcile: 512,000 for the embedding, 404,992 for the int8 body, 16,000 for the scales, 932,992 in total, which is the 0.93 MB in the results [5].
The width axis does have a cliff. In the mixture it sits between four bits and three, 0.371 then 0.276 [12]. Seven runs covered the cross product of both axes [14].
The follow-up spends the freed bytes: int4 per-row as the network, plus the largest pruned count table the remaining budget allows [23]. The winning 1 MB configuration prunes to contexts seen at least 400 times, a 24 KB table, and weights the document cache at 0.40 [20]. That pruned table is about a tenth of a percent of the roughly 25 MB full one [7]. Almost all of the non-neural contribution at that budget comes from statistics of the document being processed, and those cost nothing to store because the text is already in memory [24]. The network on its own got worse, 0.351 to 0.347, while the system improved [19].
For the 71.7% to hold in another model, the shape has to match: 8,000 rows of 128 numbers against a body of 404,992 parameters, 1,428,992 in total [1][2][1][2]. A larger body lowers the embedding's share and moves the dial with it. Out of distribution, the granularity choice buys nothing at four bits, because OOD top-1 holds at 0.185 all the way down through int4 per-tensor and only moves at int3 [21]. The research repo is not public, so the seven runs are not ones you can rerun [22].
What to watch
- Whether per-row scaling at int3 or int2 moves the four-to-three cliff, which the reported mixture numbers do not separate by granularity.
- A replication on a model with a larger body, where the embedding is well under 70 percent of parameters and the dial sits elsewhere.
- Whether the 1 MB result survives without a useful document cache, since the winning configuration weights that cache at 0.40.