A lone 50.0 activation coarsens the INT8 grid 62-fold for every other value in the tensor

Run that vector through the quantizer and the loss is visible in four numbers. At a step of about 0.394, 0.3 comes back as 0.394, roughly 31 percent high, and 0.7 comes back as 0.788, about 13 percent high ^[10][20][21]. 0.1 rounds to code zero and comes back as nothing ^[9][10]. Four of the six values occupy codes between -1 and 2, and INT8 offers 256 of them ^[6][25].

The 62x figure is not a property of the format. It is 50 divided by 0.8, the largest magnitude over the second largest ^[11][22]. Compute that ratio on a real tensor and you know what per-tensor absmax costs you before you have picked a kernel. Shrijith Venkatramana's post puts single activations at 50x to 100x their neighbours ^[3].

Per-tensor scaling persists because of the hardware: one tensor, one scale, one INT8 GEMM, very little metadata ^[12]. Column-wise scaling of activations is the obvious repair, and the multiplication will not take it. In Y = XW the activation channels are the reduction dimension, so an independent scale per input channel becomes entangled with every product accumulating into an output element ^[16]. The post's worked matrix shows what not doing it costs: a fourth column of 60.0, 55.0 and 49.0 sets the scale for three columns whose values are all under 0.4 ^[17].

The measurement that makes outlier isolation attractive came from Tim Dettmers, Mike Lewis, Younes Belkada and Luke Zettlemoyer in 2022, in the LLM.int8() work on OPT models ^[13]. At the 6.7B scale they reported roughly 150,000 outlier values per sequence, held in about six feature dimensions, around 0.1 percent of the feature values, and removing those dimensions badly damaged the model ^[14]. That is about 25,000 outlier values per dimension per sequence ^[23]. Venkatramana wrote that the problem was closer to "There are a few special dimensions that repeatedly produce large numbers, and the model actually uses them." ^[19]

For that to transfer to the model in front of you, the outlier columns have to be the same columns across tokens and layers. Six fixed dimensions can be pulled out and handled at higher precision; six dimensions that wander per token cannot, and the full range penalty comes back. The small vectors in the post are supposed examples, and the outlier counts are measurements of OPT ^[27][13].

The path the post records runs INT8, then finding the outliers, then isolating or moving them, then choosing better scaling, then floating-point 8-bit formats, with contributions from Dettmers, Song Han's group at MIT and engineers from NVIDIA, Intel and Arm ^[18]. Bit width, number representation and scaling granularity are three separate decisions ^[5]. Two stacks can both claim INT8 and differ by a factor of 60 in step size on the same tensor ^[22].