Build1 publisher3 min readPublished
Quantizing DistilBERT to int8 traded 8% of its latency for three quarters of its size
The ONNX export alone ran 1.58x faster than eager PyTorch with bit-identical accuracy. The int8 pass after it cost 8% in latency and bought a 64 MB download, small enough to serve as a static page with no backend.
The Engineer · Build desk
What happened
- The classifier is a DistilBERT fine-tune over Banking77's 77 banking intents, and its 256 MB checkpoint at about 9 ms per query on CPU was the reason it could not ship as a static page with no backend.
- Exporting to ONNX with Optimum left predictions identical and ran 1.58x faster than eager PyTorch, 5.66 ms against 8.95 ms p50, measured single-query on CPU with tokenization inside the timer.
- Dynamic quantization to QUInt8 weights cut the file from 256.3 MB to 64.3 MB, a 4x reduction the author says is the reliable part of the technique.
- The int8 graph ran 8% slower than ONNX fp32 on the same machine, 6.14 ms against 5.66 ms, an overhead the author attributes to quantize and dequantize steps on Apple Silicon.
- The live demo downloads the 64 MB model once and classifies text entirely in the browser through Transformers.js.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- decision Anyone reusing these p50 figures on a server CPU is reading numbers taken on Apple Silicon, and the author says he did not measure the AVX-512 VNNI case he expects to behave differently, so the hardware target decides whether quantization is a latency lever at all.
- cost Serving cost moves off the operator and onto the visitor, who pays one 64 MB fetch and the CPU cycles for every classification in the tab.
- constraint The table only supports single-query comparisons; the author notes batching would have flattered the larger fp32 model, so throughput-shaped deployments cannot reuse these latencies.
- capability Teams whose blocker is download size get the part of this result that holds elsewhere, because the 4x cut comes from the weight format and does not depend on the instruction set the way the milliseconds do.
The phrase "made it fast" covers two separate levers, and by the author's own account they are not the same lever [18].
The export is the free one. `ORTModelForSequenceClassification.from_pretrained(SOURCE, export=True)` writes the same weights into a different graph format [5]. Predictions were identical and accuracy bit-identical [6]. The 1.58x came out of the runtime: operator fusion, a more efficient memory layout, no Python-level overhead per operator [7].
`quantize_dynamic` with `weight_type=QuantType.QUInt8` took the file from 256.3 MB to 64.3 MB [8][10]. Accuracy moved from 91.88% to 91.75% and macro-F1 from 91.94% to 91.70%, costs of 0.13 and 0.24 points [10][3]. Latency went the wrong way, 6.14 ms against 5.66 ms p50 [10][11].
The explanation offered is the hardware the benchmark ran on: Apple Silicon already handles fp32 matmuls efficiently, so the quantize and dequantize steps wrapped around each int8 operation ate most of the savings from cheaper integer math [13]. The author expects int8 to win on latency on x86 with AVX-512 VNNI, instructions built for int8 dot products, and is explicit about the standing of that expectation: "I haven't measured that; it's a gap in this benchmark, not a claim," he wrote [14].
For these p50s to transfer to a server you would need the same class of CPU, the same single-query pattern, and tokenization inside the timer. The post reports all three as conditions of the measurement, and adds that batching "would have flattered the larger model and defeated the point of the comparison" [12].
Even carrying the 8% penalty, the int8 graph beats where the project started. 8.95 ms to 6.14 ms is 1.46x [1]. Size is what decided the deployment: 192 MB came off the file, 74.9% of it [2].
On raw accuracy, DistilBERT's 3.5-point margin over a TF-IDF and logistic regression baseline at full coverage, 92.2% against 88.7%, is thin for a model 61 times larger and 46 times slower [16]. What earned it a place was coverage under a confidence threshold. At a fixed 99% accuracy bar it auto-routed 72% of traffic against 53 to 55% for the cheaper models [17], 17 to 19 percentage points more volume [4]. The author calls that the difference between a model that "looks similar in a benchmark table" and one that "handles a third more volume without a human in the loop" [19].
One number to hold loosely: the post's headline accuracy is 92.2% [1], while the PyTorch fp32 row of its own table is 91.88% on a seeded 800-row sample of the held-out test set [10][12], a gap of 0.32 points [6].
The hosting it avoided goes unpriced in the writeup; what the author set out to have was "no server to pay for, no cold start, no infra to babysit" [3].
What to watch
- Latency numbers from a VNNI-capable x86 CPU on the same graph and the same 800-row sample.
- Whether the 0.13-point int8 accuracy drop holds on the full held-out test set rather than the seeded sample.
- Field measurements of first-load time for the 64 MB download on mobile connections.