Skip to content

Topic

Quantization formats

Reduced-precision formats (e.g., FP8, GGUF, INT4) for model weights and KV caches that shrink memory use and speed inference at some cost to fidelity.

Current stories

buildOne report1 publisher

Unsloth's figures suggest a 128 GB Mac can run NVIDIA's Nemotron 3.5 Super VL with vision

Nemotron 3.5 Super VL, NVIDIA's 120B model, had GGUF, MLX, FP8 and EXL3 conversions on Hugging Face a day after its October 15 release. The roughly 128 GB memory figure comes from a DEV Community write-up whose author did not run the model.

Publishers:dev.to

Reality

Evidence55
Adoption
Not scored
Hype gap+15
Incentives
Not scored
Confidence55
buildOne report1 publisher

The Neuron clocks NVIDIA's 74B Nemotron Bolt at 65.5 tokens per second on a laptop

The Neuron measured NVIDIA's Nemotron 3.5 Bolt 74B-A9B at 65.5 tokens per second on a laptop in a hands-on test. The build was heavily compressed, and the review's tool-assisted wins went to the hosted Super VL.

Publishers:theneuron.ai

Reality

Evidence45
Adoption
Not scored
Hype gap+15
Incentives
Not scored
Confidence50
buildOne report1 publisher

Surface Laptop Ultra edges an M4 Max Mac Studio on local models, plugged in

Microsoft's $4,300 Surface Laptop Ultra ran local models generally a bit faster than an M4 Max Mac Studio, The New Stack's reviewer found. The edge was measured on the charger with models sized to the machine, so buyers should check their own model mix.

Publishers:thenewstack.io

Reality

Evidence50
Adoption
Not scored
Hype gap+15
Incentives
Not scored
Confidence55
buildConfirmed12 publishers

NVIDIA's Bolt packs 74B parameters into a claimed 24GB by storing them at 2 bits

NVIDIA says its Nemotron 3.5 Bolt 74B-A9B agent model fits in 24GB and will ship with open weights, data and recipes. The footprint, the license terms and the quality claims stay unverified until the weights are live and outside testers have run them.

Perspective Coverage

12 publishers
Builder
Builder 40%
Operator
Operator 35%
Investor
Investor 25%

Reality

Evidence42
Adoption
Not scored
Hype gap+25
Incentives60
Confidence58
buildConfirmed3 publishers

Black Forest Labs' distilled FLUX.3 Image Klein renders in about 2 seconds on an H100 in Diffusers

Black Forest Labs' open FLUX.3 Image Klein weights now load in Hugging Face Diffusers for self-hosted image generation and editing. Hugging Face times the distilled checkpoint at about 2 seconds per 1024 x 1024 image on an H100, and The Decoder reports an INT8 build for 8 GB cards.

Perspective Coverage

3 publishers
Builder
Builder 52%
Operator
Operator 23%
Investor
Investor 25%

Reality

Evidence55
Adoption
Not scored
Hype gap+20
Incentives60
Confidence68
buildOne report1 publisher

ExecuTorch lowers PyTorch models straight to CMSIS-NN kernels on Cortex-M microcontrollers

PyTorch's ExecuTorch Cortex-M backend ran four MLPerf Tiny models at 21.5 ms or less each, using under 90 KiB of RAM on a single Cortex-M55 core. The test is the PyTorch team's own, on one Alif Ensemble E8 board, so the figures hold for similar cores and INT8 models.

Publishers:pytorch.org

Reality

Evidence45
Adoption
Not scored
Hype gap+15
Incentives55
Confidence50
buildOne report1 publisher

A clean config diff against Qwen3.8-27B does not let AREX-2 reuse the base model's serve command

BAAI's AREX-2 config declares one multi-token-prediction layer, but its weight index lists 0 such tensors against Qwen3.8-27B's 15. Its dev.to testers say the base-model drafter and quant need rechecking too, though they have not run AREX-2.

Publishers:dev.to

Reality

Evidence55
Adoption
Not scored
Hype gap−10
Incentives
Not scored
Confidence50
buildOne report1 publisher

One 80 GB GPU holds 4-bit 70B weights and at most 11 conversations of 8,192 tokens

Serving a 70B model takes about 141 GB of GPU memory for weights at 16-bit, 73 GB at FP8 and 40 GB at 4-bit, according to a dev.to post. Fine-tuning takes about 50 GB with QLoRA and 1.1 TB with full AdamW. The post maps both onto GPU setups and hourly rental prices.

Publishers:dev.to

Reality

Evidence72
Adoption
Not scored
Hype gap+5
Incentives
Not scored
Confidence68
buildOne report1 publisher

NVIDIA and Princeton cache SSM inputs in place of state to speed hybrid-model decoding

NVIDIA and Princeton's Dao AI Lab cache SSM inputs instead of state, targeting a decode step where SSM layers are the largest component up to 100K tokens. ReplaySSM is already integrated into TensorRT-LLM, vLLM and FlashInfer, NVIDIA says.

Reality

Evidence45
Adoption
Not scored
Hype gap
Not scored
Incentives55
Confidence50
buildOne report1 publisher

SoloQ's 45% FLOP cut for diffusion LLMs rests on one dev.to post and an unnamed lead author

SoloQ, a calibration-free quantizer for diffusion LLMs, cuts inference FLOPs about 45%, according to a dev.to post on an arXiv paper dated 2026-10-05. That write-up is the only source, and its cost numbers come from a scenario with an unnamed provider.

Publishers:dev.to

Reality

Evidence22
Adoption
Not scored
Hype gap+45
Incentives
Not scored
Confidence60
leadershipOne report1 publisher

Tencent open-sources a 33-language translation model that fits on a device in 440 MB

Tencent has open-sourced its Hy-MT2 translation models for 33 languages, with the smallest compressed to 440 MB to run on devices. The claim that the small model beats Microsoft's API is Tencent's own, so teams paying per call have a candidate to test.

Publishers:github.com

Reality

Evidence30
Adoption
Not scored
Hype gap+35
Incentives70
Confidence40
buildOne report1 publisher

VIDRAFT ships a 111GB Darwin-180B that pages its experts from SSD on 8GB-VRAM laptops

VIDRAFT released a 111GB, 4-bit build of its 180B-parameter Darwin model that it says runs on a laptop with 8GB of VRAM and 32GB of RAM. Air-gapped teams still need a measured token rate on that laptop before they can price it against a server.

Publishers:dev.to

Reality

Evidence30
Adoption
Not scored
Hype gap+40
Incentives85
Confidence40
buildOne report1 publisher

Strata Hits 110.8 tok/s on an RTX 4090 by Activating Only 6B Parameters per Token

Strata runs a 125B-class Qwen model at about 100 tok/s on one RTX 4090 because its sparse MoE activates only about 6B parameters per token. The design moves the hardware bill to system RAM, and the 100 tok/s figure holds mainly for code and structured output.

Publishers:dev.to

Reality

Evidence45
Adoption15
Hype gap+40
Incentives
Not scored
Confidence40
buildOne report1 publisher

NVIDIA's RTX Spark PCs fit a 4-bit 70B model in 128GB with about 90GB to spare

NVIDIA's RTX Spark PCs from Lenovo and Acer ship in October with up to 128GB of shared CPU/GPU memory, enough for a 4-bit 70B model. Long-context agents now fit on owned hardware, though a cost comparison with cloud inference waits on system prices.

Publishers:dev.to

Reality

Evidence30
Adoption
Not scored
Hype gap+35
Incentives
Not scored
Confidence30
buildOne report1 publisher

VIDRAFT's 4-bit Darwin-180B build streams most of its weights from SSD on a 40 GB laptop

VIDRAFT says POCKET-Darwin-180B, a 111 GB 4-bit GGUF build of its 180B mixture-of-experts model, runs in llama.cpp on about $1,400 of consumer hardware. The accuracy evidence so far is one MMLU-Pro comparison that VIDRAFT reports itself.

Publishers:dev.to

Reality

Evidence30
Adoption
Not scored
Hype gap+45
Incentives60
Confidence35
buildOne report1 publisher

Helion's autotuned GEMM outruns vLLM's CUTLASS and DeepGEMM paths on Hopper GPUs

Helion's autotuned GEMM beat vLLM's default CUTLASS and DeepGEMM backends on Hopper GPUs, by more than 10% throughput on some workloads, its authors report. The gain rests on per-shape tuning that can run for hours, a cost teams pay in place of kernel maintenance.

Publishers:pytorch.org

Reality

Evidence35
Adoption15
Hype gap+10
Incentives70
Confidence40

Earlier coverage

  1. Loading PrismML's 5.95 GB Bonsai 2 requires the company's own llama.cpp fork

    Build · September 17, 2026 · Confirmed2 publishers

  2. Edge0's 35B would need 22 GB/s of SSD bandwidth without the file cache

    Build · September 11, 2026 · Confirmed2 publishers