Topic
Reduced-precision formats (e.g., FP8, GGUF, INT4) for model weights and KV caches that shrink memory use and speed inference at some cost to fidelity.
NVIDIA's RTX Spark PCs from Lenovo and Acer ship in October with up to 128GB of shared CPU/GPU memory, enough for a 4-bit 70B model. Long-context agents now fit on owned hardware, though a cost comparison with cloud inference waits on system prices.
Reality
Salvatore Sanfilippo's open-source ds4 engine runs a short list of large open-weight models locally by compressing their routed experts to about two bits. Even compressed, the supported builds need high-memory Macs or GPU systems that most people do not already own.
Reality
VIDRAFT says POCKET-Darwin-180B, a 111 GB 4-bit GGUF build of its 180B mixture-of-experts model, runs in llama.cpp on about $1,400 of consumer hardware. The accuracy evidence so far is one MMLU-Pro comparison that VIDRAFT reports itself.
Reality
Helion's autotuned GEMM beat vLLM's default CUTLASS and DeepGEMM backends on Hopper GPUs, by more than 10% throughput on some workloads, its authors report. The gain rests on per-shape tuning that can run for hours, a cost teams pay in place of kernel maintenance.
Reality
Lokutor released Oído, open-source transcription of any English sentence on a $5 ESP32-S3, with every figure so far computed off the board. Whether hardware built for command lists can take open-ended speech now depends on how the real boards measure.
Reality
Gemma 4 decodes on SageMaker's smallest GPU, a T4, at 0.8x an L4's speed with matching outputs from a patched vLLM image, a dev.to benchmark reports. The T4 is the cheaper choice per token only when it rents for under 80% of the L4's hourly rate.
Reality
Llama 8B on an RTX 4060 Ti 16GB fell from 42.5 to 3.8 tokens a second with a fifth of the model in system RAM, according to a dev.to benchmark. For a local coding agent, that makes VRAM for weights plus context the first spec to check, ahead of bandwidth.
Reality
Qwen Flash-Next NVFP4 ran in vLLM at 131,072 tokens of context and 16 sequences only after loader patches and cuts to its original targets. Its one load test covers only the bfloat16 KV-cache baseline, so operators on the later B12x stack have to run their own.
Reality
Developer xbill9 rebuilt Google's QAT Gemma 4 26B as int4 and fit it on one TPU v6e with 53,888 tokens of KV cache, against 3,456 for RedHat's FP8 build. Throughput nearly doubles, with accuracy checked on one classification suite.
Reality
The weights formula is the easy part of sizing local inference. Quantization metadata, the KV cache and the runtime's own buffers decide whether a 70B model fits, and a dev.to walkthrough shows where the advertised bit width stops helping.
Reality
Ternary weights at 1.76 bits put a 27B model into a 5.9GB file and let a laptop decode it at 28.1 tokens a second. The retention figure comes from Prism ML's own benchmark suite, not the table on the model card.
Reality
Rohan Pinto says BitNet b1.58 runs a 100-billion-parameter model at five to seven tokens a second on a single CPU. That is enough for overnight batches and air-gapped servers, and short of what a cloud tier needs.
Reality
AWS counted 13 SageMaker inference launches so far in 2026. The one that changes production behaviour most is a prioritized list of up to five instance types, each allowed its own model optimization settings.
Reality
PrismML's ternary build of Qwen3.8 27B keeps 98.2 percent of the full-precision benchmark average on both of the company's inconsistent scorecards, and the loss it does take is concentrated in knowledge and reasoning.
Reality
The ONNX export alone ran 1.58x faster than eager PyTorch with bit-identical accuracy. The int8 pass after it cost 8% in latency and bought a 64 MB download, small enough to serve as a static page with no backend.
Reality
A dev.to guide to running local models on 8GB prices the KV cache between 15KB and 160KB per token depending on architecture. At 32K tokens held, that spread is the difference between 0.5GB and 5GB of a fixed budget.
Reality
A dev.to walkthrough argues that the PROCESSOR field and the startup inference compute log line narrow a slow Ollama to about five named causes, and each of the three states PROCESSOR can print points at a different fix.
Reality
Two arms on the same laptop differ by one flag. The small card wins because llama.cpp leaves Gemma 4's 1.93 GB per-layer embedding table in mmap and pulls a few rows per token, so only about 1.08 GB of body is resident.
Reality
One endpoint fronts more than 300 models, but the company running the GPUs picks the inference engine and the quantization. A dev.to writeup says the quality gap that follows turns up in the response body, while the status code still reads success.
Reality
Alibaba's 27B scores 52 on the Artificial Analysis index from a 17GB quantized file. Filling its 262,144-token window needs roughly 16 GiB of KV cache on top of that, so the file size is the smaller half of the sizing question.
Reality
Build · September 13, 2026 · 1 publisher
Build · September 13, 2026 · 1 publisher
Build · September 11, 2026 · 2 publishers
Build · September 11, 2026 · 1 publisher
Build · September 10, 2026 · 1 publisher
Build · September 9, 2026 · 1 publisher
Build · September 8, 2026 · 1 publisher
Build · September 6, 2026 · 1 publisher
Build · September 5, 2026 · 1 publisher
Build · August 22, 2026 · 1 publisher
Build · August 19, 2026 · 1 publisher
Build · August 17, 2026 · 1 publisher
Build · August 16, 2026 · 1 publisher
Build · August 15, 2026 · 1 publisher