Topic
Draft-model and multi-token-prediction techniques for accelerating decode, including DSpark and MTP heads.
KAIST and Seoul National University's AgSpec nearly doubles accepted draft length for coding agents by indexing files in the diff and JSON forms agents emit. It lives entirely in the retrieval index and leaves model weights untouched.
Reality
g factor's Qwen 3.8 27B benchmark has Together AI fastest at one stream, at 189.61 tok/s, while four of five engines finish within about 10% at 64 streams. Choosing a provider from these numbers starts with knowing how many streams the deployment will run at once.
Reality
Liquid AI shipped a 279.5-million-parameter draft model for LFM2.5-VL-3B on September 24, reporting 2.30x to 3.13x faster decoding on an Apple M5 Max. Image encoding and prompt prefill are unchanged, and the company's tests did not cover quantized deployments.
Reality
Inferact measured 709 output tokens a second on 16 Ironwood chips against 452 on 16 GB200s, at low concurrency with speculative decoding. The engineering worth reading is the hand-written memory schedule underneath.
Reality
AWS counted 13 SageMaker inference launches so far in 2026. The one that changes production behaviour most is a prioritized list of up to five instance types, each allowed its own model optimization settings.
Reality
NVIDIA's submission runs the same Qwen3.6-27B as the llama.cpp reference on the same Jetson board and finishes 6.4x sooner. Most of the gap comes from prompt tokens the runtime never has to prefill.
Reality
A September 3 playbook traces speculative decoding's draft architectures from EAGLE-3 to DFlash, and grounds the case in a 70B model that decodes at 15 to 20 tokens a second on eight H100s.
Reality
Alibaba's 27B scores 52 on the Artificial Analysis index from a 17GB quantized file. Filling its 262,144-token window needs roughly 16 GiB of KV cache on top of that, so the file size is the smaller half of the sizing question.
Reality
Red Hat says 70 to 80 percent of enterprise AI spending goes to inference, and blames a 70-billion-parameter model reading all 140GB of its weights for every token it writes. Its fix needs a draft model you train yourself.
Reality
DFlash uses a diffusion model to draft three tokens at once and claims to beat EAGLE-3. On one consumer GPU running llama.cpp on a JavaScript coding task, it trailed the drafter Google ships with Gemma-4-12B-it.
Reality
NVIDIA reports 2.5x more concurrent users for Nemotron 3 Ultra from its packaged NIM serving stack. The gain comes from five interacting layers measured on one agentic traffic shape, and NVIDIA ships the harness to retest it.
Reality
The release attaches speculative decoding to weights it publishes under MIT, which lowers what a self-hosting threat costs to stand up, even though the performance claim behind it is still the vendor's own.
Reality
A new paper fits log-linear laws for draft acceptance rate against pretraining tokens, draft capacity and decoding batch size, and argues with a roofline model that the tree you tune at batch 1 is the wrong tree in production.
Reality
Speculative decoding speedups depend on the data, and most published ones come from high-level scripts on narrow datasets. SPEED-Bench's authors argue the honest measurement happens inside vLLM or TensorRT-LLM, across concurrencies.
Reality
A 7B model split across Iowa and Oregon on free T4s went from 4.92 to 28.10 tokens per second. Most of the gain came from a drafter that stopped launching kernels one at a time.
Reality
A published NVFP4 and speculative-decoding config turns a 27B open-weights model into something you can try to serve. The 206.1 tokens per second figure is single-stream and unreplicated.
Reality
A conversion-pipeline checklist item, "MTP round-trip", turns on a distinction teams collapse: the training-time auxiliary loss is disposable, the inference-time draft head is not.
Reality
The update adds a path selector and a two-tap convolution rather than layers, recovering most of the accuracy that tripling the drafter bought at 15.2% latency, by the vendor's own numbers.
Reality
A week-long failure log on two RTX 3090s under WSL2 lands on one config at 170-210 tok/s. Everything before it died in dependency resolution, not in the math.
Reality