SPEED-Bench re-tests speculative decoding at the batch size you actually serve

The discount comes from memory traffic rather than cleverness. At low concurrency, decoding time is dominated by moving model parameters out of HBM into the GPU's on-chip caches rather than by the arithmetic itself, which leaves compute units idle during sequential generation ^[3]. A verification pass reads those weights once and checks several drafted tokens against them, so processing multiple tokens carries only marginal overhead compared with one ^[5]. That is the whole trick. It works while there is idle compute to spend.

Raise the batch and the idle compute goes away. The paper's own framing says multi-user serving optimises throughput with larger batches, which pushes inference toward a compute-bound regime and often diminishes the speedups SD shows in latency-oriented tests ^[7]. So a figure measured at low batch is a measurement of a gap your production configuration was built to close ^[13]. That is a real result, but it describes a different serving profile than the one most production stacks run.

The second variable is acceptance, and the paper is direct that draft acceptance rates are highly sensitive to data domain and entropy while new methods keep being validated on inconsistent datasets ^[8]. MT-Bench, the usual venue, is called out for limited prompt volume and a lack of intra-category diversity ^[9]. If a method's headline number came from a narrow slice of prompts, the number transfers only if your traffic has comparable entropy. Quality holds up here: combined with rejection sampling, SD matches the target model's output distribution exactly ^[6]. The variable is throughput.

Then there is the baseline problem. Papers frequently evaluate using high-level libraries such as HuggingFace that do not reflect the optimisations present in vLLM, TensorRT-LLM, or SGLang ^[10]. A speedup measured against a slow baseline shrinks against a fast one, and SPEED-Bench's response is to integrate with vLLM and TensorRT-LLM directly and expose behaviours the authors say other benchmarks mask ^[4][11].

What the suite reports, per the abstract, is a set of findings rather than a single number: synthetic inputs overestimate real-world throughput, optimal draft length depends on batch size, low-diversity data introduces bias, and vocabulary pruning in state-of-the-art drafters carries caveats ^[1]. The batch-size result is the operationally awkward one. If the best draft length moves with load, draft length is a serving knob under an autoscaler, not a constant you set once in a config file and forget.

Read the excerpt for what it is. This is an arXiv preprint whose abstract and introduction assert the quantification; the tables are not in the material I have, so treat the four findings as claims about the authors' workload until the released data says otherwise ^[2][1]. The structural argument stands on its own regardless: two splits, one curated for semantic diversity and one for speedup across concurrencies from low-batch to high-load, is the right shape for a data-dependent optimisation ^[4].

One thing narrows the escape route. Frontier models including DeepSeek-R1, Qwen3-Next, Nemotron-3, and MiMo-V2-Flash now integrate multi-token prediction heads natively to support drafting ^[12]. When the drafter ships inside the model, you cannot swap it for one better matched to your domain. You inherit its acceptance rate, and measuring what that rate does to your throughput at your concurrency becomes the only lever you actually hold.