Build1 publisher3 min readPublished Updated
Block diffusion drafting collapses EAGLE-3's eleven serial forward passes into one
A September 3 playbook traces speculative decoding's draft architectures from EAGLE-3 to DFlash, and grounds the case in a 70B model that decodes at 15 to 20 tokens a second on eight H100s.
The Engineer · Build desk
What happened
- A dev.to playbook dated September 3 frames the problem as memory traffic: generating one token from a 70B model on eight H100s means loading 140 GB of weights, so the cards are bandwidth-bound.
- At batch size 1 the post puts Llama 3.3 70B on eight H100s at roughly 15 to 20 tokens a second, with the GPUs idle about 90% of the time waiting for memory.
- The post cites 2.1x on math tasks from a head-to-head on Qwen3-4B with a five-layer drafter, credited to LMSYS, Z Lab and Modal in June 2026.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Adopting the parallel block means funding a drafter training run with a masked diffusion objective, and the inference team pays that before it sees an extra token per second.
- exposure An identical output distribution is not identical bytes, so teams running golden-output tests or replay audits inherit the failure mode the post files under bit-exact reproducibility.
- constraint The bandwidth case is stated at batch size 1, so a stack already serving concurrent requests has to measure its own idle fraction before any of the speedup argument applies to it.
- decision With the benchmark standard and the scheduler work present only as headings in the published text, a team choosing between drafters is deciding on one 4B-scale comparison.
Fifteen to twenty tokens a second works out to 50 to 66.7 milliseconds per token [4]. The post attributes about 90% of that interval to cards sitting idle waiting on memory [3]. So 5 to 6.7 ms per token is actual work [5]. Anything that only attacks the idle time on that configuration is capped near tenfold [5]. The 2.1x on math tasks the post cites came from a Qwen3-4B head-to-head with a five-layer drafter, credited to LMSYS, Z Lab and Modal in June 2026 [12]. For that ratio to transfer to the 70B stack you would need the same acceptance length and the same rejection rate on your own prompts, since those are the only two terms in the speedup expression [9].
That expression is Speedup approximately (1 + AL) / (1 + rho times D) [9]. Draft depth sits only in the denominator, multiplied by the rejection rate, so depth is nearly free when drafts are accepted and punitive when they are not [9]. EAGLE-3 pays a second, separate price for depth. It is one transformer decoder layer reading the target model's last hidden state, run autoregressively, so a depth of 11 means 11 serial forward passes and 11 kernel launches with the host-device synchronisation that implies [10][11]. Eleven launches to produce eleven tokens the verifier may then reject. The post describes the resulting utilisation curve as a sawtooth [11].
DFlash emits the whole block of D tokens in a single parallel forward pass, and it buys that by training the drafter with a masked diffusion objective [13].
The losslessness claim lives in the acceptance path. Accept the draft token with probability min(1, p/q); on rejection, resample from norm(max(0, p minus q)) [7]. The post says the output distribution is identical to the target model's, calls that "Lossless", and immediately qualifies it as holding "in theory" [8]. Its own gotchas section lists bit-exact reproducibility next to an acceptance rate paradox [17].
The supplied text stops after DFlash's first innovation. XPress, NVIDIA's five production guidelines, SPEED-Bench, the SGLang Spec V2 overlap scheduler and AceSpec's 50 Kbps edge-cloud case are section headings in it [18][20].
The rest of the week's evidence is community-scale. The post counts five arXiv papers in the 72 hours to September 3, plus an NVIDIA co-design framework and benchmark standard [14]. It points at a Hacker News thread on Slotstream, a 125B model at 12 tokens a second on a 48 GB Mac, which reached 226 points and 108 comments [15]. Forty-eight gigabytes across 125 billion parameters is about 3.1 bits per parameter before the KV cache takes its share [16], so that demo is carrying a quantisation story as much as a drafting one.
What to watch
- Acceptance length and rejection rate figures for DFlash at 70B scale, on prompts other than math benchmarks.
- Whether the SGLang Spec V2 overlap scheduler holds its win at serving batch sizes above one.
- Published guidance on the bit-exact reproducibility gotcha for teams that run golden-output tests.