Two regions, 86ms apart, 28 TPS: the WAN was not the bottleneck, Python was

Run the arithmetic on the round rather than the token and the numbers get more interesting than the headline. At 28.10 TPS with 4.07 tokens committed per accepted round trip ^[2][8], a round lasts about 145ms ^[4]. Eighty-six of those milliseconds are wire ^[6] and 25 are the draft model ^[14], which leaves roughly 34ms for the 7B verification slice on Node 1 and every other cost in the loop ^[4]. Do the same sum for the eager-mode configuration the author reports at 14.3 TPS ^[9] and the round is 285ms, of which 86ms is wire and 112ms is drafting ^[12], leaving 87ms of residual ^[5].

Those two residuals do not match. The graph capture removed 87ms of drafting per round, but round time fell by 140ms ^[6]. Fifty-three milliseconds came off something the writeup does not itemise ^[6]. Part of that is probably the mixing of peak throughput figures with an average acceptance rate, which is the weakest joint in the measurement, and it is the first thing worth pinning down before anyone treats 28.10 as reproducible.

The mechanism behind the drafter fix does check out, though, and it checks out almost exactly. Eight candidate tokens meant eight forward passes of roughly 1,500 kernels each ^[10], so about 12,000 launches per round; at the 8 to 10 microseconds Python needs to issue each one ^[11], that is 96 to 120ms of pure launch overhead ^[8]. The reported 112ms draft phase ^[12] sits inside that bracket, which means the draft phase was essentially all bookkeeping and none of it compute. A 65% GPU idle rate ^[12] is the same fact seen from the other side.

Which reframes what the network was ever costing. At the 4.92 TPS baseline ^[5], a token takes 203ms and the 86ms round trip is 42% of it ^[3]. Amortised over 4.07 accepted tokens, the same round trip costs 21ms per token ^[2]. After that, the Python launch overhead in the drafter was larger than the entire Iowa-to-Oregon-to-Ohio path ^[8][6]. The interesting engineering conclusion is not that the WAN turned out to be survivable; it is that a 2,000km hop and a t3.micro relay ^[4] were quietly cheaper than a for-loop in the same process.

The ceiling that remains is arithmetic, not effort. At 86ms per round trip and 4.07 tokens accepted, the network alone caps this topology at 47.3 TPS ^[7], and 28.10 is 59% of that ^[7]. Nothing done to the drafter moves that number. Raising it means a higher acceptance rate, a larger draft window, or fewer hops between the two shards.

The cost of the trick is a memory contract that transformers does not honour by default. CUDA graphs capture fixed addresses, and HuggingFace's DynamicCache calls torch.cat at every token step, allocating a fresh buffer that the graph never learned about ^[15][16]. The failure signature the author describes is repeated output, not a crash ^[15], which is the expensive kind: it looks like a sampling problem. Holding it together took StaticCache, in-place tensor mutation, manual position_ids updates, and an in-place KV rewind for rejected drafts ^[17]. That is four places where a library upgrade can silently reintroduce a stale pointer.