Folding 92 layers into one Pallas call put Kimi K3 at 709 tokens a second on TPU v7

During decode, a serving stack reads a layer's weights out of high-bandwidth memory, applies them to one step's worth of activations, then does the same thing for the next layer ^[9]. Inferact reports its non-speculative gains at batch sizes one through eight, where each weight read serves very few tokens ^[16]. Its kernel issues asynchronous copies that stage a later layer's weights in VMEM, the TPU's on-chip vector memory, while the current layer is still computing or communicating ^[10]. Overlapping those two needs one program on both sides of the layer boundary. The Kimi implementation puts 92 mixture-of-experts layers in a single Pallas call ^[11].

Inferact's stated position is that inference speed improves when software controls how weights move through a chip's memory instead of leaving the full schedule to general-purpose compiler optimization ^[13]. A megakernel bundles work that would otherwise run as separate kernels, and on TPU v7 the design uses a large software-managed pool of on-chip memory to stage weights ahead of their use ^[12].

709 against 452 is about 1.57 times the baseline ^[21]. The Qwen 3.8 27B pair, 1,515 against 695, is about 2.18 times ^[4][22]. Speculative decoding runs on both sides of the Kimi comparison ^[2], and the non-speculative megakernels came in at roughly 1.4 to 2 times the GB200 baseline at small batch ^[16], so the speculative Kimi ratio sits inside the band measured without speculation ^[27].

The chart's acceptance length is six ^[3]. If every verification step emits six tokens, 709 tokens a second works out to about 118 steps a second, or roughly 8.5 milliseconds per step ^[24]. 8.5 milliseconds per step is what compares to a decode step in someone else's loop, and it holds only if the reported acceptance length held across the run ^[3].

For the throughput number to say anything about another fleet, the batch has to be small ^[16] and the GPU side has to be configured the way vLLM's published Kimi K3 recipe configures it, because Inferact ran that as the baseline ^[19]. So the measurement compares a hand-written TPU schedule against a library default ^[11][19]. Chip count is equal at 16 a side ^[2], which per chip is 44.3 tokens a second on Ironwood against 28.3 on GB200 ^[23].

The compile claim generalises better than the throughput claim. Under 90 seconds against more than 30 minutes for a large model built from hundreds or thousands of XLA operations is a factor of at least 20 ^[14][25]. Anyone who has waited out an XLA compile will notice that number before the bar chart. The repository ships the Kimi and Qwen implementations, scripts, tests and an OpenAI-compatible server ^[15], and Kwon co-created vLLM at Berkeley, the engine Inferact was formed to develop ^[7]. Reproducing the Kimi bars takes 16 Ironwood chips and 16 GB200s ^[26].