Nvidia's Groq-derived LPX rack posts 3,431 tokens/sec on 128GB of SRAM

The memory ledger is the part to read first. A full rack is 256 LP30s, which totals 128GB of on-die SRAM feeding 40 PB/s of aggregate bandwidth against 315 PFLOPS of FP8 compute, with 350 ns of chip-to-chip latency in an MGX liquid-cooled, Vera Rubin-compatible frame ^[8]. One Rubin GPU carries 288GB of HBM4 ^[11]. The rack that posted the record therefore holds about 44 percent of the memory of a single GPU ^[3], and every byte of a served model has to live inside it.

Nvidia's own arithmetic puts a 31-billion-parameter model at FP8 at around 62 LPUs just to hold weights ^[11]. That is about 31GB, roughly a quarter of the rack, leaving on the order of 97GB for the 100K-context state and everything else ^[4]. Gemma 4 31B is dense and fits in one rack ^[7]. A large mixture-of-experts model runs to four figures of chips across several racks ^[11], and that case did not come up on stage ^[7].

The measurement deserves the same close reading. Artificial Analysis took the median of 50 sequential client requests at a concurrency of one, against a private pre-release endpoint served through Google Cloud ^[5]. Serving one request at a time is the condition that yields the highest per-user token rate a machine can post, and it does not map onto the multi-tenant load the rival endpoints were carrying [c5b]. Nvidia's own demo figure of 10,996 tokens per second on the same model is 3.2 times the audited number ^[6][2]; Arsovski labelled it self-reported before telling the room the aim was "third-party verified independent benchmarks that you guys can trust" [c6b].

Underneath is a pipeline with no caches, no branch prediction and no out-of-order execution, scheduled by the compiler at clock-cycle granularity, with weights resident in SRAM rather than streamed from HBM ^[9]. Determinism then earns its keep twice over. Because power draw is known cycle by cycle, Nvidia pre-orders current from the rack's regulators, which it says cuts voltage droop by more than 60 percent and overshoot by more than 70 percent against an uncompensated load ^[13]. Per-block scheduling also equalizes heat instead of throttling to the hottest tile, worth roughly 10 to 11 percent more performance under a fixed thermal limit, according to Arsovski ^[14]. Across racks, chips run against a single virtual clock and each one routes as well as computes, so the fabric needs no adaptive routing or congestion sensing ^[16].

That puts weight on the failure answer. Asked in Q&A about a chip dying mid-workload, Arsovski said users "would experience the exact same as any other hardware in the industry" and would "just checkpoint it or reconfigure the hardware" ^[15]. On a rack where one model's weights are spread over dozens of chips ^[11] and the network has nothing sensing congestion to steer around a hole ^[16], reconfiguration is not a small operation.