Nvidia's Groq-derived LPX rack posts 3,431 tokens/sec on 128GB of SRAM

Nvidia's on-stage demo showed 10,996 tokens per second on the same Gemma 4 31B model, which Arsovski flagged as self-reported.

Arsovski said the aim was "third-party verified independent benchmarks that you guys can trust."

Gemma 4 31B is a dense model small enough to sit inside a single LPX rack, and the picture at trillion-parameter mixture-of-experts scale, where memory capacity becomes the main constraint, went unaddressed.

Each LP30 carries roughly 500MB of on-die SRAM and no HBM, so a full LPX rack of 256 chips holds 128GB of memory delivering 40 PB/s of aggregate bandwidth against 315 PFLOPS of FP8 compute, with 350 ns of chip-to-chip latency in a Vera Rubin-compatible, MGX liquid-cooled rack that scales past 1,000 LPUs.

Keeping weights resident in SRAM rather than streaming them from HBM removes the memory-access latency that dominates single-token decode, and the design drops caches, branch prediction and out-of-order execution in favour of a fully deterministic pipeline the compiler schedules at clock-cycle granularity.

A Rubin GPU carries 288GB of HBM4, roughly 576 times the memory of a single LP30, so a 31-billion-parameter model at FP8 needs on the order of 62 LPUs to hold its weights, and a large mixture-of-experts model runs into four figures of chips across several racks.

Capacity is the cost of the SRAM-only design, and Nvidia is describing the LPU as for decode rather than as a general-purpose replacement for its GPUs, pitching the LPX rack as a decode co-processor.

Determinism lets the compiler predict power draw cycle by cycle, which Nvidia uses to pre-order current from the rack's regulators ahead of demand, cutting voltage droop by more than 60% and overshoot by more than 70% against an uncompensated load.

Per-block scheduling lets the hardware equalize heat instead of throttling to the hottest tile, which Arsovski put at roughly 10% to 11% additional performance under a fixed thermal limit.

Asked during Q&A about the blast radius of a chip that fails mid-workload, Arsovski said users "would experience the exact same as any other hardware in the industry" and would "just checkpoint it or reconfigure the hardware."

Nvidia synchronizes chips across racks to a single virtual clock in what it calls a plesiosynchronous network, with each chip acting as both processor and router so the fabric needs no adaptive routing or congestion sensing, and clock drift compensated at the chip-to-chip links.

Nvidia's self-reported 10,996 tokens per second is about 3.2 times the third-party measured 3,431 tokens per second on the same model.

A full 256-chip LPX rack's 128GB of SRAM is about 44 percent of the 288GB of HBM4 on a single Rubin GPU.

The roughly 62 LPUs needed to hold a 31B model at FP8 amount to about 31GB, some 24 percent of a 256-chip rack's 128GB, leaving on the order of 97GB for context state and everything else.