Samsung puts MAC trees in every LPDDR5X bank because HBM costs too much

The interesting engineering is not the multiply-accumulate logic, which Samsung has been demonstrating since 2021 and shipped through HBM stacks in AMD accelerators ^[5]. It is that this time the logic did not cost any banks. In HBM-PIM, Samsung had to remove memory banks to make room for the compute; in LPDDR5X-PIM every bank gets its own PIM unit ^[7]. A DRAM vendor that can add arithmetic without giving up capacity has a much easier conversation with a phone or automotive customer than one asking for a capacity trade.

The 8x bandwidth figure is an internal number and should be read as such. LPDDR5X-9600 peaks at 76.8 GB/s at the package pins, and Samsung's 614 GB/s is what the MAC trees see because the operands never leave the die ^[3]. Divide it out and the multiplier is 7.99x ^[17], which is roughly what you would expect from sixteen banks working in parallel on data that used to be serialised through a 128-bit bus. The end-to-end result is far smaller: 2.28x on model run time, 3.01x on tokens per second ^[1][2]. That gap between 8x internal bandwidth and 2.28x wall-clock is the whole story of PIM. Only the parts of inference that reduce to MACs against resident weights get accelerated, and everything else still runs at LPDDR5X speed.

The compatibility work is where this gets plausible as a product rather than a paper. The package keeps the standard 561-ball array, two 64-bit ranks, 16 GB modules, four dies per rank ^[8][9]. Conventional DRAM controllers still work, with the part switching between single-bank standard mode and multi-bank PIM mode on the command stream ^[10]. Samsung's stated obstacle was command reordering in conventional DRAM, and its answer, Address Align Mode, derives the register file addresses from the row and column address instead of an instruction register file ^[11]. In other words, the reorder buffer can shuffle PIM commands and the operands still land in the right registers. That is the difference between needing a bespoke memory controller and needing a driver.

The constraints are visible in Samsung's own walkthrough. A 512-byte FP8 activation is split into sixteen 256-bit packets, one per bank ^[12]; the 1 kbit vector register file holds at most four results before it must be drained ^[13]; and getting the answer back means the host flipping to conventional mode and issuing sixteen reads ^[14]. Small working set, shallow accumulation depth, frequent mode changes. This is a design for batch-one inference on a device where the weights are already sitting in memory, not for anything resembling a training rack.

That is the part worth sitting with, because Samsung led its presentation with the cost argument: memory is the bulk of AI chip cost and rising, and Micron warned at the same event that HBM wafer demand is getting worse ^[6]. The response on offer is not cheaper HBM. It is arithmetic bolted to the cheapest, lowest-power DRAM in volume production, sold on the premise that a 2.28x speedup in a phone-class part is worth more than a fraction of an HBM stack nobody can buy. Samsung is already pointing at LPDDR6X-PIM and says it hopes for an initial JEDEC specification this year ^[15], which matters more than the benchmark: a standard would let SoC vendors design against PIM without betting on one supplier.