Topic

Storage-backed inference

Serving model weights from flash or SSD on demand, often through memory mapping, so a model larger than available RAM can run.

Current clusters

build2 publishers

Edge0's 35B would need 22 GB/s of SSD bandwidth without the file cache

Edge0's Apache-2.0 engine reports a 35B model peaking at 2.9 GB of RAM on a 24 GB Mac mini. Work the decode rate against the active set and most of those expert reads have to be coming from the operating system's file cache.

Publishers:dev.toruntimewire.com

Reality

Evidence38
Adoption12
Hype gap+48
Incentives55
Confidence52