Speculators

Red Hat traces the inference bill to 140GB of memory reads per token

Red Hat says 70 to 80 percent of enterprise AI spending goes to inference, and blames a 70-billion-parameter model reading all 140GB of its weights for every token it writes. Its fix needs a draft model you train yourself.