Topic

Local CPU inference limits

What CPU-only inference does to multi-step agent loops, and the point at which per-step latency forces a move to hosted models.

Current clusters

build1 publisher

Ninety-five dollars of rented H100 time produced Kev's three-model Qwen3.5 port

Jared Palmer's Kev now ships as 0.8B, 4B and 9B variants that score typed answer options over frozen Qwen3.5 weights, and on his own development comparison hosted Jev still scores higher than the largest of them.

Publishers:runtimewire.com

Reality

Evidence62
Adoption14
Hype gap+10
Incentives62
Confidence55

build1 publisher

213 seconds per agent step evicts hybrid RAG from the local CPU

The FAISS-plus-BM25 retrieval in this writeup does address vocabulary mismatch, but the agent loop around it ran at over 213 seconds a step on CPU, and that figure decided the deployment, not the retrieval design.

Publishers:dev.to

Reality

Evidence34
Adoption14
Hype gap+42
Incentives38
Confidence56