Topic
Running large language models on consumer GPUs and Apple Silicon with llama.cpp, Ollama and LM Studio.
Ollama and llama.cpp added local endpoints within a week for Jev decision models, 144M-to-27B checkpoints that return a label and a probability. Routing calls can move off hosted models once teams check each checkpoint's licence and test it on their own labels.
Reality
Ollama 0.35.1 picks Gemma 4's prompt template from the model's name, so a copied or renamed 12B model loses four prompt tokens when thinking is off. Teams that save the library model under their own name can pin RENDERER gemma4-large in the Modelfile to keep the template that matched Google's.
Reality
VIDRAFT released a 111GB, 4-bit build of its 180B-parameter Darwin model that it says runs on a laptop with 8GB of VRAM and 32GB of RAM. Air-gapped teams still need a measured token rate on that laptop before they can price it against a server.
Reality
llama-server b11430 reports logprob 0.0 for every speculatively decoded token, dragging one test's mean logprob from -0.48 to -0.0011. Nothing in the response or the server log flags the fill-ins, so evals and calibration built on those numbers go wrong quietly.
Reality
AMD published Gorgon Halo AI benchmarks days before Nvidia's expected RTX Spark launch, claiming a 1.1x to 32.2x lead over Intel's Core Ultra X9 388H. Until RTX Spark is measured, buyers weighing Gorgon Halo machines that cost upwards of $7,099 can compare the two only on memory.
Reality
Strata runs a 125B-class Qwen model at about 100 tok/s on one RTX 4090 because its sparse MoE activates only about 6B parameters per token. The design moves the hardware bill to system RAM, and the 100 tok/s figure holds mainly for code and structured output.
Reality
Ollama since 0.34.4 lets Gemma 4 skip the requested JSON schema, returning bare text with HTTP 200 in 8 of 24 test calls. Until the open fix ships in a release, structured output on local thinking models needs a shape check in the client.
Reality
Salvatore Sanfilippo's open-source ds4 engine runs a short list of large open-weight models locally by compressing their routed experts to about two bits. Even compressed, the supported builds need high-memory Macs or GPU systems that most people do not already own.
Reality
VIDRAFT says POCKET-Darwin-180B, a 111 GB 4-bit GGUF build of its 180B mixture-of-experts model, runs in llama.cpp on about $1,400 of consumer hardware. The accuracy evidence so far is one MMLU-Pro comparison that VIDRAFT reports itself.
Reality
llama.cpp merged a /v1/systemone endpoint on October 2 that returns typed answers with probabilities from five open model families running locally. Whether it can replace hosted classification calls depends on calibration that each team has to measure on its own data.
Perspective Coverage
3 publishersReality
DeepAgents runs in a four-file Docker Sandbox kit whose agent can reach only a local Model Runner on port 12434, with no cloud keys. The egress policy is careful work, and exact reproduction still rests on what PyPI serves when each sandbox is created.
Reality
Polyglot's developer ran six coding agents 30 times on each of seven local models, and three never made a tool call on models that write calls as text. The author's own error bars say 30 runs can sort agents into tiers but cannot rank two agents inside one.
Reality
Ollama 0.35 adds a /v1/systemone endpoint returning a choice, yes/no or score from models run on the device. Its 9B Nimble model matched human moderation labels 70.3% of the time in its maker's test, so each team still sets its own review threshold.
Reality
Prism's ternary Bonsai 2 27B ran at two to three tokens a second on a CPU-only Hetzner VPS in a dev.to test, against Simon Willison's 20 to 44 on a Mac. The sub-6GB file fits a 16GB box easily, but at that speed it only suits batch jobs.
Reality
One developer's 10-second inference probe pulled 3 of 4 healthy Macs from a Caddy pool where normal inference takes about 31.8 seconds. Stall detection now runs on real requests, so users who hit a wedged node supply the timeouts that get it removed.
Reality
Llama 8B on an RTX 4060 Ti 16GB fell from 42.5 to 3.8 tokens a second with a fifth of the model in system RAM, according to a dev.to benchmark. For a local coding agent, that makes VRAM for weights plus context the first spec to check, ahead of bandwidth.
Reality
One developer's audit of Claude Code found 97 of 200 agent steps could run on a local model, against 4 of 100 whole requests. That makes the agent step the unit to route on, on evidence from one person's sessions and one RTX 4070.
Reality
Cupertino is now selling desktops as an alternative to token bills. On the configurations that can host a useful model, payback runs two to four years, and the model you can host is a tier down.
Reality
Oasis Security says a malicious page can reach the unauthenticated API on port 11434 and poison every later conversation. NVIDIA's own source puts that bind on one platform path.
Reality
Quantized to about 4 GB, a 7B model still runs out of GPU memory near 30,000 tokens as its KV cache grows, according to a dev.to walkthrough. Sizing a card by its weight file alone leaves that growing cache off the memory budget.
Reality
Build · September 24, 2026 · 1 publisherOne report
Build · September 23, 2026 · 1 publisherOne report
Build · September 22, 2026 · 1 publisherOne report
Build · September 19, 2026 · 1 publisherOne report
Build · September 19, 2026 · 1 publisherOne report
Build · September 17, 2026 · 2 publishersConfirmed
Build · September 17, 2026 · 1 publisherOne report
Build · September 17, 2026 · 1 publisherOne report
Build · September 17, 2026 · 1 publisherOne report
Build · September 17, 2026 · 1 publisherOne report
Build · September 16, 2026 · 1 publisherOne report
Build · September 16, 2026 · 1 publisherOne report
Build · September 16, 2026 · 1 publisherOne report
Build · September 15, 2026 · 1 publisherOne report
Build · September 15, 2026 · 1 publisherOne report
Build · September 14, 2026 · 1 publisherOne report
Build · September 14, 2026 · 1 publisherOne report
Build · September 14, 2026 · 1 publisherOne report
Build · September 14, 2026 · 1 publisherOne report
Build · September 13, 2026 · 1 publisherOne report
Build · September 11, 2026 · 1 publisherOne report
Build · September 11, 2026 · 1 publisherOne report
Build · September 10, 2026 · 1 publisherOne report
Build · September 10, 2026 · 1 publisherOne report
Build · September 10, 2026 · 1 publisherOne report
Build · September 8, 2026 · 1 publisherOne report
Build · September 8, 2026 · 1 publisherOne report
Build · September 8, 2026 · 1 publisherOne report
Build · September 6, 2026 · 1 publisherOne report
Invest · September 4, 2026 · 1 publisherOne report
Product · September 3, 2026 · 1 publisherOne report
Build · August 31, 2026 · 1 publisherOne report
Build · August 30, 2026 · 1 publisherOne report
Build · August 29, 2026 · 1 publisherOne report
Build · August 25, 2026 · 1 publisherOne report
Build · August 22, 2026 · 1 publisherOne report
Build · August 22, 2026 · 1 publisherOne report
Build · August 21, 2026 · 1 publisherOne report
Build · August 21, 2026 · 1 publisherOne report
Build · August 20, 2026 · 1 publisherOne report