Loading today’s stories
Build1 distinct publisher3 min readPublished
Single-user inference streams weights out of memory, so the sizing question for a private document assistant is RAM and prompt length rather than which accelerator a vendor quoted. The laptop in question was three years old.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
The arithmetic runs on two clocks, and only one of them is the 11 tokens per second [3]. Decode is the steady clock: memory-bandwidth bound, consistent per token once the weights are warm [9]. Prefill is the other, compute bound, and the author puts a 2,000-token prompt at a second or two before the first output token appears [10]. Take the midpoint. A 1.5 second pause costs about the same wall time as generating 16 tokens at that rate [1]. That is why he says the felt quality of a CPU model tracks prompt length, and why capping context is a latency lever rather than a quality compromise [11].
The sizing exercise is subtraction. Four-bit GGUF cuts weights roughly 4x against 16-bit floats, which is what puts an 8B model inside a laptop's budget at all [12]. Subtract that weight file from 16GB and about 11GB remains for the operating system, the KV cache and whatever else the machine was bought to do [2]; the higher-quality Q8_0 build leaves about 7GB [3]. Integration is cheap on the same axis: llama.cpp ships a server binary speaking the OpenAI-compatible chat/completions API [14], so a client already pointed at a hosted endpoint needs a new base URL, not a rewrite.
Then the transfer question, because 11 tokens per second is a measurement of one laptop answering one person's contract questions [3]. It carries over if your batch size is one, your quantized weights fit in RAM, your prompts stay short, and your memory subsystem is comparable. That last condition does more work than the badge on the lid. The author credits Apple's unified memory, where the same fast memory serves CPU and GPU, for a fanless MacBook Air beating a larger Windows laptop under llama.cpp [15]. That is a bandwidth argument, not a thermal one.
The stopping points are narrow. At 11 tokens per second you are running at 55% of the comfortable reading speed the author cites [4], with three tokens per second of headroom above the point he calls sluggish [5]. He draws the upper limit himself: a chat product serving a thousand users, which this setup is not [4].
One caveat about the evidence. The piece promises real token rates measured on actual hardware rather than marketing ones [17], but the text supplied here carries no device table; the figures that survive are one laptop at 11, reading speed at 20, and sluggish at 8 [6]. The NPU line is softer again. The claim is that the NPUs in 2024 and later Windows and Intel laptops are built for low-power token generation and that software support is maturing quickly [16], which reads as a roadmap rather than a decode rate.
Ranked by verification strength, evidence, and original report placement.
A friend running a small consulting firm in Dubai asked the author what GPU he needed to run AI locally and had already been quoted a four-figure price for a workstation; what he actually wanted was a document chatbot over his firm's contracts.
The friend owned a three-year-old office laptop with 16 GB of RAM and no discrete GPU; two hours later the author had a 7B-parameter model running on it at about 11 tokens per second, answering questions from the firm's own PDFs.
The author describes about 11 tokens per second as not fast enough for a chat product serving a thousand users, but fast enough for a private assistant that costs nothing per query and keeps every contract on the machine.
The author states that training and fine-tuning are compute-hungry, while inference means one forward pass per token and is memory-bandwidth bound far more than compute bound at the small batch sizes a single user generates; what matters is how fast weights stream from memory, not how many FLOPS are available.
The author gives comfortable reading speed as roughly 20 tokens per second, and says that below 8 tokens per second interactive chat starts to feel sluggish.
The author notes that a batch job such as summarizing 400 documents overnight does not care about interactivity at all, which changes the calculus completely.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 30, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
The flash_attn error in llama.cpp is a layout constraint, and it decides your context window1 distinct publisher
build
Qwen 3.8's Apache-licensed 27B is the one you can actually own, and its KV cache is why1 distinct publisher
build
A 284B model at 25 tokens a second on one 5090, and 192 GiB of DDR5 doing the quiet part1 distinct publisher
build
Your 2026 GPU Decision Is Arithmetic: Bytes Per Parameter, Times Parameters, Plus Cache1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Solid plumbing, unaudited headline
Two very different grades of proof sit side by side. The mechanics — quantization roughly quartering weight size, GGUF as the container, llama.cpp's OpenAI-shaped server, the compute-bound prefill against bandwidth-bound decode — are standard and any reader can confirm them by running the clone, build and curl lines dev.to prints. The number in the headline cannot be confirmed at all: 11 tokens per second on one unnamed laptop, reported by the person who configured it, published immediately after a promise of measured per-device rates that the text never delivers.
One laptop in Dubai
Adoption evidence amounts to a single machine: one consulting firm's contracts, one three-year-old laptop, one afternoon. The tools involved are described as ubiquitous — llama.cpp as the engine behind virtually every local-LLM tool you have heard of — but that is the author's characterisation, unaccompanied by download counts, deployment disclosures or anyone else's numbers. Nothing here tells you whether the pattern survived past the demo.
Cool rhetoric, thin proof
The framing is unusually restrained for the genre — it tells you where CPU inference stops, names a sluggishness floor, and refuses to call 11 tokens per second product-grade. The overstatement is quieter than the prose: a promise of real measured hardware numbers that resolves into three round figures, and an NPU trajectory declared to be maturing quickly with nothing measured behind it. Modest claims still outrun their evidence when the evidence is one laptop.
Nothing being sold here
Follow the money and it runs out fast. Every tool named — llama.cpp, Ollama, llama-cpp-python, GGUF builds — is free, no vendor is credited or thanked, and the recommendation costs the reader nothing and steers him away from a four-figure purchase. What remains is the ordinary pull of a developer-platform post: the anti-vendor opening is a rhetorical position that rewards the author with attention, and a story where the writer solves a friend's problem in two hours flatters the writer.
One voice, easy to retest
We are relying on a single author at a single publisher, so the confidence ceiling is low by construction. What lifts it off the floor is that the falsifiable parts are cheap to falsify: anyone with a 16 GB laptop can pull the same quantized build and see whether the tokens come out near 11. The judgements we cannot retest — that a fanless Air beats a bigger Windows machine, that NPU support is arriving fast — are exactly the ones left unquantified.