Loading today’s stories
Build1 distinct publisher2 min readPublished
One question over nine files assembled a 13,773-token prompt, and llama.cpp reused three tokens of it. Reading beat writing by about five to one.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
On this box a prefill token costs ten milliseconds and a generated token costs a hundred and twenty five [3]. Priced that way, a 13,773-token prompt is about 138 seconds of reading before the model writes a word [1], which is what it would cost to generate roughly 1,100 tokens of answer [6]. The answer itself, if writing really was the thin sliver the profile left it [5], was on the order of 350 tokens [4].
The nominal figures do not add up to 291 seconds, and the gap is informative. 17,373 tokens at 100 a second is 174 seconds, about sixty per cent of the wall clock [2]. For prefill to be north of 85% of the profile [5], effective prefill on that turn had to be nearer 70 tokens a second [2], which is the sag the author flags as context grows [6]. A long prompt is billed at a worse rate than the one you measured on a short one.
The retrieval bug was not a relevance failure. Both engines found the right paragraph. They had indexed it with different chunk boundaries, so a union deduplicated by chunk id kept both copies [8]. About 9,000 tokens per question were the same text twice [9], and nobody spotted it because the assembled prompt is a wall of debug text that scrolls past looking exactly as it should [15]. Switching the key to normalized content returned 141 seconds, forty eight per cent of the original wait [3][10].
The cache line is the one worth pinning above a desk. llama.cpp will skip prefill for any prefix a slot has already seen [11], and the log shows it reused three tokens out of 13,773, about 0.02% [12][5]. The query rewriter, which is a good feature, carried its own system prompt into the conversation's slot. Prefix matching diverges at token three, so the two prompts took turns evicting each other, on every turn, for as long as the feature had existed [13].
Underneath both bugs sits the assumption that indexing is reading. It is not. Embeddings and a vector store are built once when files are added, and retrieval hands over a table of contents that the model still chews from token zero on every question [7]. Which is why the hardware answer is the expensive one: a box with twice the throughput would take this turn from 291 seconds to roughly 146 [7], and changing a dedup key got 150 [10] for the price of a commit.
Ranked by verification strength, evidence, and original report placement.
Prefill throughput sags as the context grows, and one user "question" is more than one model call.
A user opened a project containing nine files, asked "what are these documents about?", and waited 291 seconds for the first answer; the second question also took minutes.
The setup was an offline desktop app on llama.cpp, an M4 Pro with 24 GB, a 14B model at Q5, and nine files in the project.
From the author's own logs on that box: generation ran at about 8 tokens/second and prefill at about 100 tokens/second, both described as normal for a 14B at Q5.
That one question assembled a 13,773-token prompt, and the turn as a whole pushed 17,373 prompt tokens through the model once the side service calls are counted.
Reading dominated writing by roughly five to one on that turn, and the profile came out north of 85% prefill.
Follow any of these and your For You feed starts watching them — no settings page required.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Detailed first-party logs, single unreproduced box
The mechanisms are unusually well specified for a blog post: exact token counts, a quoted llama.cpp slot log, named dedup keys, and a before/after latency pair. But all of it comes from one author on one machine with no runtime version, no second configuration, and no third-party reproduction, and the headline 85% prefill share does not reconcile with the stated nominal throughput.
One self-reported deployment
Adoption evidence is limited to a single first-party disclosure about one offline desktop app plus its self-measured before/after and slot log. llama.cpp's prompt-cache behaviour is described as generic runtime behaviour, but no other deployments, users, or downstream adopters are evidenced, so this is an n=1 usage report rather than a diffusion signal.
Mildly overstated by generalization
The substance is measured and the author explicitly warns the arithmetic will not divide cleanly, which pushes against hype. The overstatement is one of scope rather than magnitude: a title and framing addressed to "your local RAG" generalize a single self-measured box with two specific bugs, and the 85%-plus prefill share is quoted as a profile while the stated throughput implies nearer 60% at nominal rates. Small positive gap, not a promotional one.
Reputational, not commercial
The author writes about his own application on a developer publishing platform, so there is a credibility and audience incentive in reporting a dramatic latency win. No product name, pricing, vendor, sponsorship, or funding relationship is disclosed in the supplied material, and the post's central move is admitting a design assumption the author held for a long time, which limits the promotional pull.
Mechanisms credible, magnitudes single-sourced
High confidence that the two described failure modes are real and generalizable: id-based dedup across engines with different chunk boundaries will duplicate passages, and sharing a llama.cpp prompt-cache slot between prompts that diverge at token three will thrash the cache. Moderate confidence in the specific numbers, which are self-measured on one machine, partially unreconciled, and uncorroborated by any second publisher.
build
A 27B Apache-2.0 model in 17GB makes local inference a wiring decision, not a demo1 distinct publisher
build
The load average had already peaked: reading 11.08 / 38.69 / 23.59 in the right order1 distinct publisher
build
Three manual interventions in a month, and every guard was working as designed1 distinct publisher
build
Six MariaDB versions, one real difference: the only reason to leave 10.6 is the July 2026 clock1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 26, 2026