Loading today’s stories
Build1 distinct publisher3 min readPublished
Across 20 paired writing-correction cases on Ollama for Windows, the two models succeeded on the same 18 and failed on the same 2, while the 4B averaged 23.99 seconds of cold start against 54.37. That reorders local shortlisting.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
Read the label on the timing number first. The author reports "mean total cold-start times" [14], so weight loading and answer generation sit inside one figure, and nothing in the run separates them. That decides whether the 2.27x [7] transfers. It is spendable only where every request pays the load, which is the first call after Ollama drops the model, not the tenth call to a resident one. No warm-path figure is reported, so there is nothing here to size a keep-alive deployment with.
For the rest to transfer you also need the same quantization on both sides of the comparison. The Qwen3 entries are pinned, qwen3:4b-q4_K_M and qwen3:8b-q4_K_M, while the older model is listed as bare qwen2.5:7b [2]. The one tag without a pinned quantization is the one being displaced.
Then the marginal-case arithmetic. Moving from the 4B up to Qwen3 8B buys one extra complete case, 19/20 against 18/20 [4], and one extra localized error region, 21/21 against 20/21 [9]. It costs 36.25 seconds more per cold call [1], which is 151% more wall clock, a factor of 2.51 [2][3]. On a 20-case set a single case is worth five percentage points [4], so the 95% against 90% gap is one disagreement rather than a trend you can plan against. The percentage claims in the writeup are consistent on their own terms: both the 55.9% and the 9.7% figures are computed against the slower model [5][8].
The paired design is the part worth borrowing. Equal totals can hide compensating failures, since two models at 18/20 could in principle miss four different cases and still tie. Here the discordant count was zero across all 20 [5], which says more about substitutability than the score line does. The author is explicit that this is not universal equivalence, only that no complete-case difference appeared inside this controlled set [12], and that the times describe one local Windows and Ollama configuration [13].
The prior going in was that the 7B would keep a noticeable edge, and this run did not find one [11]. The earlier experiment in the same framework put Qwen3 8B and 14B at 19/20 each, with the 8B averaging roughly half the 14B cold start [10]. Two runs, same direction. For this narrow task, on this rig, the generation label carried more than the parameter count, and the larger model is the one that now has to justify its load time. What it has to justify against is a scoring rule that only counts a case complete when every expected correction lands without an incorrect replacement, a false error, or configured fact loss [3], which is a stricter bar than a diff-based pass rate.
Ranked by verification strength, evidence, and original report placement.
The benchmark uses 20 paired writing cases, comprising 16 sentences with expected errors and 4 already-correct controls, each evaluated with the same frozen reference and the same evaluation framework across three locally hosted Qwen models, producing 60 local responses in total, executed through Ollama on Windows under the same benchmark conditions.
The three models tested are listed as qwen2.5:7b, qwen3:4b-q4_K_M and qwen3:8b-q4_K_M; the two Qwen3 tags carry an explicit q4_K_M quantization suffix and the Qwen2.5 7B tag does not.
For the primary metric, a case counted as complete only when all expected correction requirements for that case were satisfied without an incorrect replacement, a false error, or configured fact loss.
Primary correction results: Qwen3 4B 18/20 complete cases (90%), Qwen2.5 7B 18/20 (90%), Qwen3 8B 19/20 (95%).
In the paired comparison of Qwen2.5 7B and Qwen3 4B, both succeeded on the same 18 of 20 cases, both failed on the same 2, and there were 0 cases with different complete-case outcomes.
Mean total cold-start times in the tested configuration were 23.99 s for Qwen3 4B, 54.37 s for Qwen2.5 7B and 60.24 s for Qwen3 8B.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 29, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
Before you buy another GPU, check num_ctx and the rope base1 distinct publisher
build
OpenClaw makes the channel the architecture, and the reasoning loop a lodger1 distinct publisher
build
Your first MCP workflow should be a draft queue, not an agent with keys to the inbox1 distinct publisher
build
A Gulf bank's compliance rule priced out to $133 of GPU per seat1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Precise, disclaimed, unreplicated
The measurements are more disciplined than most hobby benchmarks: a pass condition stated before the scores, per-case outcomes rather than aggregates, cold starts to two decimals, and the author himself refusing to generalize from either the pairing or the timings. What caps it is structural. Twenty cases means the entire quality separation between models is one sentence; the Qwen2.5 tag is unquantized while both Qwen3 tags are 4-bit, which the write-up lists but never treats as a confound; and the prompts, harness and raw responses are described rather than shared, so no reader can reproduce a number without rebuilding the setup.
One workstation, twice
The only usage anywhere in this reporting is the author's own: two runs of his writing-coach evaluation framework on a local Windows box, 60 responses in this one. No product ships on the finding, no team reports switching models because of it, and nobody outside dev.to has repeated it. That is enough to show the models run locally at these sizes and nothing more.
Framing runs ahead of twenty sentences
The overreach is not the author's — he disclaims universal equivalence and universal speed rankings in plain terms. It is what the finding becomes when compressed: 'matches case for case at half the cold start' invites a shortlist decision that 16 error sentences and one seeded-error-per-region-and-a-half cannot carry, especially with the quantization mismatch unnamed. Small positive gap, mostly from the packaging rather than the measurement.
House framework, house verdict
No vendor money is visible and no model maker is being flattered — Qwen wins against Qwen. The pull is different: this is the second installment of a series published under a writing-assistant brand's own dev.to handle, scored by that brand's own evaluation framework, and the framework looks better when its runs produce a clean, quotable reversal. Self-published, self-scored, self-refereed, with the author volunteering the caveats an outside reviewer would have demanded.
Cheap to check, checked by no one
We are confident about what was reported and much less confident about what it means. Every figure is stated unambiguously and internally consistent — the 2.27x and 9.7% claims both reconcile against the means — so the risk is not misreading. It is that a 20-case, single-run, single-machine test with mismatched quantization could land differently on a second pass, and no second pass exists.