Loading today’s stories
Build1 distinct publisher3 min readPublished
According to a dev.to writeup, generated docs go wrong exactly where a reader cannot check them without opening the source, so the useful pass feeds the model signatures, tests and commits, and allows UNKNOWN as an answer.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
The pathspec is where the design shows. `git diff --name-only "$BASE...HEAD" -- '*.py' ':!tests/*' ':!*_pb2.py' ':!**/migrations/*'` selects changed Python sources and drops tests, generated protobuf stubs and migrations from the set that gets suggestions [16]. Tests come back as context through `grep -rl "$mod" tests/ | head -3 | xargs -r cat`, with `$mod` set to the file's basename [17]. Migrations do not come back, and neither does a generated OpenAPI spec, though the same post names both as artefacts that already encode the truth [9]. If the fact you need is `ALTER TABLE charges ADD COLUMN settled_at timestamptz NULL`, which carries three facts in one line [11], the pass writing your docstrings never sees it [4].
The history block is the sharper gap. It is `git log --oneline -5 -- "$f"`, five subject lines [18]. The commit the post uses to justify the whole approach was found with `git log --follow -p`, and the load-bearing text sat in the message body: "revert jitter, it broke replay determinism in tests" [12]. Subjects do not carry bodies, so that sentence is not in the prompt [1]. The input still shows a flat sleep with no reason attached, which is the exact spot where the model reached for tenacity's vocabulary [6].
The test path has its own edges. Matching is a basename substring, `head -3` caps it at three files, and `xargs -r cat` emits contents with the paths stripped [2]. Usually survivable for docstrings, thin if you wanted provenance.
Then the transfer question. Three fabricated facts in about 900 lines is one per 300 lines [3], and that is an illustration in a blog post rather than a measurement on your repo. The rate depends on how close your abstractions sit to the popular ones: a hand-rolled retry wrapper is exposed precisely because tenacity is good [6]. A domain type with no public analogue should draw fewer confident inventions and more UNKNOWN, provided UNKNOWN is a permitted output [13].
Adoption is cheap. `pipx install llm`, then `llm keys set anthropic` [20], a model pinned in the script to `claude-sonnet-4-5` [19], and one file, `prompts/docstring.md`, where the post says the constraints live [22]. Output is one markdown file per changed source under `.docsuggest`, and nothing is written back into the tree [23].
That last detail is the honest part of the design. The script does not claim to verify anything. It stages a suggestion beside the signature it came from and leaves a person to compare the two, which is the whole job, because the failure mode it is built against survives for weeks and surfaces as a bug filed against the library by the first person to run the example [5].
Ranked by verification strength, evidence, and original report placement.
The script's history section is git log --oneline -5 -- "$f".
The post lists artefacts in a codebase that already encode the truth: type signatures, tests, OpenAPI or JSON Schema specs generated from code (FastAPI, tsoa, utoipa), migrations, and git history.
The signature def charge(account_id: UUID, amount: Money, *, idempotency_key: str) -> ChargeResult already states every parameter, which are keyword-only, and the return type, so a model constrained to the signature cannot invent a fifth parameter.
ALTER TABLE charges ADD COLUMN settled_at timestamptz NULL states three facts: the column exists, its type, and that it is nullable.
The script selects files with git diff --name-only "$BASE...HEAD" -- '*.py' ':!tests/*' ':!*_pb2.py' ':!**/migrations/*', excluding tests, generated protobuf stubs and migrations.
Test context is gathered with mod=$(basename "$f" .py) then grep -rl "$mod" tests/ | head -3 | xargs -r cat, labelled in the prompt as the only examples known to run.
Follow any of these and your For You feed starts watching them — no settings page required.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
One self-published practitioner account with inspectable scripts but no measurements
The mechanical claims are strong because the artefacts are printed in full: the file-selection pathspec, the test-gathering grep, the git log invocation, the llm call, the output path and the CI steps can all be read and checked line by line. Everything about model behaviour is weak: the opening failure example names no model, prompt or repository, the claim that UNKNOWN is readily emitted is untested, and the 4,000-line-diff and weeks-of-survival figures are rhetorical. One publisher, no independent corroboration, no before/after data.
No adoption evidence in the supplied sources
The post describes a pattern the author advocates. There is no release, deployment, benchmark, usage disclosure, download count, repository, team, or before/after result reported — not even for the author's own codebase. Naming llm, claude-sonnet-4-5, doctest and pytest-codeblocks as dependencies is not evidence that this pass is running anywhere.
Mostly deflationary, but its behavioural laws outrun its evidence
The framing is anti-hype: it argues against pointing a model at a repo, prefers reformatting over reasoning, keeps output in a scratch directory and pushes verification into CI. That pulls the gap toward zero. It is nudged positive because the strongest assertions are stated as general laws about models — errors survive for weeks, models will emit UNKNOWN when allowed, no model would guess the revert rationale — on the strength of one undisclosed example, and because the shipped script is weaker than the doctrine it illustrates: commit subjects only, no migrations, no generated specs.
Practitioner blog post; light vendor pull, no disclosed affiliation
Publication is an individual dev.to post with no stated employer, sponsorship, product or affiliation, and it sells nothing: the tooling recommended is a paid model behind an open-source CLI plus stock pytest plugins. The residual pull is attention economics — a confident, prescriptive how-to headline benefits from strong general claims — and an incidental steer toward one commercial model endpoint.
Confident on the mechanics, low on the effect
High confidence that the script, prompt rules and CI gates say what the assessment says they say, since they are printed in full and internally consistent. Low confidence that the workflow delivers the improvement claimed: one publisher, no adoption signal, no measurement, an undisclosed example, and a demonstrated gap between the stated artefact inventory and what the script feeds the model.
build
Fable 5 at $50 per million output tokens turns model routing into a budget line2 distinct publishers
build
263 green tests, every value undefined: the schema mirror that agreed with itself1 distinct publisher
build
Your bearer token signs nothing: HMAC body signing is the gap in money-moving APIs1 distinct publisher
product
PayPal stopped saying no. Payments teams should now plan for a Stripe-owned checkout rail3 distinct publishers
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 27, 2026