Build1 distinct publisher3 min readPublished
A dev.to workflow puts AI-drafted docstrings behind three CI gates. Only one of them executes anything. The shipped name check looks in the direction that cannot catch a docstring for a function that was deleted.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
Read `documented_names` against the failure the post opens with. It calls `public_functions(path)`, which walks the AST and keeps every `FunctionDef` and `AsyncFunctionDef` name that does not start with an underscore [7]. It then returns the subset of those names for which the file's own text contains the name wrapped in backticks [6]. The result is a subset of the code's public functions by construction, so `missing = funcs - documented` can only ever list names that exist [15]. The check runs in one direction only, flagging code that nothing mentions; it has no way to flag a mention that no code backs. The docstring describing a function renamed or deleted in the last commit, which is the exact failure the post uses to motivate the workflow [1], passes this gate untouched [15].
That leaves the doctest run holding the load. Of the three gates proposed, one executes the documented behavior and two test for the presence of text [16]: pydocstyle or a small AST script asserts a docstring exists [3], the coverage script asserts a name appears [6], and `python -m doctest module.py` is the only step that runs anything [4]. A docstring with a summary, an Args section and no `>>>` lines clears both presence gates and hands doctest nothing to execute [17]. It can also clear the name check by mentioning itself. The author is straight about this and calls the coverage script a heuristic rather than a proof, on the grounds that a docstring can name a function and still be wrong [8].
For the executing gate to transfer, your setup has to match a couple of conditions this one meets. Your examples need stable output, because the sample asserts that `format_bytes(1024)` renders exactly `'1.0 KiB'` [9], which is fine for a formatting helper and painful for anything returning a dict, a float, or an object with a default repr. And your language needs an example runner at all; doctest is what makes this cheap in Python [4], and I would not expect the shape to survive a move to a stack with no equivalent.
The economics arrive with a disclosure. The post says it was prepared as part of MonkeyCode's product outreach [14], and the nightly drafting job leans on that project's advertised free access, currently stated as 10 million tokens with the README named as the authority [13]. No per-function token figure appears anywhere, so the cost of scanning every new public function every night is yours to measure before you plan a budget around a free tier.
If I adopted this, I would keep the first two gates and rewrite the third to run the other way: pull the backticked names out of each docstring, resolve them against the module namespace, and fail on the ones that no longer resolve. That is a short loop over `getattr`, and it catches the case the article describes while the shipped script cannot [15]. The division of labor holds up regardless. Drafting is repetitive and low risk, but reviewing the draft still demands real scrutiny [18].
Ranked by verification strength, evidence, and original report placement.
MonkeyCode is described as an open-source project providing free model access, currently advertised as 10 million tokens with readers told to check the official README for the latest quota, plus a free server option useful for running a nightly script that sends new function signatures to the model for a draft.
The dev.to post argues that AI assistants can produce a polished docstring describing a function that no longer exists, or an example that fails on the first run, and that the more docs are generated the staler they become if nothing checks them.
The post's diagnosis is the absence of a feedback loop: code has compilers, linters and tests, while documentation has only the cursor and the reader's patience.
Gate one: every public function should have a docstring, enforceable with pydocstyle or a tiny AST script, and CI fails if a new function lands without one.
Gate two: docstring examples must be executable, with Python's doctest named as the classic tool; running python -m doctest module.py turns the examples into tests, and if the function changes behavior the docs fail loudly.
Gate three: a CI script compares the set of public names defined in the code with the set of names mentioned in the documentation, so that a changed signature is flagged.
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 31, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
An AI test suite hit 94% coverage and missed the one branch that mattered1 distinct publisher
build
Your 90% Cache Hit Ratio Is a Lagging Indicator. Alert on Cold Misses Per Key1 distinct publisher
build
SSE promises framing, not JSON: the streaming bug that only appears on long answers1 distinct publisher
build
Sequence-level equivalence catches the cache a single-call test suite waves through1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Auditable code, unreproduced workflow
The strongest thing this story has going for it is that the author showed the work: the doctest block and the whole of check_doc_coverage.py are on the page, which is why the sharp finding — that the set difference cannot flag a docstring whose function is gone — can be verified line by line rather than argued about. Everything beyond the listings is assertion. No repository, no before-and-after drift numbers, no second account of anyone running this pipeline.
No usage signal
Nobody in this reporting says the workflow is running anywhere. There are no repositories using the gates, no team accounts, no download or install figures, and MonkeyCode's 10 million token line is an advertised quota rather than a measure of anyone consuming it. We would rather say nothing than infer traction from a code sample.
Three gates, one executes
'Test-first documentation' and 'three test gates' promise verification, and the arithmetic underneath delivers less: a docstring with no example clears presence and name-mention untouched, doctest is the only step that runs anything, and the checker shipped to catch drift looks in the direction that cannot see the stale docstring the piece opens with. The author does hedge — heuristic, not proof — which keeps this a gap in framing rather than a false claim.
Disclosed vendor placement
The piece says outright that it was prepared as part of MonkeyCode's product outreach, and the workflow it recommends happens to need exactly what MonkeyCode gives away: batch model calls on a nightly schedule against a free server. Disclosure earns real credit here, and it does not change the shape of the arrangement — the quota figure comes from the sponsor, and the reader is sent to the sponsor's README to confirm it.
Firm on the code, thin on the rest
We are close to certain about the mechanics, because they were published verbatim and the structural conclusions follow from reading them. We are much less sure whether any of this holds up in a working repository, how many docstrings would carry executable examples in practice, or whether the free quota survives contact with a production pipeline. One sponsored post is a narrow base for anything but the code review.