Making docs cite a test at a known commit turned into a coverage audit

The maintainer of agent-cost, a small open-source CLI that reads local Claude Code and Codex CLI usage logs to estimate token cost, replaced hand-written documentation with a corpus of claims that each point at a specific file, test, or spec section, plus a content digest of that source at a specific commit ^[1][2][4]. Writing those pointers surfaced two behaviors that were correct in the source but asserted by no test ^[6], which is the more interesting result than the docs themselves.

The corpus is not large: 17 observations grouped into 5 topics, covering pricing-catalog validation rules, cache-write lower-bound behavior, unpriced-fact handling, decimal arithmetic for money, and the v1 contract of the `measure` command ^[3]. Each observation carries a claim kind, an epistemic status, and at least one provenance entry ^[4]. The forcing function, according to the author, is that an honest provenance entry for a behavior claim has to name the test that exercises it, so every claim required reading the test rather than the implementation ^[5].

Two claims failed that step. GAP-01 sits in `agent_cost/readers/claude.py`, in `parse_session_facts`: when a usage event's `cache_creation` field is a dict, the reader computes `leftover = cache_creation_input_tokens - (ephemeral_5m + ephemeral_1h)` and emits a `cache_write_unknown` fact if that is positive ^[9]. Tests covered a breakdown that exactly accounts for the total, and no breakdown at all, but not the reachable middle case where the dict is present and the two counters sum to less than the total ^[10]. GAP-02 sits in `agent_cost/aggregate.py`, where `_STATUS_RANK` and `build_rows` are meant to give a row the worst pricing status among its facts, ordered unpriced (0) < lower_bound (1) < priced (2) ^[11]. The existing test mixed an unpriced fact with a priced one; nothing built a row from a lower_bound fact and a priced fact to confirm the row lands on lower_bound ^[12]. Both went into `docs/claims/gaps.yaml` and became scoped pull requests adding regression tests with zero implementation changes ^[7].

Neither was a live bug, and the author is explicit that this is the point: the suite already covered both extremes of each piece of logic, and what was missing was the middle case that is easy to reason past while reading code ^[8]. The author's counterfactual is the load-bearing part of the argument. If a later refactor reordered `_STATUS_RANK` or changed the leftover math, CI would have stayed green while the cost output quietly under- or over-reported cost floors ^[13].

Worth noting what was already in place: a README section titled "What this measures, and what it doesn't", and a `price_fact()` docstring explaining the lower-bound design decision ^[14]. Prose did not fail because it was sloppy. It failed, on the author's account, because writing a sentence never generates the question "is there a test for the partial case", and because a reader cannot tell verified sentences from best guesses when every claim carries the same visual weight ^[15][16]. Two of 17 observations produced a coverage gap, roughly 12 percent ^[1].

This is one repository and one author, so treat the rate as an anecdote rather than a base rate. The things to watch are whether the content digests actually catch a stale pointer on a real refactor, and whether the gap rate falls as the corpus grows or stays flat because new code arrives faster than claims do.