Build1 distinct publisher3 min readUpdated
One unmatched component was scored compliant, another deviant, and neither had a rule behind it. The repair was a third verdict state and two coverage counts.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
A checker that always returns a verdict is not measuring compliance, it is filling a column. Part four of this audit compared a stated routing policy against what actually ran and reported 425 decisions, 96 deviations and $1,248.13 of spend, none of it on the main thread [1]; every row carried a verdict because the policy file's `default_tier` quietly absorbed anything no rule matched [3]. The check came from a reader. Brian Jin asked in the comments whether "no applicable rule" should be a first-class unresolved state rather than falling through to a default, on the grounds that it separates real compliance from policy coverage gaps [2]. On the frozen corpus, 424 of the 425 decisions matched an explicit rule and one did not [4]. The exception was a `product-manager` component that no rule in the file mentions; it inherited the default `frontier` expectation, ran Opus, and was scored compliant at $0.91 [5]. Replaying the same policy against the live window of 784 decisions, the fall-through population had doubled [6], to two rows [8]. The second was a `claude-code-guide` component that first appeared on July 30, matched nothing, inherited the same default `frontier` expectation, ran Haiku, and was flagged as a deviation at $0.08 [7]. Same silence, opposite signs. The author's read is that the fake compliant row is a missed catch, while the fake deviation is worse because it puts noise into the one table the exercise asks you to trust [11]. In that case the routing was correct: Haiku for a docs-lookup subagent is what he would have chosen, and the defect is the policy's silence, not the model [9]. Acting on the row would have meant "fixing" a correct decision [10]. The combined cost of both fabricated rows is $0.99 against a $1,248.13 window [23], which is the useful framing: the exposure is not the spend, it is the table. Note the direction of travel. The second fall-through arrived because a component type arrived [7]. As a share of decisions the rate barely moved, 0.24 percent to 0.26 percent [25], but the count doubled without anyone editing the policy, and it will do that again the next time a component ships ahead of a rule. Measuring the gap surfaced its mirror. The file has seven rules and one of them, `research-explore` outside the main thread to the cheap tier, matched zero decisions in both windows [12] - one rule in seven, or roughly 14 percent of the file, inert [14]. Not for lack of traffic: the frozen corpus contains 83 `research-explore` decisions, but every research-capable component has its own component-level rule earlier in the file and the specificity tie-break shadows the task rule every time [13]. The two coverage labels are Brian's, adopted after the change shipped [22]. The actual side fails the same way. When `claude-opus-5` entered the corpus with no entry in the tier table, 5,992 traces sat unscoreable for two weeks: not compliant, not deviant, absent [15]. Shipped: `Policy.match` returns no match instead of falling through, and a `verdict` column carries `compliant`, `deviation`, `unresolved:no_rule` and `unresolved:unknown_model` [16]. `default_tier` survives but now applies only to a rule that matched and omitted its expected tier [17]. Nine new tests, 437 passing [18]. The summary prints decisions out of coverage and rules that never matched, next to the verdict tallies [19]. On the current 789-decision corpus, `unknown_model` is zero, because the `claude-opus-5` tier went in two days before the verdict column and those 5,992 traces would all have landed there [20]. That is why the two classes are counted apart: `unknown_model` is an operational gap that self-clears once the tier table is filled, while `no_rule` rows sit until somebody writes a rule or rules the case out of scope [21]. Watch the `no_rule` count against new component types.
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
Part four of the series measured a stated routing policy against what actually ran: 425 decisions, 96 deviations, $1,248.13 of spend, none of the deviations on the main thread.
Reader Brian Jin asked in the comments: "Have you considered making 'no applicable rule' a first-class unresolved state rather than allowing it to fall through to a default? That seems useful for separating real compliance from policy coverage gaps."
The author's policy file has a default_tier key, and any decision that matched no rule quietly took it, so every decision received a verdict and the table had no gaps.
On the frozen corpus of the same 425 decisions from part four, 424 matched an explicit rule and one did not.
The single unmatched decision was a product-manager component that no rule in the policy file mentions; it took the default frontier expectation, ran Opus, and scored compliant at $0.91.
Replaying the same policy on the live window of 784 decisions, the fall-through population had doubled.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Precise, internally consistent self-measurement on one private corpus
The account is quantified at every step - 425 and 784 and 789 decisions, 424 matches, two fall-throughs at $0.91 and $0.08, 83 shadowed research-explore decisions, 5,992 unscoreable traces, nine new tests with 437 passing - and the derived arithmetic checks out against the stated figures. What holds the score down is provenance: every number comes from one author's own trace store, there is no replication, no external audit, and the mechanism claims (specificity tie-break shadowing, verdict fabrication) are demonstrated only on that corpus.
One maintainer, one corpus, no third-party deployment evidence
Adoption evidence is limited to the author's own use: a shipped change, coverage counts in his summary output, and figures drawn from his private corpus of up to 789 decisions. A one-line agent_lint entry point is offered to readers, but no supplied material shows anyone else running it. The only other practitioner signal is a commenter describing a related defect in his own orchestrator, which is not adoption of this tool.
Slightly understated against its own evidence
The framing stays at or below what the numbers support. The author volunteers that the fabricated rows total under a dollar, that the unknown_model count of zero is a timing artifact rather than a clean bill of health, that his (unit, component) grain would miss a commenter's retry defect, and that he has not fixed it because one private corpus is a poor basis for a schema change. The generalising closing line - if your checker has a fallback default it is producing verdicts nobody wrote a rule for - reaches beyond the single corpus, which keeps the gap near zero rather than strongly negative.
Mild self-promotion of the author's own tool, disclosed in line
The author is the maintainer of the traceguard checker the piece is about and closes by pointing readers to its entry point, so there is a visible interest in the tooling looking useful. That interest is disclosed rather than hidden, the piece leads with its own defect rather than a capability, credits a reader for the design and the naming, and no vendor sponsorship, funding, pricing, or commercial relationship appears anywhere in the supplied material.
High internal detail, single unverified publisher
Confidence is capped by structure: one publisher, one author, one corpus, and no contradicting or corroborating source in the cluster. Within that limit the reporting is unusually legible - specific counts, a named mechanism, an explicit test tally, and candid statements of scope - so the descriptive claims about what this checker did and now does are reliable, while any generalisation to other checkers or fleets rests on argument rather than measurement.
product
A 2x LLM bill is not a bug report: token spend is an observability problem1 distinct publisher
build
2,513 tool calls, zero refactorings: what agents actually do when you ask them to refactor1 distinct publisher
build
A Retention Policy for Agent Memory: Flag Unused Skills at 30 Days, Archive at 901 distinct publisher
build
Thirteen tasks green, then "give up (Recommended)" on the one that needed understanding1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 15, 2026