Skip to content

Build1 publisher3 min readPublished

A compliance checker with a default tier manufactures verdicts in both directions

One unmatched component was scored compliant, another deviant, and neither had a rule behind it. The repair was a third verdict state and two coverage counts.

The Engineer · Build desk

Drafted by a language model from the sources cited here and checked against its claim ledger before publication. How we use AISend a correction

Illustration accompanying A compliance checker with a default tier manufactures verdicts in both directions
Generated illustration

What happened

  • Part four of the series measured a stated routing policy against what actually ran: 425 decisions, 96 deviations, $1,248.13 of spend, none of the deviations on the main thread.
  • Reader Brian Jin asked in the comments: "Have you considered making 'no applicable rule' a first-class unresolved state rather than allowing it to fall through to a default? That seems useful for separating real compliance from policy coverage gaps."
  • The author's policy file has a default_tier key, and any decision that matched no rule quietly took it, so every decision received a verdict and the table had no gaps.
  • On the frozen corpus of the same 425 decisions from part four, 424 matched an explicit rule and one did not.
  • The single unmatched decision was a product-manager component that no rule in the policy file mentions; it took the default frontier expectation, ran Opus, and scored compliant at $0.91.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

A checker that always returns a verdict is not measuring compliance, it is filling a column. Part four of this audit compared a stated routing policy against what actually ran and reported 425 decisions, 96 deviations and $1,248.13 of spend, none of it on the main thread [1]; every row carried a verdict because the policy file's `default_tier` quietly absorbed anything no rule matched [3]. The check came from a reader. Brian Jin asked in the comments whether "no applicable rule" should be a first-class unresolved state rather than falling through to a default, on the grounds that it separates real compliance from policy coverage gaps [2]. On the frozen corpus, 424 of the 425 decisions matched an explicit rule and one did not [4]. The exception was a `product-manager` component that no rule in the file mentions; it inherited the default `frontier` expectation, ran Opus, and was scored compliant at $0.91 [5]. Replaying the same policy against the live window of 784 decisions, the fall-through population had doubled [6], to two rows [8]. The second was a `claude-code-guide` component that first appeared on July 30, matched nothing, inherited the same default `frontier` expectation, ran Haiku, and was flagged as a deviation at $0.08 [7]. Same silence, opposite signs. The author's read is that the fake compliant row is a missed catch, while the fake deviation is worse because it puts noise into the one table the exercise asks you to trust [11]. In that case the routing was correct: Haiku for a docs-lookup subagent is what he would have chosen, and the defect is the policy's silence, not the model [9]. Acting on the row would have meant "fixing" a correct decision [10]. The combined cost of both fabricated rows is $0.99 against a $1,248.13 window [23], which is the useful framing: the exposure is not the spend, it is the table. Note the direction of travel. The second fall-through arrived because a component type arrived [7]. As a share of decisions the rate barely moved, 0.24 percent to 0.26 percent [25], but the count doubled without anyone editing the policy, and it will do that again the next time a component ships ahead of a rule. Measuring the gap surfaced its mirror. The file has seven rules and one of them, `research-explore` outside the main thread to the cheap tier, matched zero decisions in both windows [12] - one rule in seven, or roughly 14 percent of the file, inert [14]. Not for lack of traffic: the frozen corpus contains 83 `research-explore` decisions, but every research-capable component has its own component-level rule earlier in the file and the specificity tie-break shadows the task rule every time [13]. The two coverage labels are Brian's, adopted after the change shipped [22]. The actual side fails the same way. When `claude-opus-5` entered the corpus with no entry in the tier table, 5,992 traces sat unscoreable for two weeks: not compliant, not deviant, absent [15]. Shipped: `Policy.match` returns no match instead of falling through, and a `verdict` column carries `compliant`, `deviation`, `unresolved:no_rule` and `unresolved:unknown_model` [16]. `default_tier` survives but now applies only to a rule that matched and omitted its expected tier [17]. Nine new tests, 437 passing [18]. The summary prints decisions out of coverage and rules that never matched, next to the verdict tallies [19]. On the current 789-decision corpus, `unknown_model` is zero, because the `claude-opus-5` tier went in two days before the verdict column and those 5,992 traces would all have landed there [20]. That is why the two classes are counted apart: `unknown_model` is an operational gap that self-clears once the tier table is filled, while `no_rule` rows sit until somebody writes a rule or rules the case out of scope [21]. Watch the `no_rule` count against new component types.

Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories