Skip to content

Build1 publisher3 min readPublished

A 0.9 purity rule leaves the completion proxy with a smaller n than the row beside it

In one agent telemetry report, core process metrics were attributed to model epochs inside threads while the completion proxy was computed once per eligible main thread, so the same model label carried two different sample sizes.

The Engineer · Build desk

Illustration accompanying A 0.9 purity rule leaves the completion proxy with a smaller n than the row beside it

What happened

  • A telemetry report with medians, interquartile ranges, sample sizes and bootstrap intervals, its rows grouped by model, turned out to be several different studies sharing one table.
  • Core process metrics such as tool-error rate, re-edit rate and output tokens were attributed per assistant turn to the model epoch that produced it, so a multi-model thread produced several epoch rows.
  • Pre-relaunch and relaunch observations were split into separate model epochs and not pooled, even though both sat under the same display name.
  • The author gives the practical key for a row as at least model_epoch x role x analysis_unit, adding policy and task family only where those fields are actually observed.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • constraint Confidence intervals do not repair a mismatched row key: an epoch-attributed error rate and a thread-level completion proxy cannot stand in for tasks completed by a model, however many bootstrap resamples sit beside them.
  • cost Honest epoch boundaries cost measurable sample: every edit validated across a model switch drops out of the validation-sequence count on both sides, so boundary-heavy workloads undercount most.
  • exposure Any per-policy comparison over history rests only on sessions that were stamped with a policy hash at the time, because the UNKNOWN rows were left unfilled, with no inference from the current router.
  • decision When a metric is invalid across roles, the choice is between narrowing its valid population and footnoting the chart; this report narrowed, and lost the cross-role comparison outright.

Two sample sizes under one model label can both be correct. The completion proxy in this report ran once per thread, only for main threads whose model purity was at or above 0.9, and censored threads were dropped [2]. The core metrics were attributed per assistant turn, to the model epoch that produced that turn [1]. The proxy table therefore reports a smaller n for the same label, and neither number is an error [2].

The purity threshold filters hard. If purity counts assistant turns, a ten-turn thread admits one turn from another model before it falls out of the proxy population [2]. A thread that routed elsewhere twice still contributes to the core table while failing the purity rule for the other one [3]. Resampling across that boundary narrows the interval around a population you did not set out to study [18].

Role is the second axis. A main thread can carry repeated reads, edits, recovery and user turns; a sidechain is often a bounded delegated run [4]. A re-edit rate compared across those two populations mixes model behaviour with session structure, and the metric cannot distinguish productive iteration from repair [19]. One sidechain harness commonly ended on a tool_result line, in 92 of 99 threads in one recorded cell [5], about 93 percent [1]. A last-line heuristic scores that ending as incomplete even when the delegated work finished [6]. The report excluded sidechains from completion comparison, on the author's stated ground that a warning below a cross-role chart does not undo a denominator that never meant the same thing across roles [7].

For the 92-of-99 figure to transfer to your dashboard, your delegated runs would have to exit the way that harness exits. It describes a harness's last line. A harness that ends on a final assistant message would score the same finished work as complete [6].

Splitting epochs costs sample size on purpose. Sequence metrics were computed within each epoch fragment, so an edit performed by one model and validated after a model switch was credited as a within-epoch validation sequence for neither side [9]. Pooling the two epochs would produce a larger sample by erasing the boundary that explains why the sample is heterogeneous [10]. "A friendly model name is presentation. It is not a stable experimental identity," the author wrote [11].

Where dispatch policy was stamped at the source, rows were grouped by the policy hash; many historical sessions had no known policy version, stayed UNKNOWN, and were not backfilled from the current router [12]. Task family went unobserved entirely [13].

The linked repository, hexisteme/hard-gate-hooks, holds two MIT-licensed Stop-hook examples, their tests and a read-only scanner. The post calls that code adjacent implementation examples; the telemetry instrument it describes is not among them [14]. This is one author's account of one report [20]. Carrying the 0.9 threshold or the 92-of-99 ending into someone else's dashboard is extrapolation from a single instrument.

What to watch

  • Whether agent telemetry dashboards start exposing role and analysis_unit as columns instead of one model label.
  • Whether dispatch-policy hashes get stamped at source going forward, shrinking the UNKNOWN policy bucket instead of backfilling it.
  • Whether other sidechain harnesses also terminate on tool_result, which would show the 92-of-99 pattern is not specific to one implementation.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories