Build1 publisher3 min readPublished
agent-inspect grades an agent release from traces already on disk
Three commands read persisted runs and emit a CI exit code, so no model call decides the merge. The cohort figures in the write-up are labelled synthetic, and the post never shows the threshold syntax behind the gate.
The Engineer · Build desk
What happened
- agent-inspect 6.17.6 splits an agent release check into three local layers: a suite over named cases, a baseline-versus-candidate cohort comparison, and a gate that emits a CI exit code with evidence.
- The sample refund suite names three cases and pins tools per case, with eligible-order requiring lookup_order and refund_order while unknown-order forbids refund_order outright.
- The cohort numbers in the post are labelled synthetic: ten baseline runs at 0% error and 820ms average against ten candidate runs at 10% error and 970ms.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The bill comes due in fixture work, not compute: every captured run needs cohort, scenario and model metadata written at inspection time, and someone has to keep a deliberately bad trace failing forever.
- constraint A wrong answer that fired the approved tools in the approved order still clears the gate, because the exit code fails only on structure. Domain judgement has to be paid for somewhere else.
- decision Teams adopting this write their own pass criteria without a published example to copy, so the thresholds behind a green merge get set locally and are only as reviewed as the local review makes them.
- capability A reviewer can rerun the verdict on the same stored evidence and get the same answer, and can reopen the failing case later without paying for another agent execution.
Take the cohort report at face value and it is a small table. Ten baseline runs at a 0% error rate against ten candidate runs at 10% is one failing run out of ten [12][1]. Ten runs per cohort is also the resolution limit: error rate can only move in 10-point steps [4]. Average duration goes from 820ms to 970ms, a difference of 150ms, about 18% [12][2]. To move a ten-run mean by 150ms, the runs have to add 1,500ms of wall time between them [3].
The post labels that result synthetic, so it is an illustration of the output format [12]. For numbers like these to mean anything on your service, the fixture set has to resemble your traffic and both cohorts have to be captured under comparable load. The author, writing on dev.to, states the limit himself: "The CLI can describe the recorded groups; it cannot make a small, biased fixture set representative." [13] The report does at least print the run count alongside the aggregates, which is how you can see that the 10% was one run [14].
The determinism comes from the read path. Suite, cohort and gate all read traces persisted in a directory, and none of them reruns the agent, invokes a model, or decides whether prose is semantically good [1][2]. Expectations live in the config: the unknown-order case pins runId to refund-unknown and sets forbidTools to refund_order, so the case fails if a refund fires on an order that does not exist [3]. Validation is its own command, separate from the run [4].
This design removes one failure: the case that never executed. If the trace for refund-unknown is absent, the case is skipped with a diagnostic instead of being treated as a pass, and an all-skipped suite cannot become a green release signal [5]. The post also asks for at least one known-bad trace or mutation that must fail [8]. The reason is stated plainly: "A check that has never rejected evidence and a check that silently stopped running can produce the same green icon." [9] The author opens on that same asymmetry: "I can make almost any agent change look good with one carefully chosen prompt." [18]
The gate example passes --suite, --format github, --output ./agent-gate-artifacts and --evidence, and it arrives only after the line about cases and thresholds reflecting actual release policy [17]. Then the trail stops: no threshold syntax appears anywhere in the post [20]. The exit code is deterministic given a policy, and the policy is yours to write.
Adopting this costs you capture discipline. Every run has to be labelled at the point of inspection, with metadata carrying cohort, scenario and model, into a traceDir [10]. The comparison then needs the cohort key and grouping passed explicitly, along with the metrics you want: errorRate, duration, toolChoice, observationFailure [11]. The suite is versioned beside the application, and expectations change in the same review as the behaviour [7]. The starting case list for a refund agent runs to six scenarios, including a missing-data path, a retryable dependency failure and a non-retryable tool failure [6].
The gate stops short of judging the answer. A structurally correct run can still answer badly, and a fluent answer can still hide an unauthorized mutation; the post says no single column of its review matrix can substitute for the other two, though the matrix rows do not appear in the text [19]. An extra tool call might be waste or a newly required authorization check, and a 20% duration improvement might be a better path or a skipped retrieval [15]. As the post puts it: "Structural metrics describe what changed. Product assertions and semantic evaluation decide whether the change is acceptable." [16]
What to watch
- Whether a later agent-inspect release documents threshold syntax, so the gate's pass criteria can be reviewed like the case list.
- Whether anyone publishes cohort deltas from captured production traffic instead of the synthetic ten-run example.
- Whether the review matrix referenced in the post appears with its rows, showing how structure, product assertions and semantic evaluation are scored separately.