Build1 publisher3 min readPublished
Why one team skipped 2,284 of 2,490 possible test runs, already settled by deterministic assertions
agent-tooltrust ran 206 live model calls instead of 2,490 and reports identical coverage, on the argument that the only thing a real LLM adds is proof that each framework adapter can surface all four verdicts.
The Engineer · Build desk
What happened
- The field test for agent-tooltrust, an open-source gate for AI agent tool calls, ran 206 live model calls where the full 83-agent by 30-scenario cross product would have been 2,490 runs.
- The 2,284 skipped runs would have used a local 4B model to re-prove behaviour that 2,490 deterministic assertions, made with zero LLM calls, had already exercised across every decision path.
- The reduced design keeps two guarantees: every scenario exercised by at least one real agent across 10 frameworks and 5 agent classes, and every decision type hit at least once per framework.
- Seven live runs failed, all of them because the model never called the guarded tool, and none because the engine returned the wrong verdict.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint The equal-coverage claim only holds where the gate's engine is framework-agnostic; a team whose adapters carry their own verdict logic cannot drop 91.7 percent of the matrix and claim the same proof.
- decision Cutting a matrix this far forces a team to classify failures first, because the author says conflating a model that never called the tool with a gate returning the wrong verdict ships either a flaky gate or a silently loose one.
- capability The author credits this design with making the field test runnable at all, so adapter behaviour got checked in an afternoon instead of waiting on a twelve-day serial matrix.
The cost estimate turns on the latency range. The post puts each live run at 30 to 80 seconds, and the full matrix at about 2.7 hours on 10 workers [2][3]. Work that backwards and it implies 39 seconds a call, near the bottom of the stated range [3]. At the top of the range the same matrix is 55 hours of model time, or 5.5 hours of wall clock on 10 workers [2]. The twelve days quoted for a serial run only appears once you also apply the 4-5x debugging multiplier: 55 hours times five is about 11.5 days [4][5][4]. The 206 runs, by the same arithmetic, are 1.7 to 4.6 hours serial, or 10 to 27 minutes on 10 workers [5].
Behind the gate, 2,490 deterministic assertions had already exercised every decision path, with zero LLM calls [9]. The assertion count and the cross-product count are the same number, and the post does not say whether that is by construction. That left the live runs one job: show that each framework adapter surfaces allow, audit, escalate and deny inside a real agent loop [10]. The two run counts the author calls defensible, 83 for breadth and 123 for depth, add up to the 206 that ran [15][1].
Equivalence rests on one assumption, stated plainly in the post: "If a scenario is going to behave differently under a real model, some agent in the 83 is going to hit it" [19]. That is a belief about the population of agents, not a measurement of it. For the roughly 12x cut to transfer to another suite, two things have to hold: the engine has to be genuinely framework-agnostic, and a scenario's behaviour under a real model has to be independent of which agent drives it [12]. If any verdict path differs per adapter, the dropped cells are the ones that would have caught it.
The failure data leans against the second condition. All seven failures were not-available, the model declining to call the guarded tool [16], and the explanation offered is about the loop rather than the gate. "A mock agent always calls the tool it's told to. A real 4B, given five tools at once, sometimes just answers in prose," the developer of agent-tooltrust wrote [17]. If tool-calling depends on how many tools sit in front of the model, the agent wrapping a scenario is part of the test. Seven failures in 206 live runs is 3.4 percent [6].
The post is blunt about why the cross product keeps getting picked: 83 x 30 is a number you can defend in a room without explaining covering problems [21]. "83 and 123 are both numbers I could defend in a review. 2,490 was a number I was too embarrassed to defend, which is how I knew it was the wrong one," the author wrote [15].
The scenario-to-agent mapping is published in the v0.1.1 field test report [20].
What to watch
- A scenario in the published v0.1.1 mapping touched by only one agent class is where the breadth plan is thinnest.
- A later run that returns an unexpected-decision failure would break the framework-agnostic premise the cut rests on.
- Swapping the local 4B for a larger model on the same depth runs would show whether the seven not-available failures were a function of model size.