Loading today’s stories
Build1 distinct publisher3 min readUpdated
Progressive disclosure on a 20-tool agent saved about 30% of input tokens overall, but the per-task split shows cost stops being flat and starts tracking your traffic mix.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
A team switched on progressive disclosure for an agent with 20 registered tools, deferring the schemas until the model searches for a capability, and measured input tokens down about 30% and total cost down about 21% [1]. Then, according to the dev.to writeup, they split the same 80 runs by task type and the average came apart: three of the four tasks saved between 21% and 30%, while the fourth saved nothing on one transport and cost 12.3% more on the other [3].
The setup is small enough to argue with. Four business tasks, each needing exactly one tool, with all 20 schemas registered in both arms [5]. The always-on arm serialised every schema into every request; the deferred arm wrapped the same 20 in Pydantic AI's DeferredLoadingToolset, so the model had to search for a capability and load it before calling [2]. Twenty runs per cell, sequential, no retries, gpt-4o at temperature 0 with parallel_tool_calls disabled, run on 2026-08-06 against pydantic-ai-slim[openai] 2.24.0 [6]. Both Chat Completions and the Responses API were tested, because tool search executes server-side on one and through a local fallback on the other [7]. All 80 runs are published, and the whole experiment cost $0.279585, about $0.0035 a run [4][8][5].
Pooled, the result looks clean. Input tokens fell 30.6% on Chat Completions and 26.1% on Responses, cost fell 21.2% and 16.8%, and correctness held at 80 of 80 exact matches [10][11]. The tell is the standard deviation column: 10.3 and 10.6 tokens in the always-on cells against 190.8 and 263.1 in the deferred cells [9], which is 19 to 25 times more variable [4].
The author's first assumption was run-to-run nondeterminism, and reports that it was wrong [16]. Grouped by task inside each cell on the Responses API, always-on puts all four tasks within 22 tokens of each other, every one with a standard deviation of 0.0, because the 20 schemas dominate the prompt [13]. Under deferral the same four tasks spread from 807.0 to 1,431.8 mean input tokens, a gap of 625 tokens or 1.77x between cheapest and dearest [14][3]. Inventory-reorder fell 40.0% on input tokens, shipment-delay 38.8%, currency-conversion 30.7% [2]. Defect-threshold rose 5.0% [1], and it is also the only task with meaningful run-to-run spread under deferral, standard deviation 95.6 against 0.0 or 2.7 for the others [14]. Retrieval that is unstable and retrieval that is expensive appear to be the same failure here.
The reframing is the part worth keeping: deferral does not make cost noisy, it makes cost task-dependent, so the bill starts tracking traffic mix rather than tool count [15]. Always-on charges the same amount whatever the user asks [13].
So watch the mix before you quote a savings figure. If defect-threshold-shaped traffic is a tenth of your volume the mean survives; if it is most of your volume it does not, since that task type went backwards rather than flat [3]. Watch the transport separately, because the 12.3% regression showed up on one and a wash on the other [3], which is consistent with tool search running server-side in one path and locally in the other [7]. And watch the sample size: five runs per task per cell [13], one model, one library version [6], one 20-tool agent [5]. The numbers may not transfer. The method - group by task and by transport before believing the average - costs about $0.28 to repeat [8].
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
A team turned on progressive disclosure for an agent with 20 tools; deferring the tool schemas cut input tokens by about 30% and total cost by about 21%.
Two arms were compared: always-on, with all 20 schemas serialised into every request, and deferred, with all 20 wrapped in Pydantic AI's DeferredLoadingToolset so the model has to search for a capability and load it before calling it.
Three of the four task types saved 21% to 30%; the fourth saved nothing on one transport and cost 12.3% more on the other.
All 80 runs are public and the experiment can be reproduced offline for free.
The benchmark used four business tasks, each needing exactly one tool, with twenty tool schemas registered always.
Twenty runs per cell, sequential, no retries, gpt-4o at temperature 0 with parallel_tool_calls=False, run on 2026-08-06 against pydantic-ai-slim[openai]==2.24.0.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Transparent single-author benchmark with published raw runs
The quantitative core is unusually well instrumented for a blog post: pinned SDK version, model, temperature and sampling flags, 20 runs per cell across four cells, cell-level means and standard deviations, per-task means, per-task costs, 80/80 correctness, and a public raw JSONL bundle plus pasted real command output. What holds the score down is scale and independence: 5 runs per task per cell, one model, one provider, four synthetic single-tool tasks, no third-party replication, and an explicitly unrecorded mechanism for the outlier task.
One team's harness; no external adoption evidence
Adoption evidence is confined to the author's own use: a self-reported enablement of progressive disclosure on a 20-tool agent and one benchmark run of a public harness. There are no other deployments, no user counts, no vendor usage disclosures and no third-party replication in the supplied material, so this measures a single practitioner exercising an existing framework feature rather than diffusion.
Slightly understated: author debunks his own headline
The post's framing runs against its own promotional interest: the attractive 21% average is immediately qualified by a per-task breakdown showing one of four tasks got nothing or cost 12.3% more, and the author flags a failed hypothesis and an unrecorded mechanism. That earns a mildly negative gap. It is not more negative because the generalising conclusion that 'your bill now tracks your traffic mix' rests on four synthetic tasks, five runs each, and a single model.
Moderate: promotes the author's own benchmark harness
The write-up drives attention to a benchmarking harness repository operated by the author, which is an incentive to produce quotable percentage deltas. That incentive is partly neutralised by publishing the raw 80-run bundle and the exact reproduction commands, and by leading with a result that damages the cleaner headline. No vendor sponsorship, funding or commercial relationship with the framework or model provider is disclosed in the supplied source.
Moderate confidence, narrow scope
Internal consistency is high and the arithmetic is checkable against published raw runs, so the specific findings about this suite are trustworthy. Confidence is capped by the single-source, single-author, single-model design, small per-task samples, and the author's own admission that the bundle does not record which schemas search returned, leaving the outlier's cause unexplained.
build
Bedrock turns GPT-5.6 throughput into a routing choice, with residency as the price1 distinct publisher
leadership
Re-baseline AI procurement on cost per completed task, not dollars per million tokens1 distinct publisher
build
GPT-5.6 ships as three models, and that makes model choice a deployment decision1 distinct publisher
build
The Tokenizer Is Your Real Price List, Not the Per-Million Rate Card1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 18, 2026