Loading today’s stories
Build1 distinct publisher3 min readUpdated
A proxy on the MCP stdio pipe caught one client killing most of its own tool calls before they reached the server. On the wire it looked exactly like a weak model.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
A proxy placed on the MCP stdio pipe for 90 trials produced a token table, and then produced something more useful: on the client version the run replaced, tool calls were failing inside the client before a byte reached the server, and the wire record looked exactly like a model that tried very little and answered wrong [2][3]. If you are scoring agents, that is the gap between "this model is weak at browser tasks" and "this client shipped a validator with one meta-schema missing".
The run itself is modest and says so. Three servers, filesystem, playwright and github, five scripted tasks each, three trials per task per client, suite version 1.0.1, 90 trials and 87 successes [4]. Servers were pinned, including a github-mcp-server container launched untagged that reported itself as v1.9.0 [5]. Clients were Claude Code 2.1.235 on claude-sonnet-5 and Gemini CLI 0.55.1 on gemini-2.5-flash, with model ids read back out of each trial's own client JSON rather than assumed from the flag [6]. Claude Code 45 of 45, Gemini CLI 42 of 45, held as counts and never rates, and the author is explicit that three failures do not establish that one client is more reliable [8]. The three Gemini failures are three unrelated things: an off-by-one line count, a response the client's own tool layer rejected after six calls had already reached the server, and a final message that declared a task done without restating the price the check looks for [9].
The same instrumentation shows why per-call cost is not a server property. On GH-01 both clients called get_file_contents with identical arguments, and the result measured 1,561 tokens for Claude Code against 161 for Gemini CLI, roughly ten times the payload for the same request [12][1]. The difference is a _meta serverInfo block carrying two PNG icons inlined as base64, 2,215 characters wrapped around a 472-character answer, about 4.7 characters of wrapper per character of content [13][2]. All 81 github responses on Claude Code carried it, none of the 93 on Gemini CLI did, and the two sessions had negotiated different protocol revisions [14][15].
The diagnostic case comes from the shakedown hours earlier, on Gemini CLI 0.18.4, kept as superseded records rather than folded into the 90 [16]. All five playwright tasks failed there, each trial with exactly one call on the wire [17]. That client validated arguments against the schema the server advertised, and its bundled validator had no JSON Schema draft 2020-12 meta-schema registered [19]. @playwright/[email protected] declares 2020-12 on all 24 of its tools, so most calls died inside the client and never reached the pipe: one onto the wire per trial, three or four lost internally, which is 75 to 80 percent of intended calls never arriving [20][3]. The same client ran filesystem and github clean the same day, because those declare draft-07 and no $schema at all [21].
Note the honest limit. Read from the frames alone, the classifier called this a capability failure, and once a decline because the model apologised [18]. Wire logging gave the impossible signal, one call per trial; the client's own error surface gave the cause.
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
The finding the author considers most important came from a shakedown earlier the same day, on the client version the run then replaced: one client was failing calls inside itself, before a byte reached the server, and on the wire that looked exactly like a model that tried very little and answered wrong.
Read from the frames alone, the classifier bucketed the playwright shakedown failures as a capability failure, once as a decline because the model apologised, and the author states that reading is false.
Gemini CLI 0.18.4 validated each call's arguments against the schema the server advertised, and its bundled validator had no JSON Schema draft 2020-12 meta-schema registered.
@playwright/[email protected] declares draft 2020-12 on all 24 of its tools, so most of its calls died inside the client with 'no schema with key or ref "https://json-schema.org/draft/2020-12/schema"' and never reached the pipe: each trial got exactly one call onto the wire and lost three or four more inside the client.
The author's previous post, on 2026-08-18, measured what 14 MCP servers cost a context window before an agent does any work, and announced Tier 2 as real clients, real tasks, every frame logged.
Tier 2 ran as 90 trials across three servers, two clients and fifteen scripted tasks, three trials each, through a proxy on the MCP stdio pipe.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Detailed single-author instrumentation, unreplicated
The account is unusually specific for a single-source post: pinned server and client versions, model ids read back from per-trial client JSON, a named tokeniser basis, per-response counts (81 of 81 versus 0 of 93), the verbatim validator error string, negotiated protocol revisions, retained superseded shakedown records, and a cited upstream issue and PR for the fix. The author also bounds his own inferences, refusing rates and refusing to claim the server keys on the protocol revision. What is missing is external verification: one publisher, one operator, three trials per cell, no independent re-run, and the token table referenced in the text is not shown in the supplied body.
Ecosystem artifacts real, harness use is n=1
The components exercised are shipping, widely used software with concrete version evidence, and the underlying defect had already been fixed and released upstream in Gemini CLI 0.28.0 with an issue and PR trail, which is genuine ecosystem adoption of the remedy. Adoption of the finding itself is minimal: the proxy harness, suite 1.0.1 and the tool-call gap detector are the author's own, used in one run by one person, with no evidence in the supplied source of any other team adopting the method or of how many production deployments run affected client versions.
Slightly understated relative to its own evidence
The framing is restrained rather than inflated. The author caps his conclusions below what the data would tempt: counts never rates per spec 3.3, an explicit statement that three failures do not rank client reliability, a refusal to publish a per-server price, and a refusal to conclude the server keys on the negotiated protocol revision from one server version. He also names himself as the cause of the stale client and points to the upstream fix that predated his run. The generalisable result, that wire-side evidence can silently misattribute client-internal call loss to model incapability, is arguably more consequential than the modest presentation suggests, which is why the gap sits below zero. It is not strongly negative because the underlying sample is small and unreplicated, so a louder claim would not have been justified either.
Self-published series promotion, no vendor stake disclosed
The visible incentive is audience-building for the author's own measurement series: the post opens by referencing his prior Tier 1 piece and frames this as the promised Tier 2, on a developer platform where such series accrue reputation. Against that, the supplied source shows no vendor affiliation, no product being sold, no sponsored tool, and behaviour that runs contrary to promotional incentives, including attributing the defect to his own stale client, crediting the upstream fix that already existed, disclosing an untagged container and a side model call, and withholding the comparative rates that would have driven sharper engagement. Incentive pressure is therefore present but low.
Mechanism solid, magnitudes provisional
Confidence is moderate. The causal mechanism, a client-side validator lacking a draft 2020-12 meta-schema discarding calls to a server that declares that dialect on all 24 tools, is supported by a named error string, a control condition in which the same client ran draft-07 and no-$schema servers cleanly, and a post-upgrade re-run that passed with a zero gap. That chain is hard to explain another way. The magnitudes are weaker: three trials per cell, a single shakedown trial per task, one operator, one publisher, no replication, and the 75 to 80 percent loss figure rests on a stated three-or-four range rather than an exact count. Conclusions about the mechanism deserve more trust than any specific number here.
product
A 2x LLM bill is not a bug report: token spend is an observability problem1 distinct publisher
science
OX Security says MCP command execution is a design choice, so server owners own the risk1 distinct publisher
build
Claude Code's new default is a confession: the approval prompt was never a control1 distinct publisher
build
Your MCP tool definitions cost 64% more than your token counter says1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 21, 2026