Skip to content

Build1 publisher2 min readPublished

A $4 rewrite of MCP tool descriptions moved Toolmetry's SQLite success from 34% to 100%

Toolmetry edited only the strings that tell an agent what each MCP tool does and how to call it. On a single published run, its three test servers finished between 96.4 and 100 percent for $4 of API spend.

The Engineer · Build desk

Illustration accompanying A $4 rewrite of MCP tool descriptions moved Toolmetry's SQLite success from 34% to 100%

What happened

  • Toolmetry, an open-source project that measures how well agents use MCP tools, had an LLM rewrite the descriptions on existing MCP servers, three strings per tool.
  • The model behind the agent stayed the same and so did the harness, so the descriptions were the only variable in the run.
  • The Memory server moved from 61.8 to 96.4 percent and the Git server from 75.0 to 96.7 percent in the same experiment.
  • Total API spend for the whole experiment was $4.

Compiled by The EngineerSomething wrong?How this is made

Why it matters

  • cost An hour of reading descriptions and under $5 of API spend is the cheapest thing a team can try before paying for a bigger model. The person who pays it maintains the MCP server, not the agent.
  • constraint The SQLite fix pins a five-table list inside a description, so whoever adds a table next has to edit prose to keep the agent calling correctly.
  • decision A low tool-call success rate now has to be diffed against your own descriptions before it is charged to the model, because none of the three named defects show up in a model benchmark.
  • precedent A per-server tool-call success number is cheap enough to run in CI. Once it is there, a description edit is a change that can regress a build.

Before the rewrite, the SQLite server's query tool carried this description: "Execute an SQL query against the database. First enumerate available tables using list_tables, then construct your query against those tables." Agents did what it said. They called list_tables before every query, including when the schema was already known from the previous call. Toolmetry counts that as two extra API calls per interaction and doubled latency for no benefit. The description was the only thing requiring that call. Specifically, the word "first".

The replacement string moves the answer into the description: "Execute an SQL query against the database. Available tables: users, orders, products, invoices, and audit_log. Use standard SQL syntax. This tool handles both reads and writes. No need to call list_tables first." The ritual calls stopped. The schema now lives in two places, which is one more than it lived in before. Add a table next quarter and that description is wrong in the way the git server's descriptions were wrong. Those referenced deprecated parameter names, agents passed the old names faithfully, and the calls failed. Toolmetry's stated rule is to update the description in the same commit as the API.

The other SQLite defect was a pair of tools. The read and write query tools were described almost identically, and most failures were the agent calling the read tool when it needed the write one.

The three servers gained at different rates. SQLite had 66.0 points of room to 100 and took all of it. Memory had 38.2 points and took 34.5, or 90 percent. Git had 25.0 points and took 21.7, or 87 percent. After the rewrite the three land within 3.6 points of each other, against a spread of 41.0 points before. Scores that converge that tightly near the ceiling usually mean the task set has been exhausted. The dev.to write-up does not give a trial count per server or name the model driving the agent. I would want both before repeating the 100 percent figure.

For 66 points to transfer, your server has to carry the defects Toolmetry found: two tools a careful human could confuse, a description that implies a prerequisite step, a parameter name the API has dropped. Descriptions that already match the API have nothing to give back. Finding out is the cheap half: list every tool, print each description, read them in pairs, and diff the parameter names against the current API. The write-up prices that at about an hour and under $5. On models it is blunt: "A cheaper model with broken descriptions still fails. An expensive model with broken descriptions still fails."

What to watch

  • Whether Toolmetry publishes trial counts per server and the model behind the agent. Those two numbers decide whether 34 percent was a stable baseline.
  • Whether the rewritten SQLite description survives a schema change that nobody propagates into the docstring.
  • Whether the same rewrite pays on servers with dozens of tools, where confusable pairs multiply faster than descriptions get read.
Loading claim ledger
Loading source directory links
Loading share composer
Loading topic controls
Loading related stories