Green Tests, Useless Server: MCP Needs Task-Level Evals, And Call Count Moves First

An engineer writing on dev.to has put a usable name to the gap between a green MCP integration suite and a server an agent can actually work with: the MCP eval, defined as a realistic task a model must complete using only your server's tools, rather than an assertion about one call ^[1]. The reason to care is that the two artefacts answer different questions. A test asks whether the call worked; an eval asks whether an agent could get the right answer ^[2].

The structural argument is the strongest part of the post. When you write a test, you have already made the choice the model has to make: you picked the tool and you picked the arguments ^[3]. A test looks like an assertion that `callTool("list_issues", { status: "open" })` returns a non-empty list ^[4]. An eval looks like a question phrased the way a user would ask it, with a call budget and an expectation attached, for example: which open issue in the billing project has been waiting longest and who is it assigned to, budget four tool calls, expected output names a specific issue and an assignee ^[5]. Nothing in that eval names a tool, because choosing the tool is the thing under test ^[6]. The model gets the tool list, the descriptions and the schemas, and nothing else ^[7].

The failure mode the author describes is worth reading twice. A docs server exposes `search` and `get_page`; a user asks which regions a product supports; the agent searches, gets five ranked results, opens the first, and finds two regions mentioned in an example snippet, then answers "us-east and eu-west" ^[8]. The real answer was eleven regions, on the page ranked fourth ^[9]. That is roughly 18 percent of the correct answer ^[1], produced with every call returning 200, valid JSON, no timeouts, clean logs ^[10]. There is no exception to catch and no status code to alert on; the only way to see it is to check the answer ^[11].

Hence the argument for four outcomes instead of pass/fail: pass, wrong answer, too many calls, and untestable, with the last one not counted as a failure ^[12]. The operationally interesting one is call count. Eight calls against a four-call budget, twice the allowance ^[2], is described as the tool-description signal and the one to track over time, because call count degrades before pass rate does ^[13][14]. Untestable runs are the agent guessing, and each of those calls costs latency, tokens and metered API spend ^[15].

Two caveats, both from the source. Evals are not deterministic, and pretending otherwise produces a number you cannot trust ^[16]. And the author has an interest here: they have spent the past few months building an eval engine for arbitrary MCP servers ^[17].

The spec point lands cleanly regardless. The MCP specification says nothing about usability, and the author's position is that it should not, because conformance is not usability; the 2026-07-28 release tightened the protocol considerably and changed none of this ^[18].

What to watch, if you run an agent-facing server: mean calls per completed task, tracked per release rather than per incident, and the share of runs that blow their budget. Both should move on a tool-description edit before any pass-rate dashboard notices ^[13][14]. Also watch what testing still earns its keep for. The post is explicit that all three layers are worth having, and that testing is faster and cheaper than evals for protocol and correctness bugs ^[19].