Build1 publisher3 min readPublished
Three of four layers in a proposed MCP testing pyramid run as plain pytest
Three of the four layers in a dev.to testing pyramid for MCP servers are plain pytest checks on schemas, error envelopes and session expiry. Only the fourth puts a model in the loop, and the available text breaks off before describing it.
The Engineer · Build desk
What happened
- A dev.to post argues MCP servers have two kinds of client: deterministic harnesses such as CI, and the LLM, which picks when to call a tool, what to pass and how to read the reply.
- It splits testing into four questions: valid calls, safe failure on bad input, session and auth lifecycle, and whether a model given only the schema would call the tool correctly.
- The post names one test to keep if a team keeps only one: an expired session must return an explicit expired error, never stale cached data.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Error wording becomes part of the tested contract, because the assertions match substrings like timeout and expired; rewording a message for users breaks CI.
- exposure A server that falls back to cached data after a session expires hands the model stale figures it will report as current, and no crash or alert flags the fault.
- constraint Handlers that take parameters the model never supplies must be filtered out of the signature comparison, or the static check fails on every commit.
The case for an MCP-specific test layer rests on one sentence in the post. A tool description that is valid JSON Schema but ambiguous in plain English "will get called wrong in production even if every unit test passes," it says [2]. In that framing, the schema and its descriptions are the contract the model reasons over [2]. The support is experience. The post lists three failures: a drifted schema the model starts calling wrong, a token that expires mid-conversation and returns empty results, and a string return that becomes an object and breaks downstream agents [4].
Read the code and three of the four layers turn out to be deterministic pytest [15]. The expired-session test builds a session with ttl_seconds=1, sleeps two seconds, calls list_documents and asserts on is_error and the word "expired" [10]. No model takes part [15]. The MCP-specific part is where the assertions point: at the response envelope, not just the business logic [7]. The bad-input test checks that is_error is true, that the text names invoice_id, and that the body is never empty [7].
The static layer is a set comparison. It takes the keys under each tool's input_schema properties and compares them with the parameter names inspect.signature returns for the handler [5]. According to the post, this catches "the single most common regression: someone adds a required parameter to the handler and forgets to update the schema" [6]. It checks names only. Types go unchecked, and so does the return value, so the post's own string-to-object refactor passes this layer [13]. Catching that takes a separate assertion on output shape [13].
The best advice in the post is the least MCP-specific. Write one test per failure mode seen in production, such as timeouts, malformed upstream responses, rate limits and auth expiry, and turn each incident log entry into a regression test [8]. The session layer is the one the post says almost everyone skips, and the one it blames for "the scariest production bugs" [9]. I think the ordering is right for a team shipping a server with auth. These failures produce no crash to investigate, so the cheap deterministic checks have to find them [9]. The expired-session test waits out its own one-second TTL with two real seconds of time.sleep on every run [17]. An injected clock would get the same answer without the wait.
The fourth question asks whether a model given only the description and schema would call the tool correctly. It is the only layer that needs a non-deterministic client in the loop [3]. The post calls it the layer that is "genuinely new," and the available text ends mid-sentence at that point [12]. How to run that check, and how many sampled calls would count as a pass, is not in the available text [12].
What to watch
- The post's model-facing layer: how many sampled calls it runs per tool and what pass rate it treats as a merge gate.
- An output-shape check added to the static layer, closing the gap a string-to-object return change slips through.
- MCP SDKs shipping their own schema-to-handler consistency check, making the hand-written set comparison redundant.