A go/no-go rubric for MCP servers scores agent behavior separately from protocol tests

The retry case in the post is the one I'd put in front of a release review. An agent retries a call that already partially succeeded ^[2]. Part of the write landed, the agent saw a failure, and it called again. The task ends with a success message and two records. Nobody ordered the second one. A scorer that reads only the final response misses that duplicate, and it misses silent failures for the same reason ^[3]. Protocol tests never get to this state. A tools/list schema check or an initialize handshake has no agent in the loop to retry anything ^[1].

According to the author, treating the two surfaces as one is "the single most common mistake teams make" when they call a server tested ^[14]. I'd sequence the review the same way the post does. Agent evals run on a shaky protocol layer give noisy, non-repeatable results, the post says, so the review stops at Tier 1 until that tier passes ^[5].

Each tier tolerates one failed row. A server can clear both tiers with two failures out of twelve ^[13]. Rows score 0 or 1 with equal weight ^[6]. The row that checks end state counts the same as any other, and it can be one of the two a passing server fails. For servers whose tools create records or move money, I would make the state row a veto by itself. For a read-only server the flat score is fine.

A Tier 2 score measures one agent on one task set. The post's own example, an agent choosing between five similar-sounding tools ^[2], depends on which model is choosing and what else sits in its context. For a 5/6 to hold in production, the eval agent has to be the production model, seeing the production tool list, on tasks shaped like production traffic. The post allows a simulated agent ^[6]. Each step away from the production model weakens the transfer.

The agents are non-deterministic ^[2]. One run per row is a sample of one. The post's repeatability advice targets drift between deploys: fixed task sets, versioned scorers, a diff after each release ^[10]. It does not say how many runs turn a row into a 1.

The strongest part of the post is about governors, and I agree with it. A server with no budgets, quotas or side-effect limits leaves the defensive work to the orchestrator. The author wants that written down: such a server is "not production-complete on its own" ^[9].

The tiers, thresholds and gate order are all in the free post ^[5][6]. The author also sells a scored version as reusable YAML task definitions with a minimal harness scaffold ^[11]. The post says it was written with AI assistance and reviewed for accuracy ^[12].