A SKILL.md layer quietly rerouted an agent off the MCP tools it was given

Count the outcomes before the anecdote. Four of the six paired runs came out the same with the Skill as without: evals 1 through 3 showed little difference, and eval 6, the time-sensitive arrival case, produced an identical result from the same MCP capability ^[6][9]. One run got worse, with the Skill changing route selection to a less optimal route ^[7]. One got better ^[8]. Net across the suite: one improvement, one regression, four ties, in exchange for another artifact to keep in sync with the server ^[14].

The single win is narrower than the tally suggests. It landed on the request where the user had explicitly asked to minimize transfers ^[8], so the guidance helped the agent honour a constraint the prompt already carried. On the three request classes the server was built for (station-to-station routing, attraction and landmark routing, time-sensitive arrival planning), the tools handled the work unaided ^[10]. The guidance was competing against a baseline that already worked.

Then the interesting failure. According to the developer's dev.to writeup, with the Skill in place the agent sometimes preferred web search over the MCP tools it had been handed ^[3], and nothing crashed ^[4]. For a transit server that is a substitution of source: the answer keeps its shape, while whatever the server guaranteed about fares and live data is quietly not what produced it. The post does not claim to know the cause, and the author's next step is Google ADK-based tracing precisely because comparing final outputs does not see tool selection closely enough ^[13].

That is the part that generalises past Tokyo trains. A tool has a signature and a return value, so you can pin it with a fixture. SKILL.md has neither. It edits the agent's priors about when the tools are worth calling at all, so its blast radius is every request the server handles, and its failure mode is a plausible answer obtained the wrong way ^[3][4]. Tool-level tests pass through all of that untouched.

So the assertion has to move. A regression suite for a guidance artifact has to check the path: which tool was called, how many transfers the chosen route had, what fare came back, whether the model left the server at all. That is close to what this paired evaluation already tracked, tool usage, route selection, transfers, travel time, fares and output quality ^[5], and the reason it caught anything is that the same request ran twice with the layer as the only variable. Without that harness, adding the Skill is an untested change to every code path the server owns.

The author's stated conclusion is the modest one: adding guidance is not automatically an improvement, and its value depends on the task, the underlying tools, and how the Skill itself is defined ^[11]. His numbers support a sharper corollary. On this suite, at this stage, the layer has not earned its place, and the only reason anyone can say so is a side-by-side run that the server's own tests would never have prompted.