Build1 publisher2 min readPublished
Claude Code's plugin eval spends six agent runs per case to measure a plugin's lift
Anthropic's plugin eval command runs every case with the plugin loaded and again without it and prints the delta, though the CI threshold it documents still gates on each case's absolute score.
The Engineer · Build desk
What happened
- Claude Code's plugin eval command runs an eval suite twice, once with the plugin loaded and once without, and reports WITH, W/OUT and a delta that is the with-arm score minus the without-arm score.
- Cases live in an evals/ directory beside the plugin manifest, and each one is a realistic user prompt plus graders chosen from regex, tool_used, tool_order, file_exists, llm and baseline.
- Every case runs three times with the plugin and three times without it by default, six agent runs for a single case, because the documentation treats one agent run as noise.
- The CI example in the docs passes --threshold 0.8, --model claude-sonnet-5, --judge-model claude-haiku-4-5, --no-publish and --max-cost-usd 20, and the run exits 1 when a case falls below the threshold.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint The grader that proves your skill fired is removed from both arms' totals, so the score a CI threshold reads cannot register a trigger failure, and a human has to open the report to find one.
- cost Each llm or baseline grader multiplies against six runs a case, so the eval bill grows with suite size and lands on whoever owns the CI budget.
- contradiction A green build at 0.8 can still ship a skill that changed nothing, because a case may score high in both arms. That outcome is exactly what the delta was added to expose.
The first result most skill authors get is a delta near zero with a failing `tool_used: Skill` grader, according to a dev.to writeup of Anthropic's plugin eval documentation [11]. Claude never picked the skill on natural phrasing. The change goes in the skill's `description`, then you re-run and compare [11]. That loop only works because the prompt body is written as a request a user would actually type, with the skill left unnamed [15].
Six runs for one case is the default [7]. A ten-case suite is therefore 60 agent runs [22]. The drafting shortcut is `--runs 1 --ablation none`: one run instead of six, and no comparison arm [14][21]. The documented reason for three runs per arm is that one agent run is noise [14]. Whether three is enough depends on how far your own cases move between runs, and the default is a claim about agent variance in general, not a measurement of your suite. The example in the docs sets `--runs` to 1 for cheap iteration [14].
Four grader types are free, because `regex`, `tool_used`, `tool_order` and `file_exists` only read the transcript and the filesystem [13]. `llm` and `baseline` call a judge model and land in the run's list-price estimate [13]. The recommended CI line answers that by pinning `--model claude-sonnet-5` for the agent, `--judge-model claude-haiku-4-5` for grading, and `--max-cost-usd 20` for the run [16].
The gate is narrower than the reporting. Exit 0 means every case met `--threshold`, and exit 1 covers a case below threshold, files that failed to load, or missing trust [17]. The delta is what the command reports [1], and the CI flags the writeup lists gate on score [9]. The same invocation writes `--json results.json` [16], so a merge rule that fails on delta is a script you write against that file, with a threshold you choose.
Two-arm mode drops `tool_used` graders on `Skill` from both arms' scored totals, so an author cannot invent lift by checking something the without-arm could never do [12]. A case can also score 1.00 with the plugin loaded and still be worthless, since Claude may already solve the prompt [10].
MCP-backed skills get mocks under `evals/mocks/<server>/<tool>.md`, so CI does not need the real service [18]. `--scaffold` is the flag to handle carefully: scaffold scripts run as you, outside the agent sandbox, and the writeup says to use it only for suites you trust [19]. All of this needs Claude Code v2.1.269 or later [3].
What to watch
- A flag that fails a run on the delta directly would remove the results.json script teams have to write themselves today.
- Whether the list-price estimate for llm and baseline graders tracks real invoices on suites with dozens of judged cases.
- The command needs v2.1.269, so both the pinned CI images and the flag names can still change under teams already depending on them.