A no-plugin baseline doubles the agent runs in Claude Code's plugin eval suite

The default threshold is the line to read before wiring this into a merge gate. A run's score is the fraction of its graders that passed, weighted if you set weights, and the case's score is the mean across its runs ^[7]. A case passes when that mean meets `--threshold`, which is 1.0 unless you change it ^[8]. With three runs per case ^[6], a single-grader case that fails one run scores 0.667 and does not pass ^[1].

Then the model bill. The page puts a suite at roughly cases times runs agent runs with the plugin, as many again for the no-plugin baseline, plus three short judge calls per `llm` or `baseline` grader per run ^[12]. Ten cases, the default three runs, one judged grader each: 30 agent runs in the with-arm, 30 in the without-arm, and 180 judge calls ^[2]. Those calls go out on your credentials and count against your plan's usage limits or your API bill, and `claude plugin eval init` spends the same way ^[13]. When the command reports a cost, that figure is a list-price estimate of those calls ^[14].

The baseline is the good engineering here. A high score alone does not show the plugin helped, because Claude might do as well without it, so the runs are repeated with nothing loaded and you get a WITH score, a W/OUT score, and their difference ^[9]. The documentation says it directly: if a case scores 1.0 in both arms, "the plugin isn't what made it pass" ^[10]. You pay list price to learn that your plugin contributed nothing.

The delta from three runs is coarse. A single-grader case can only score 0, 0.333, 0.667 or 1.0, so the smallest non-zero delta it can report is 0.333 ^[3]. One grader flipping once in the without-arm produces that number; so does a real improvement. More graders and more runs give finer resolution at proportionally more model calls ^[12]. Turning the baseline off ^[11] halves the agent runs, 60 down to 30 in the example above ^[4], and drops the only column that separates the plugin from the model.

A few requirements decide whether you can run this at all. The floor is Claude Code v2.1.269 ^[15], and the harness wants a `plugin.json` or `.claude-plugin/plugin.json` manifest, or a skills-directory plugin ^[16]. `claude plugin validate` still only checks files for syntax and schema errors ^[19]. Authors already keeping an `evals/evals.json` for the skill-creator plugin get no reuse, because this case format is separate from that file ^[18]. The page names teams that gate plugin changes in CI as an audience ^[17], and the documentation does not include a CI example.