Loading today’s stories
Build1 distinct publisher3 min readUpdated
A Google AI series on dev.to shows how Inspect AI turns "is this MCP server worth my tokens" into a measured question, using a cheap grader model and three runs per test.
The Engineer · Build desk
Compiled by The EngineerSomething wrong?How this is made
A series published on dev.to under Google AI's account walks through evaluating an agent skill with open source eval frameworks instead of adopting it on impression, naming Inspect AI and Harbor as the tooling for the job [1][3]. That framing is the useful part for anyone running agents in production: the cost of a mediocre skill is not a bug report, it is quota burned re-prompting a model with the same schemas, descriptions and scripts over and over [2].
The mechanics are ordinary, which is the point. The companion codelab runs evals with Gemini CLI, Inspect and Inspect SWE inside an isolated Docker sandbox, asking the agent the same question with each skill in turn so the skill is the variable [4]. The author reports building the sweep around three architectural dimensions: external configs, isolated sandboxing so parallel attempts cannot corrupt each other's state, and multidimensional rubrics [9]. The `--epochs` flag is set to 3 so each test runs three times and non-deterministic output gets averaged out, with metrics aggregated by the mean of those runs [10].
The cost engineering is worth copying. According to the post, two models act as solvers under evaluation, google/gemini-3.5-flash-lite and google/gemini-3.6-flash, while a previous-generation model, google/gemini-3.1-flash-lite, does the grading [5]. Grading was deliberately pushed onto the older model: with rubric criteria reduced to strict binary decisions and the reduction applied programmatically, the author says it is robust enough without consuming solver quota [6]. For a production eval system the same post recommends newer, more capable graders, on the grounds that they will likely produce narrower confidence intervals [7]. Note what that arithmetic implies: two solvers at three epochs is six solver runs per test case before you count grader calls, which is exactly why the grader model choice is a budget decision and not a taste one [16].
Reproducibility gets real attention. The runs were benchmarked on Python 3.13 with inspect-ai 0.3.247, inspect-swe 0.2.66, inspect-viz 0.4.1 and pandas 3.0.3, with exact version pins in the README in case upstream PyPI releases break something [8]. Followers along at home are told to clone Google's public skills repository into a local `google-skills/` directory before running the sweeps [13], to install `inspect view` in advance [11], and to expect evals to take a few minutes depending on machine and quota [11]. `inspect view` then serves the results locally and opens up individual traces [11].
Two honesty notes. The published text reports the last run taking "8 minutes and 21 minutes", which is not a readable number, and the bullet describing multidimensional rubrics repeats the sandboxing description verbatim, so the rubric design is asserted rather than shown [12][15]. Harbor is named in the framing but the demonstrated stack in this installment is Inspect and its companion packages [17]. The post also discloses that its diagrams are AI-generated alongside screenshots and hand-drawn edits, with AI assisting minor copy editing [14].
What to watch: the promised follow-up extending the evaluation into visualizations via Google Sheets and Data Studio [18], whether the pinned dependency set survives upstream churn [8], and whether anyone publishes rubric definitions concrete enough that a second team can reproduce a skill's score rather than just its run command [9].
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
Grading was intentionally offloaded to a previous-generation model; the author states that by changing rubric criteria to strict binary decisions and applying a reduction programmatically, it delivers sufficiently robust evaluations without burning through solver quota.
Readers following along are told to install inspect view before running the evals and to expect a couple of minutes of waiting depending on machine and quota usage; inspect view opens a local dev server to view eval results and dive deeper into individual traces.
A dev.to post published under Google AI's account, titled "Designing AI Evals: Clarity Now and Visualization Next", is the opening of a series on designing objective evaluations of AI tooling such as MCP servers, agent skills and agent plugins.
The post frames the problem as: modern LLMs can likely one-shot many specific tasks, but that specificity may require wasting tokens and time repeatedly prompting them with the same resources, descriptions and scripts, raising the question of whether a skill is worth your time, tokens and quota.
The post names Inspect AI and Harbor as open source eval frameworks for evaluating agent skills.
In the companion codelab, the author ran evals with Gemini CLI, Inspect and Inspect SWE in an isolated Docker sandbox to understand how well each skill aids the agent in answering the same question.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Single first-party tutorial, method shown but results absent
Everything rests on one dev.to post authored under the vendor's own account, with no independent corroboration in the cluster. The reproducibility scaffolding is unusually concrete for a blog post (pinned package versions, an explicit CLI invocation, a git clone step), which lifts it above pure assertion. Against that: the supplied text reports no scores or comparative outcomes, the robustness of the cheap grader is asserted rather than measured, and the post contradicts itself on run repetition and on its own list of architectural pillars. Two ledger details (runtime figures, a duplicated bullet description) could not be located in the available text at all.
One author's demo plus a public skills repo dependency
Observable adoption is limited to first-party usage: one executed sweep by the post's author, a pinned Inspect-family toolchain, and readers being pointed at Google's public skills repository. There are no third-party deployments, no download or user counts, no organizational rollouts and no evidence that anyone outside the author has run the harness.
Rigor framing modestly outruns what is shown
The post's own register is fairly measured: it calls the setup a demo, warns that production systems should use stronger graders, and hedges its robustness claim. The gap is mild rather than severe. It comes from the promise of 'objective evaluations' and a comparison of open source eval frameworks being delivered as a single-vendor Inspect walkthrough with no results in the available text, no cost figures despite quota being the premise, an unexercised Harbor mention, and internal inconsistencies in the very parameters that would establish rigor.
Vendor-authored content promoting its own stack
The piece is published under Google AI's dev.to account and its recommended path runs almost entirely through Google products: Gemini solver and grader models, Gemini CLI, Google's public skills repository, and a promised follow-up built on Google Sheets and Data Studio. The measurement layer is third-party open source (Inspect AI), which slightly dilutes the promotional pull, and the post discloses AI-generated diagrams and AI copy-editing assistance, which is a transparency point. No conflicting or independent voice exists in the cluster to offset the vendor framing.
Method claims traceable, substance and outcomes unverified
Confidence is capped by cluster structure: one vendor source, no independent verification, a body truncated mid-sentence, and two ledger claims that the available text does not support. What the post says about its own configuration is directly quotable and therefore reliable as a description of intent, but the internal contradictions on epochs and architectural pillars mean even the self-description is not fully coherent, and none of the evaluation outcomes can be checked.
leadership
Cost per successful task, not per token: a 2,400-run benchmark reorders the model shortlist1 distinct publisher
product
Docker puts Verified Publisher behind a signup form, and pull data behind a plan1 distinct publisher
build
A five-check gate is the only docs accessibility test that will survive contact with CI1 distinct publisher
build
Two Next.js apps instead of one, because isAdmin is a privilege escalation waiting to happen1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 18, 2026