Build1 publisher3 min readPublished
Google's Agent Skills team scores judge accuracy by aggregating true/false checks
The team's argument is that reliability in an LLM judge comes from how the rubric is written rather than which grader you buy, and that holding every question to an observable boolean lets a smaller model do the grading.
The Engineer · Build desk
What happened
- A Google team publishing Agent Skills says deterministic tests such as compile checks cannot be built at scale for open-ended answers and retrieval tasks, so those responses go to a model-based grader.
- Each response is judged against a structured rubric of true/false questions, and the aggregated answers become that response's accuracy score.
- The guidance tells authors to write rubrics as formal specifications using RFC 2119 terms MUST, MUST NOT and REQUIRED so every question tests an observable outcome.
- A compound check asking whether a response contains a metadata property and formats output as JSON is to be split in two, because the judge otherwise guesses which clause matters more.
- Because agents will tailor answers to any test they can see, the post says scoring rubrics belong in a separate system and should target functional outcomes rather than broad keyword matches.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The saving from a cheaper grader is contingent on the questions staying single-fact, so any check that quietly requires judgement or unstated domain knowledge puts you back on the expensive model.
- decision Whoever splits or merges a question is setting the metric, since an equally weighted rubric re-weights itself every time the question count changes.
- constraint Anything you cannot state in advance as an observable fact falls outside this harness, which leaves answer quality and reasoning needing a separate instrument rather than a better prompt.
- exposure Storage location becomes an integrity control: whatever the agent under test can read, it can be tuned against, and these skills live in a public repository.
Decomposition changes the scoring, not just the wording. If the grader aggregates boolean answers into an accuracy score [3], every question carries equal weight unless you say otherwise. Take a ten-question rubric: each check is worth 10 percent. Split one compound question in two, which is exactly what the post prescribes for "Does the response contain a metadata property and format the output as JSON?" [7], and you now have eleven questions, and that single requirement carries 2/11, about 18 percent [16]. The rule against overlapping questions [8] is the same arithmetic from the other end: two questions about one concept double the penalty for one mistake. Question authoring is weight authoring. Nobody writes those weights down, which is why they drift.
The rest of the guidance is a testability filter. Intent, quality and reasoning are out, because they require interpretation rather than observation [9]. The recommended vocabulary is RFC 2119: MUST, MUST NOT, REQUIRED [10]. Those are all obligations, and that is the point, since a requirement you would merely prefer has no TRUE or FALSE. The best line in the post is the one about negative constraints: instead of asking whether the agent used best practices, check that it did not suggest a specific deprecated feature [11]. "Followed best practices" is not decidable by any grader. "Did not recommend feature X" is.
The cost claim rides on that filter. The team argues a smaller, faster model suffices because strict boolean evaluation is a less complex task [6]. For that to transfer to your harness, each question has to be answerable from the response text alone, with no environment access and no domain knowledge the small model lacks. A deprecation check qualifies only if the rubric names the feature; leave it implicit and the cheap grader is guessing about your API surface.
What the material does not contain is a measurement. The reduction in rating variance is asserted as practice [12][17], with no agreement figures between graders or between runs. That is the number I would want before swapping in a smaller judge, and it is cheap to produce: run the same responses twice and count disagreements per question. The other half of the risk is coverage, and the post is direct about it. Grading requirements the prompt never stated manufactures false negatives [14], so a rubric misaligned with the task measures the rubric.
One structural consequence is easy to miss. The advice to keep scoring rubrics in a separate system exists because agents tailor answers to whatever tests they can see [13], and these skills are published on GitHub [1]. Public artifact, private assertions. If the rubric ever lands next to the skill file, the score stops being evidence.
What to watch
- Whether the team publishes run-to-run or cross-model agreement figures for its boolean graders, which is what would justify the smaller-judge claim.
- Whether the public Agent Skills repository ever ships rubric files next to the skills, putting the assertions inside the agent's reach.
- Whether checks that could be deterministic get promoted out of the judge entirely, shrinking the model-graded share of each score.