Proposed eval harness would refuse to score when changelog and case grader_version disagree

Call `structural_grade` and nothing leaves the process. If the case sets `json_required`, the function runs `json.loads` over the whole completion and appends `json_parse` when that raises. It then lowercases the completion and tests each `must_include` needle as a substring, each `forbid` needle the same way. Last it runs a case-insensitive regex for `as an ai language model` and appends `boilerplate`. The return value is a boolean plus the reasons joined by commas, or the string `ok` ^[8].

The substring test is the weak edge. A match anywhere in the text counts, so a completion that quotes the requirement back and then ignores it still satisfies `must_include` ^[17].

`semantic_grade` is where the spend is. It POSTs a prompt instructing the judge to score 1 only if the completion satisfies the rubric and does not contradict the structural contract, and to reply with `{"score":0|1,"reason":"..."}` ^[9]. It sends a Bearer token, waits up to 45 seconds, reads `text` or `completion` off the response body, and returns `False` with the note `semantic_parse_failed` when the judge's reply will not parse ^[10].

That except clause catches `json.JSONDecodeError`, `TypeError` and `ValueError`. A timeout raises `URLError`, which is not in the tuple, so a slow judge aborts the run instead of quietly recording a zero ^[14]. In my view that is the right default, and it is the sort of thing that usually gets papered over with a bare `except`. It does put judge availability on the critical path: one call per case at a 45-second ceiling bounds a serial 100-case suite at 4,500 seconds, or 75 minutes, of waiting in the worst case ^[15].

`Grade` types `semantic_ok` as `bool | None`, so "not evaluated" is representable and a structural failure can skip the judge call entirely ^[11]. The published listing breaks off inside `evaluate`, just after it reads `changelog.get("semantic_grader")`, so the short-circuit path is not shown ^[13].

The version check itself rests on `load_changelog`, which reads a file line by line, skips blanks and `#` comments, skips anything without an `=`, splits on the first `=`, and strips surrounding quotes ^[12]. The `grader_version` comment points at `graders.toml` ^[7]. TOML table headers contain no `=`, so they are skipped rather than tracked, and two tables holding the same key name flatten into one entry ^[16].

You need a completion endpoint you already operate, which the post says to swap in for the sample client, and you have to accept its instruction not to read the sample cases as a benchmark of any hosted model ^[6]. You also accept that bumping a rubric is a diff across every case pinning the old version, because the runner fails closed when the cases and the changelog disagree ^[5]. The post labels the file a proposal and an unexecuted example, and reports no run ^[5].