Build1 publisher3 min readPublished
A six-label scorer keeps a 429 out of the model's error column
A dev.to post plants 24 fake API responses across six failure classes to show what a boolean pass rate does with a timeout, and its author says up front that the percentages are the fixture talking.
The Engineer · Build desk
What happened
- The post's premise is that most agent evals still score success as HTTP 200 plus the expected string appearing in the text, which files a timeout as a model error.
- It replaces that boolean with six labels for a model call: drop, empty, truncated, schema, wrong and right, each with a bucket of its own.
- The test set is twenty-four envelopes held in memory, four for each label, so the author controlled every failure instead of waiting for a live fleet to produce one.
- A disclosure says the article was prepared as part of MonkeyCode's product outreach, whose free model access and free server option are the author's example of a hostile mailbox.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost Adopting the split costs one stored record per call: status, finish_reason and the raw body. A harness that kept only a pass/fail boolean cannot re-grade the runs it already published.
- exposure According to the author, unpaid paths drop, stall and truncate in ways a dedicated box will not, so comparisons run on free tiers carry the most of this noise in their pass rates.
- capability Because the scorer never opens a socket, a team can prove the grader recovers failures it planted itself before anyone trusts a number that came off a live endpoint.
- decision Anyone shipping a single headline pass rate now has to defend it against publishing two; the author's position is that the product of yield and accuracy hides the path that produced it.
classify() looks at the HTTP status before anything else. The first branch returns drop for 0, 408, 429, 500, 502, 503 and 504; the next returns drop for any status that is not 200 [7]. Every member of that set is already not 200, so the second branch would catch all of them and the list works as a comment on which failures a free path is expected to produce [22]. The same rule sends a 400 or a 404 to drop [23]. Your own harness's malformed request lands in the same bucket as a reset connection.
A 200 has more gates to clear. A body that is not a dict, a missing or empty choices list, and a content field that is None or an empty string all return empty [8]. If finish_reason is "length", read from the envelope or from the first choice, the label is truncated, and json.loads on the content runs only after that check [9]. So an empty completion arriving under HTTP 200 never reaches the comparison against the expected answer [8].
The Envelope dataclass carries http_status, elapsed_ms, finish_reason, body and expected [10]. classify() reads the status, the body and the finish_reason; elapsed_ms goes unread [24]. Slowness reaches the scorer only if the client converts it into a status, which is what the 0 at the head of the drop set is for [24]. If the client raises on read timeout and the harness logs nothing, there is no envelope to classify [24].
Over the planted set, the naive scorer reports 4 of 24, about 17 percent [18]. Accuracy-on-yield reports 4 rights against 4 wrongs, or 50 percent [19]. Twelve envelopes, the drops, empties and truncations, hand back nothing gradeable [20]. The three-to-one gap between the two reports is a property of the planted proportions [21], and the author says so: "The percentages are the fixture talking, not a vendor scoreboard." [14]
For a gap that size to appear on your own runs, drops, empties and truncations would have to account for half your calls [26]. The post does not claim they do. Its disclosure says the author is "not naming models, quoting quotas, or claiming uptime" [13].
The label I would look at hardest is schema. Accuracy-on-yield is rights over rights plus wrongs, so a body that parses as JSON and omits a required field counts in neither term [25]. Drop a required field reliably and the model stays invisible to both numbers. That makes schema the label most likely to file a capability problem as a formatting one. The post gives no retry policy and no timeout budget, and the code listing breaks off inside the JSON parse branch [17].
What to watch
- Whether the author publishes yield and accuracy-on-yield from live calls, with the measured drop rate of the endpoint attached.
- A version of the scorer that counts retries separately from drops. That version would settle whether a 429 costs anything in the final numbers.
- Harness maintainers adding a non-200 allowlist so 4xx responses to malformed requests stop counting as path drops.