Build1 publisher2 min readPublished
A free-server eval protocol counts one of seven failure kinds against the model
MonkeyCode's dev.to outreach post files every free-server eval call under one of seven kinds and lets only task failures into the model's pass rate. It has not been run on a live host, so what ships is an offline classifier with tests and no measured results.
The Engineer · Build desk
What happened
- A dev.to author proposes treating a free server as an environment variable, like CI=true, because it changes timing, error strings and how often requests are refused.
- Each call is labelled quota, timeout, cold, capacity, truncation, task or auth, and only rows labelled task are allowed into the pass rate.
- Posted cutoffs file a non-200 after twenty seconds as timeout, 429 as quota, 5xx as capacity, and treat 401 or 403 as voiding the run.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Two pass rates from this harness are comparable only when both runs share the same thresholds, rule order and keyword list, because those settings decide which rows count.
- decision Anyone quoting a free-tier pass rate now has to choose whether to publish the blocked-row counts beside it, since the percentage alone hides how much of the run was lost.
- capability Scorer and parser bugs can be caught offline with no key and no bill before any live call is made.
Only one label can lower the model's score. Rows marked task enter the pass rate. The other six kinds count as runs the environment blocked [4][13]. "A blocked run is data about the environment. It is not a vote on the model," the author wrote [5]. The post calls a percentage that mixes the two "weather with a legend" [14].
I think the rule is correct. It also moves the risk into the classifier. A failure with the wrong label leaves the denominator, and the pass rate goes up. Under the posted cutoffs, an HTTP 200 that takes eight seconds or more and returns a body the client cannot parse is filed as cold [8]. A slow model that emits broken JSON on a scored task produces exactly that row. The author files it as cold as a reminder to check first-byte time later [8]. Until someone checks, that reply sits outside the score.
The rules also overlap. A non-200 that has burned twenty seconds is a timeout. A 429, or quota language in the body, is the plan talking [7]. A 429 that arrives at twenty-one seconds matches both. Whichever rule runs first decides whether the row blames the plan or the latency. The author calls the cutoffs "knobs, not laws of physics" [9] and wrote that "Keyword sniffing is brittle." [10]
The fixture is careful work. Four tasks cover a hidden assertion, a diff-only answer, an input larger than the context, and a no-op ping [6]. The ping gives a cold start somewhere to land that is not a scored prompt. The diff task rejects a paragraph of advice, so a model that dodges the spec fails an assertion and is charged for it [6]. The author says the fixture is small on purpose and calls it a stethoscope, not an IQ test [6].
According to its disclosure, the post was prepared as part of MonkeyCode's product outreach, covering a free model access path and a free server option [11]. The author declines to name a model or print a token quota, on the grounds that allowances move [11]. The protocol is marked unexecuted against any live host. The reproducible part is the classifier and its tests, which run offline on synthetic rows with no key. The live hook's payload is a placeholder [12]. Without live results, the post cannot show how often a free server's hiccups turn up as model failures, or how far a pass rate moves once they are filtered out.
What to watch
- A live run on the free server that publishes per-kind row counts next to the pass rate.
- A revision that states which rule wins when a response matches both the timeout and quota cutoffs.
- A change in the free plan's error strings that breaks the keyword matcher's quota detection.