Count invalid JSON as a failed classification, and model choice becomes a reliability problem

A developer writeup on dev.to lays out how to pick a text classification and tagging API for an e-commerce backend that scores job candidates against a rubric, and its operative move is a definition rather than a benchmark: any response that cannot pass the application's JSON Schema is a failed classification, however sound the prose ^[1][6]. The resulting service-level indicator is valid, schema-conforming classifications divided by attempted classifications ^[7], which quietly changes what a model bake-off is measuring. The reframe is worth taking seriously because it captures the failure that bills you. A classification call can be inexpensive and still operationally costly when malformed output causes retries, or when a shared invoice makes one tenant's usage invisible ^[3]. Since retried calls are attempts and sit in the denominator, a schema miss is paid for twice and counted once, so cost per successful classification drifts above the posted per-call price even while aggregate graphs look calm ^[23]. The author's first capacity question follows from that: not which model is smartest, but how many candidate records can fail validation before the pipeline misses its SLO ^[4]. The metric has an obvious weakness, and the post names it. A parser can succeed on garbage, so latency and rubric agreement are tracked separately to stop a fast parse from hiding bad labels ^[8]. The payload is small, a rubric score plus tags such as seniority, location and must_have_skills ^[5], which is exactly the shape where a well-formed object with wrong contents passes silently. Test design is conventional and therefore useful. Use real, previously reviewed candidate records, including sparse resumes, long resumes, ambiguous evidence and text in every language the application accepts ^[9], and send the same rubric version, schema and decoding settings to each candidate model ^[10]. Retain model ID, prompt version, input and output token counts, schema result and tenant ID with every decision ^[11]. The author distrusts aggregate cost graphs here, on the grounds that a large tenant can dominate traffic while the mean stays flat, and without the tenant dimension a platform team cannot set a budget, explain a bill, or decide which account needs a smaller prompt ^[12]. Keeping schema result and tenant ID on the same record is what makes cost per successful classification computable per account rather than in aggregate ^[24]. There is no universal pass mark. The threshold comes from the human-reviewed acceptance set and the error budget for that specific hiring workflow ^[13], and because model behavior changes, the held-out test belongs in release verification rather than a one-time procurement spreadsheet ^[14]. OpenAI, Claude and Gemini are expected to earn the production slot on that test, not on a general benchmark ^[2]. The last third is a buy-versus-build argument, and readers should price it accordingly. The post distinguishes direct integration, where the application owns a provider-specific path, from portable, where it owns one contract and routes models behind it, with cost deliberately not the first column ^[15]. It recommends Infrai for the portable row, citing a single API key and consolidated bill instead of dozens of keys and invoices ^[16], plus per-call cost, vendor and latency metadata and keyless public schema discovery ^[17]. We have not verified those product claims; they are as described by the author, who also concedes this is contract portability and not model interchangeability, so prompts still need evaluation and any model change still needs a canary ^[18]. Stay direct, the post says, when a binding provider agreement, provider-specific controls or a minimal request path matter ^[19], and note that Infrai has no dedicated moderation endpoint, so text safety falls to chat output constraints and application policy ^[20]. Two things to watch. Whether the schema-failure rate is reported per tenant rather than in aggregate, since that is where retry cost concentrates ^[12], and whether the held-out test actually runs on every release instead of once at purchase ^[14].