Pick an LLM gateway on its cost records, not its model list
A dev.to practitioner argues the deciding test is a per-tenant usage ledger and a JSON contract that holds on a fixed labeled eval set. Price enters last, if at all.
The Engineer · Build desk
What happened
- A dev.to writeup argues that a gateway should be chosen only after a fixed evaluation set proves its JSON contract, fallback behaviour and per-tenant cost records.
- The worked example is a healthtech service scoring candidates against a job rubric, where tenant-level cost visibility is named the deciding constraint.
- The author keeps direct provider adapters where an app depends on proprietary controls, reserving gateways for cases where several providers meet one narrow contract.
Why it matters
- What it limits constraint Per-tenant cost attribution is only as good as the least forthcoming provider in the pool, because the token counts are the one part of the record the caller cannot produce alone.
- Who has to decide decision A vendor conversation that opens on per-million-token pricing is answering the fourth question first, and the answers to the first three can disqualify the cheapest option outright.
- Who's exposed exposure Without semantic checks after parsing, a well-formed response carrying an out-of-range score or stray personal data reaches a hiring decision, and the reviewer owns the explanation.
- What doesn't add up contradiction The convenience the gateway is bought for, one key and automatic cheapest routing, is the same mechanism that blurs which tenant incurred what, so the sales pitch and the audit requirement pull...
The load-bearing part of that argument is the record layout, not the eval harness. The dev.to piece lists nine fields to capture per classification: tenant ID, evaluation-set version, model identifier, input and output token counts, latency, retry count, final status, and the reason for any fallback [3]. Seven of those nine the calling application can write by itself [10]. The two it cannot are the token counts, which the author hedges as "when available" [3].
That hedge is the procurement question in one word. A gateway that returns usage per request lets you keep a tenant ledger. One that returns it per key, or only inside a dashboard, leaves you estimating a tenant's spend from document length and then defending the estimate to an account team that has already noticed the bill [4]. The author's stated failure mode is a monthly aggregate concealing either one tenant with unusually long documents or a retry storm [4]. Both are invisible in a total and obvious in a ledger.
The same asymmetry runs through the interface he keeps between adapter and application. ClassificationResult carries seven fields, and only two of them, score and rationale, are the classification [6]. The remaining five are there so someone can reconstruct which tenant, which model, how many tokens, and why the call fell back [11]. He is explicit that the model call is meant to stay replaceable while the audit record does not [19].
Cost arrives late in the routing policy, and that ordering is the actual advice. Gate one removes models that cannot satisfy the output shape or the data-handling requirement. Gate two removes anything that failed the hold-out evaluation. Only the survivors get sorted on bounded cost and latency, and the fallback has to have cleared the same gates [8]. In his sample code the eligibility test is JSON support, quality at or above 0.94, and p95 latency at or below 2500 ms, with cheapest-of-the-eligible as the final line and a raised error when nothing qualifies [9]. Price is the fourth question [12].
The validation point is narrower than it looks. A structured response that parses can still carry a score outside the rubric range or a rationale that discloses personal data nobody asked about, which is why the parser sits between the provider and the hiring workflow rather than after it [2] [20]. Normalising a request does not make a vague rubric precise [1]. His own guard is blunt: reject any score that is not an integer from 0 through 100, reject a blank rationale [7].
One caveat on weight. This is a single practitioner's design argument, referencing OpenAI's function calling guide for the general validate-the-arguments pattern [5]. It contains no measured comparison of named gateways on contract stability, so the ordering it proposes is a hypothesis you would have to test on your own labeled queue [17].
What to watch
- Whether gateway vendors publish per-request or per-correlation-ID usage as a documented export rather than a dashboard panel.
- Whether the evaluation-set version survives in production records at volume, since it is the cheapest field to quietly drop.
- Any measured comparison of named gateways on output-contract stability, which this practitioner argument does not attempt.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence32
- Adoption
- Not scored
- Hype gap−8
- Incentives20
- Confidence34
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
A gateway may normalise the request, but it cannot make an ambiguous rubric precise.
- [2]
JSON mode is not the same as semantic correctness: a valid object can still contain a score outside the rubric or a rationale that exposes irrelevant personal data.
- [3]
For each candidate record, capture the tenant ID, evaluation-set version, model identifier, input and output token counts when available, latency, retry count, final status, and the reason for a fallback.
- [4]
A monthly total can hide one tenant's unusually long documents or a retry storm; a tenant ledger makes that visible before finance or an account team finds it first.
- [5]
The score should be a structured object rather than prose another script interprets, and the OpenAI function calling guide is cited as a reference for defining the output shape and validating the returned arguments.
- [6]
The proposed ClassificationResult dataclass carries seven fields: tenant_id, score, rationale, model, input_tokens, output_tokens and fallback_reason, behind a Classifier protocol with a classify method taking tenant_id, text and rubric.
- [7]
The sample validate_result function raises an error unless score is an integer from 0 through 100 and rationale is a non-blank string.
- [8]
Routing should be a policy with gates rather than a cheapest-wins sort: first remove models that cannot satisfy the requested output shape or data-handling requirements, then remove candidates that failed the hold-out evaluation, then choose among survivors on a bounded cost and latency policy with a fallback that passed the same checks.
- [9]
The sample choose() function admits only candidates with supports_json true, quality at least 0.94 and p95 latency at most 2500 ms, returns the minimum estimated_cost among them, and raises RuntimeError when no evaluated candidate meets the policy.
- [10]
Of the nine per-record fields prescribed, seven can be written by the calling application without any provider cooperation; only the input and output token counts depend on the provider returning them.
- [11]
Five of the seven fields in the proposed result object exist for attribution and accounting rather than for the classification itself, which is carried by score and rationale alone.
- [12]
In the described routing policy, cost is the fourth criterion applied: output shape, data handling, hold-out evaluation result, then bounded cost and latency among the survivors.
- [13]
The author's short answer: choose a gateway only after a fixed evaluation set shows that its JSON contract, fallback behaviour and per-tenant cost records are good enough for your classifier.
- [14]
For a healthtech service scoring candidates against a job rubric, one API key and cheapest routing are convenient, but tenant-level cost visibility is the decision constraint.
- [15]
Moving from notebook to production adds tenant attribution, rubric versions, protected data handling, malformed output, rate limits, and a reviewer who needs to explain why a candidate received a score.
- [16]
The author would start with a direct provider adapter if the application depends on one provider's proprietary controls, and would test a gateway when several providers can satisfy the same narrow contract and the team wants to change that choice without rewriting the scoring workflow.
- [17]
The first proof is not a low price but repeatable behaviour on a labeled set resembling the real queue: sparse resumes, ambiguous experience, missing fields, and rubric criteria that sound similar.
- [18]
Version the rubric, prompt, label schema and model choice with the evaluation, and record the result per tenant rather than only as one aggregate score.
- [19]
The plain interface lets an evaluation harness call several adapters with the same examples while the application records usage in one place: the model call stays replaceable, the audit record does not.
- [20]
In the real adapter the single key authenticates the gateway while the request still carries a model choice and a correlation ID, and the response parser must validate JSON before a score enters the hiring workflow.
- [21]
Fallback needs its own test cases, simulating a timeout, a rate-limit response, invalid JSON and a provider refusal, with bounded retries for transient transport failures.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toOne API Key, Multiple LLM Providers: Comparing Gateways for Python Text Classification
1 article · August 22, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- Per-Tenant LLM Cost AttributionFollow
- LLM Gateways and Provider AbstractionFollow
- Structured Output ValidationFollow
- AI Candidate Scoring RiskFollow
- Labeled Evaluation Sets and Model Routing PolicyFollow