Leadership1 publisher3 min readPublished
TypeSafe's own evals put Jev level with GPT-5.6 Terra at a 75th of the cost per decision
Jev returns typed answers with probability scores and no free-form text, and TypeSafe prices a decision at four hundredths of a cent. Its accuracy figures are scored against other models' probabilities.
The Board Room · Leadership desk
What happened
- TypeSafe released Jev, a model built only for classification, scoring and routing: it returns typed answers with probability scores in 70 to 500 milliseconds and produces no free-form text.
- Across the four decision workflows in TypeSafe's published evals, Jev averaged 68% accuracy at $0.0004 and 0.4 seconds a case, and GPT-5.6 Terra averaged the same 68% at $0.03 and 10 seconds.
- TypeSafe's own figures put Jev at 40 to 200 times faster and 40 to 400 times cheaper than general-purpose models, depending on the task.
- Those workflow benchmarks score model outputs against reference probabilities produced by other models, not against human-labeled ground truth.
- On listing moderation, the UK events site NearHere measured 96% from Jev against 86% from Gemini Flash-Lite, at 58 times less cost per decision.
Compiled by The Board RoomSomething wrong?How this is made
Why it matters
- decision Arize's case for re-architecting rests on a share it never quantifies, so the saving for any one team comes out of counting its own pass/fail and routing call sites and what each is billed at.
- constraint The published evals cannot tell a team whether 68% is safe for its own human-review gate. Settling that needs labels from the team's own traffic, which is the data cost LLM judges were adopted to avoid.
- capability One tester ran 777 judgments in under 0.7 seconds for about a quarter of a cent, fast enough to run a whole eval suite inside a single request.
Set the three price points against each other and the gap in the middle row is 75x: GPT-5.6 Terra's $0.03 a case buys the same 68% as Jev's $0.0004 [1]. On time per case, 10 seconds against 0.4, the multiple is 25x [2]. Opus 5's extra five points cost $0.18 a case, which is 450 times Jev's price on the published figures; the Arize post rounds it to 440x [7][3][8]. Per thousand decisions: 40 cents, $30, $180 [4].
Arize AI, which published the comparison, said it will run its own benchmarks as soon as it can [11]. Jev had been public for two days when it wrote that [12]. Until human-labeled data replaces the model-generated reference probabilities, the 68% is a measure of agreement with other models [6].
68% is not a number to put in front of production traffic where the gate cannot be wrong. The spread by task decides it: on NearHere's listing moderation test Jev came in ten points above Gemini Flash-Lite [5]. In that same test the general-purpose model spent about 910 output tokens reasoning to each yes-or-no answer, and Jev spent 85 and does not bill for output tokens [15]. One of TypeSafe's four published workflows is reviewing a finished agent trace to decide whether a human needs to look at it [4].
Arize's architectural claim is that an enormous share of LLM-powered components in AI applications are being asked to make decisions, pass or fail, route A or B, which of five labels [17]. No percentage is attached. So the saving is a per-application count, and the inputs to it are already to hand: which call sites return a label, a score or a route, and what each one is billed at today.
LLMs became the default judge because they need no training set [20]. The older options wanted hundreds to thousands of labeled examples, then a training run, then someone to maintain the result, and a team shipping a first version of a product has none of that [20]. Jev appears to keep the zero-shot property: a developer ran it with no examples over 18,514 spam emails and landed a statistical tie with a classifier trained on the labels [16].
TypeSafe says Jev "can't hallucinate" [18]. Arize's post calls that an over-reach, because the model cannot return an answer outside the schema it was given but can still be wrong inside it [19]. TypeSafe trained the model with what it calls Reinforcement Learning for Calibrated Decisions, so that a higher stated probability means a higher chance the answer is right [22]. The probability score is the hint that an in-schema answer is wrong, and something downstream has to read it [21].
The independent record is three single runs by three parties [7]. That does not make a migration for this quarter. The count of decision call sites does not depend on which vendor wins. On TypeSafe's figures the gap between $0.03 a case and $0.0004 a case is 75x, and it applies to every step that only has to return one of five labels [1][17].
What to watch
- Arize's own benchmarks, which would replace TypeSafe's model-generated reference probabilities with human-labeled ground truth.
- Per-workflow accuracy, since the 68% mean hides the task-by-task spread.
- Whether any team publishes a before-and-after bill after moving decision call sites off a general-purpose model.