Build1 publisher2 min readPublished
TypeSafe AI's Jev swaps LLM-judge text generation for typed answers against structured state
LangChain clocked TypeSafe AI's Jev at 0.44 seconds and $0.00035 per call against three LLM judges on the same eval set. Whether that price transfers depends on how much structure your traces already have.
The Engineer · Build desk
What happened
- LangChain tested Jev, a model from TypeSafe AI that does not generate text and instead returns typed answers and probabilities, as a third kind of agent evaluator alongside code-based checks and LLM judges.
- On continuous scoring, Jev's quality-score variance came in 92 to 913 times lower than GPT-5.6 Luna, Terra and Claude Sonnet 4.6.
- Jev averaged 0.44 seconds and $0.00035 per call, totalling $0.34 for the run against $28.17 for Claude, and LangChain reported it as both the fastest and the cheapest evaluator in the test.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Because Jev scores structured state, a team that adopts it owns a trace-to-state extractor and a set of typed questions, code the prompt-based judge never required.
- exposure A harness swapped on the variance number alone has an unmeasured relationship to human scores, since the published takeaways report consistency and price only.
- decision Under thirty dollars separates the two judges on a single run, so the port only pays for teams that know how often the eval set actually executes.
Jev evaluates typed questions against structured state and returns typed answers with probabilities [10]. An LLM judge does something else: it accepts the question, the trace and the evidence as unstructured input, and a prompt tells it how to score [9]. That difference decides who does the parsing. With a judge prompt, the model reads the trace. With Jev, you build the state first, then ask atomic questions against it, and those questions can run in parallel against the same state [11].
The questions have to fit one of three shapes. Choice picks one option from a fixed set and returns probabilities and confidence. Score rates an answer against an ordered rubric, 1 to 5 in LangChain's example. Noul returns a float from 0.0 to 1.0 for the probability that a yes/no judgment is true [5].
Divide the run total by the per-call price and the eval set was about 970 judge calls [14]. Claude Sonnet 4.6's bill over the same set works out near 2.9 cents a call [15], and the ratio between the two totals is about 83 [16]. In absolute money, the run saved $27.83 [18]. Run that set nightly and the saving is roughly $10,200 a year; run it weekly and it is about $1,450 [22][23].
TypeSafe AI's own figure for Jev is up to 200x faster inference and 400x lower cost than comparable LLMs on classification tasks [8]. LangChain's measured cost gap against Claude landed near a fifth of that ceiling [17].
The variance figure needs the same handling. The 92-913x range covers GPT-5.6 Luna, Terra and Claude Sonnet 4.6 [3], a tenfold spread [19], so the improvement depends on which judge you are replacing. Variance measures repeatability, not correctness. A judge that answers 4 to everything has no variance at all. LangChain's four key takeaways cover variance, latency and cost, and do not include an agreement figure against human labels [21].
In my context the port is small for rubric scoring over traces I already serialise into a schema, and I would still run both judges against a labelled set before retiring the LLM. For open-ended judgment the state extractor is new code with its own failure modes, and open-ended tasks are exactly where code-based evaluation already breaks down, because encoding every acceptable answer as deterministic logic does not scale [12]. LangChain wrote: "The results are promising, but early." [6] It described the test as narrow [7].
What to watch
- Whether LangChain or TypeSafe AI publishes Jev's agreement with human labels on the same eval set, which is the figure the cost comparison cannot substitute for.