Science1 publisher3 min readPublished
A 10,000-call probe puts Jev's probability readout at the end of prefill
archerhume timed latency against context length and question order to work out how TypeSafe's closed model returns distributions instead of text. Verifying that those numbers are calibrated takes labelled outcomes he lacked.
The Scientist · Science desk
What happened
- archerhume says he made 10,000 calls to TypeSafe's Jev API, timing latency against context length and reordering questions, to infer how the closed model is built.
- The support-routing example showing payments at 0.91 and urgent escalation at 0.42 is schematic, and the author states those probabilities are invented.
- Jev's weights are closed and TypeSafe keeps its research private, in the author's account, so his description of the design comes entirely from outside the API.
Compiled by The ScientistSomething wrong?How this is made
Why it matters
- constraint Timing a black box can bound where it spends compute, but it cannot reach calibration, so the claim that Jev's probabilities track outcomes stays a description until someone scores it against labels.
- decision A team weighing this for fraud screening now has to budget its own outcome-labelled evaluation, because the probabilities in the write-up are illustrations of the API's shape.
- capability If the readout really ends after prefill, per-call cost for high-volume classification stops tracking how many tokens the answer spells and becomes one forward pass plus a small head.
- precedent One person with an API key and a stopwatch produced the only public architecture description Jev has.
The part of the essay that needs no probing at all is its opening argument. An ordinary LLM writes out the words 90% confident as text, archerhume wrote, and its probability of producing those words does not establish a 90 percent probability of being right [4]. He set that against the reaction on X, including a post he quotes as saying, "12 million views for a JSON classifier? Yeah, we're in a bubble" [18]. Fraud screening, moderation, routing and risk assessment are built on the pattern he objects to: pay for token-by-token generation, then treat an unvalidated confidence claim as a probability the software can act on [5]. That is a claim about what a token probability measures, and it needs no API access to make.
The 10,000 calls did something narrower. archerhume looked for signatures in latency scaling under different context lengths and under question reordering, read the public documentation, and looked for prior art [9]. Latency scaling is evidence about where computation goes, not about whether a returned 0.91 corresponds to 91 correct calls out of 100 similar ones. That check needs predictions paired with observed outcomes, and the essay does not report one [19].
As the essay describes it, Jev keeps a pretrained model's knowledge and replaces generated confidence claims with decision probabilities read from internal representations and trained against outcomes [6]. The API takes shared state, a set of questions and the allowed answers, then returns the distributions in parallel without generating text [7]. In the support-routing example, payments gets 0.91 and urgent escalation 0.42, and the author says the probabilities are invented to show the structure [14].
The parallel part is less exotic than it sounds. Causal attention constrains which positions can use which information, not the order in which input tokens are executed; during prefill every input token is already known, so their positions can be processed together inside a layer while the mask blocks access to later positions, and the layers still run in sequence [16]. Decoding adds the dependency that prefill does not have, because the next token does not exist until the previous prediction has been chosen. A model that stops after prefill and a readout never pays it [17]. For a three-queue choice, the readout is a small function from that representation to three numbers [15]. The model hands back those three numbers without spelling "payments" one token at a time [21].
On the architecture, archerhume settles on a causal transformer, probably sparse mixture-of-experts, repurposed for decisions, with shared-state encoding, isolated question branches and direct probability readouts [8]. His hedges are explicit: "The sparse backbone is the least certain part" [10], and "This is clearly all quite speculative" [11]. On the method he wrote: "Black box APIs make it shockingly easy to throw a blanket over the ghost and get a rough shape of what the architecture looks like" [12]. Jev's weights are closed and, in his words, "TypeSafe refuses to share their research" [3].
If you are pricing this for screening work, the architecture is the cheaper half of the question. The single-pass shape is what makes a low per-decision cost plausible [21]. Whether the returned number is worth acting on is a calibration question, answerable by bucketing the probabilities against your own labelled outcomes and comparing each bucket's claimed rate with the observed one.
What to watch
- TypeSafe publishing reliability curves against held-out outcomes would turn the calibration claim into something a buyer can check.
- An open-weight model with the same prefill-plus-readout design would allow a direct test of the compute-shape claim without latency inference.
- Per-call pricing and latency figures from TypeSafe would show whether the single-pass shape reaches the buyer as a lower cost per decision.