Build1 publisher2 min readPublished
Jev collapses structured output into a single forward pass
Sean Goedecke argues that a model emitting only typed answers is fast because it skips autoregression, and that most of that speed is already available from any LLM you can prefill and constrain to a list of choices.
The Engineer · Build desk
What happened
- Sean Goedecke writes that Jev takes a human-language prompt and returns only structured output, which he calls a System One model rather than an ordinary chat model.
- Because it emits only structured output, Jev is not autoregressive, and it can answer many questions in parallel inside a single forward pass.
- Goedecke reports a fastest response of around 70ms and a worst case of 500ms, against a couple of seconds for a normal LLM doing the same job.
- He argues the same speed is reachable today by prefilling the response structure and generating one token restricted to the caller's list of choices.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- capability A decision that returns inside half a second can live in a game loop or a request handler; a two-second chat call has to be moved off the hot path or hidden behind a spinner.
- decision Teams currently paying chat latency for routing and labelling can test prefill plus a constrained sample on the inference stack they already run before taking on another model dependency.
- constraint The speed only covers answers drawn from a choice list the caller supplies; anything needing a long JSON document goes back to grammar-constrained decoding at one forward pass per token.
- cost Anyone building a budget from this has one blog post's millisecond figures and no price per call, so a token-cost saving has to be inferred from how many output tokens you stop generating.
Grammar-constrained decoding is how most structured output gets served now. The model generates autoregressively and the logit sampler discards any token that would break the schema, so it cannot emit a "]" before it has emitted a "[" [9]. The schema scaffolding therefore costs forward passes. Goedecke's example lists "{", then " ", then "answer", then ":", each its own pass, before the model reaches the value you actually asked for [6].
His alternative moves that scaffolding into the prompt. You prefill the response with `"choice": "` and generate exactly one token, restricted to the choices the caller supplied [11]. Input tokens are ingested in parallel, so the prefix costs far less than generating it does [12]. In his own example that removes at least four generation passes and leaves one [4]. Several choices can ride in the same forward pass through ordinary inference batching [13].
"My biggest problem with Jev is that I think fast structured output is already available," Goedecke wrote [15]. His diagnosis of why nobody has already shipped it: (a) nobody really cares about structured output, and (b) the people who do care want big JSON blobs [10].
The floor he reports is about a thirtieth of a two-second chat call [1]. Stated as a decision rate, the reported ceiling allows two decisions per second and the floor about fourteen [2]. For either end to transfer, the answer has to come from a choice list the caller already knows [11], the prompt has to be short enough that ingesting it is cheap [12], and you have to be willing to batch [13]. Long-form structured output does not fit this shape at all, and stays on the autoregressive path [14].
The Doom demo is a test of loop rate. Goedecke describes feeding a text representation of the game state plus choices such as whether the trigger should be held down and what the current goal should be, and says latency is low enough that the model plays well in real time [7]. He also notes that ordinary LLMs can play Doom, slowly [8]. Evidence on answer quality in the post is qualitative on both sides: the model plays well [7], and the prefill experiments people started running after the announcement seem to be working OK [16].
If the parallelism comes from prefill plus a one-token constrained sample, it is a serving-configuration change on inference stacks teams already run. Goedecke put it as a hope: the model "is not that different from an ordinary LLM with structured output, but the interface it uses is very cool and I hope it becomes more widespread" [17].
What to watch
- Whether anyone publishes reproducible latency figures for the prefill-and-constrain path on a named open-weights model and batch size.
- Whether inference vendors expose response prefill plus a choice-restricted sampler as a first-class API parameter instead of a grammar file.
- Whether Jev's developer publishes a price per call and accuracy figures on standard classification tasks.