TypeSafe's Jev reaches frontier AUROC on transcript monitoring at about 1/200th of Opus 4.7's cost

A Jev call carries a state and a set of typed questions. The model is non-autoregressive and cannot emit text ^[1]. The state is text describing the situation, such as an agent's transcript ^[5]. For suspicion scores the authors used a score question. Its levels are written in words, for example five running from "clearly benign" to "clearly deceptive", and Jev returns how likely each level is ^[6]. Every question in a request is answered in a single step, with no text generated ^[5]. TypeSafe pitches it as a "System One" model for classification, routing, scoring and verification ^[4].

The LessWrong write-up reports the monitoring results two ways, and they point in different directions. Judged by ROC, Jev is near the frontier ^[10]. A deployed monitor runs at one threshold, though, set by how many false alarms reviewers will tolerate. At the strict 1.5% false-positive rate the authors tested, Jev caught 45.9 on SHADE-Arena against GPT-5.5's 59.3, a 13.4-point gap ^[10][15]. At that threshold the coding-sabotage gap is 2.8 points and the sandbagging gap is 22.2 ^[16][14].

The 1/200th figure carries over only under two conditions. The comparison models ran without chain of thought, and the cost ratio is quoted against Opus 4.7 ^[2]. A team whose current monitor reasons before it scores is starting from a different baseline than the one measured. A team already on a cheaper model than Opus 4.7 will see a smaller ratio. Either way, the saving assumes transcripts that resemble the sabotage and sandbagging sets the authors scored.

The reasoning weakness has two sources. Jev cannot write a multi-part answer, so the authors asked one question per part in a single request and counted the answer correct only if every piece was right ^[7]. A Sudoku became 81 choice questions ^[7]. Sudoku is an odd exam for a classifier, and the authors say as much, calling weaker performance on these tasks unsurprising ^[7]. Multi-part tasks, chess and Tower of London among them, were its weakest category ^[11].

NCRI is the cleaner test. Its free-form questions were converted to multiple choice, using lists of 255 consecutive integers where the answer is a number ^[13]. Each item then takes a single pick, and Jev still ranked below about three quarters of the field ^[3][17]. Chained lookup holds up better. On N-hop table lookup it performs strongly up to 6 hops ^[12].

For routing, the useful output is the probability Jev assigns to every option of a choice question ^[6]. A router can take the top option when it clears a confidence bar and pass the request to a reasoning model when it does not. I think both jobs the authors propose, routing and always-on monitoring, should be built that way ^[18]. In my context, where a reasoning model is affordable on a small share of calls, Jev scores everything and the flagged or uncertain cases go to a model that writes its reasoning down.