Splitting one fuzzy judgment into four questions cut shell-gate false positives to 1.8%

The shell gate is where the decomposition is easiest to see. Four `noul` questions go out about one command, each independent: is it destructive, does it touch secrets, does it exfiltrate, is it irreversible ^[4]. Each returns its own probability. Code outside the model combines them into deny, review or allow ^[4]. "Model gives calibrated local judgments; code holds the control flow," the author wrote in the dev.to post ^[7].

Routing uses the same division of labour in two stages. `choice` competes across the shortlist first, then `noul` verifies the winner ^[6]. The reported figure is a conditional Hit@1: it counts only the queries whose gold candidate was already in the shortlist, and whether Jev then picked it ^[12]. It isolates the selector from retrieval ^[12]. Both lessons come from one model build, jev-1.13.0, run throughout ^[1].

The total cost checks out. About 22,500 calls consumed 52.2M input tokens, and at the posted $42 per billion that bills at $2.19, the total the post reports ^[1][3][26]. The per-call number needs a caveat. 52.2M tokens over 22,500 calls averages roughly 2,320 tokens a call ^[27], or about $0.0001, some 2.4 times the advertised $0.00004 ^[2][28]. That advertised price corresponds to a state and question set of about 950 input tokens ^[29]. Short states come in under the average: the InjecAgent run billed $0.0213 across 1,105 calls, about $0.00002 each ^[16][30]. The post notes output-token billing would add slightly if it applies; the model does not write sentences ^[2].

Two of the comparisons are claims about someone else's setup. MetaTool's paper reports 69.1% for ChatGPT on the same "similar choices" subtask, and the author calls the exact setup different and the number a magnitude reference ^[21]. The InjecAgent breakdown matters more for transfer: at threshold 0.5, recall was 98.5% on data-stealing attacks and 80.6% on direct-harm ones, with benign samples averaging 0.03 ^[14]. The spread is about 18 points by attack type ^[34], so a team's own threat mix sets its recall before any threshold tuning does. On the synthesized tool-output injections the same threshold gave precision 97.1, recall 85.0 and a 4.9% false positive rate, with the misses being Cyrillic homoglyphs and payloads disguised as code snippets ^[17]. For the shell-gate false positive rate to transfer, a harness needs a command mix resembling that 130-command risk set and criteria that name the harms it cares about ^[4].

The failure modes are semantic. On the 1,140-sample function-relevance task the model read technically possible as semantically intended, and a generic `requests.get` was judged relevant ^[22]. Ranking errors cluster on near-duplicate tools, and the author's fix is writing explicit `not_for` boundaries into the descriptions ^[21]. Gating has a similar problem: `noul` has no confidence field, so the coverage curve proxies certainty with max(p, 1-p) ^[11]. A policy built on that figure rests on a quantity the harness computes for itself. The model is text only, and the author reports English as primary with CJK accuracy dropping ^[10].