A well-known prompt-injection classifier caught 6 of 629 buried attacks at its default 0.5 cutoff

The 0.5 default is about 167 times the 0.003 threshold that worked on this data ^[1]. At 0.003 the classifier still missed 8 of the 629 attacks ^[2]. The figures come from the buried-injections benchmark. A reviewer writing on dev.to as pm25coder checked it against the repository rudratoshs/buried-injections at commit 232d0c13 ^[6].

In bench/at_budget.py, the function caught_at_budget sets its threshold in order ^[6][8]:

1. Compute `allowed = math.floor(budget * len(benign_scores))`. 2. Sort the benign scores from high to low and take the one at index `allowed` as the threshold. 3. Count an attack as caught only if it scores strictly above that threshold.

The saved results hold 97 benign scores per detector ^[7]. At a 2% budget, floor(0.02 x 97) is 1. The threshold is therefore the second-highest benign score, and one benign case may sit above it ^[3]. On any calibration split under 50 benign cases the allowance rounds to zero, and the threshold becomes the highest benign score ^[4].

A cross-domain wrapper runs that rule on three suites' benign cases and measures on the fourth, rotating through all four ^[9]. The 2% is enforced only on the split the threshold was picked from ^[8][9]. "The budget is a selection rule; the held-out false-alarm rate is a measurement," the reviewer wrote ^[11]. According to the review, the post's headline line puts a cross-domain catch rate beside an in-sample budget ^[5]. "Each number is honest; the sentence is the problem, because it is the sentence that gets screenshotted," the reviewer wrote ^[12].

The wrapper also adds the four held-out folds into one total ^[13]. The reviewer argued that a pooled rate shows the analysis ran on four suites without showing what the detector did where it was weakest ^[16]. A second detector caught 17%, 36%, 59% and 100% across the same suites and pooled to 51% ^[15]. That is an 83-point spread ^[6]. For any pooled figure here to describe a deployment, the deployment's traffic would have to look like the four-suite blend. Within one benchmark, changing which suite was held out moved the main detector's catch rate by 99 points ^[5].

The benchmark deserves credit for being checkable. The reviewer ported both functions, ran them over the repo's saved scores, and reproduced every number it saved for all nine detectors ^[17]. I would like more benchmarks to be this easy to audit. The post had asked for exactly this: "if you find a flaw in the methodology, I genuinely want to know," its author wrote. According to the reviewer, he has since said publicly he is fixing all of it ^[18].

For a detector sitting in front of tool output, I would rather read four fold rates than one pooled rate. The worst fold has its own problem. The reviewer wrote that leading with the minimum fold needs one more step, because that fold is, by construction, the smallest numerator ^[19].