Jev's confidence scores cleared 86.5% of graded federal RFQs for automation at 96.7% precision

An automation boundary is a single number. Rows above a confidence cutoff go straight into the CRM. There a hardware label can trigger a distributor price lookup, and a services label starts a statement-of-work workflow ^[5]. Rows below the cutoff go to a person. In the benchmark author's dev.to write-up, a cutoff counts only if it holds 95% precision on quote-graded rows ^[9]. Jev's expected calibration error was 0.049 ^[8].

At Jev's 0.94 cutoff, the 641 accepted rows hold about 620 correct labels and 21 wrong ones ^[1]. Across all 741 gold rows, its 91.9% accuracy works out to 681 correct and 60 wrong ^[2]. So about 39 of those 60 errors land in the 100 rows sent to review ^[3]. With 620 hits in 641, the Wilson 95% lower bound comes to about 95.0% ^[4]. The cutoff clears the target with almost no margin. The author reported that the bound stayed at or above 95% along the cutoff envelope ^[10].

Qwen ran under a different harness. It was prompted with a strict JSON schema and its answers were mapped onto the shared taxonomy, while Jev and Laya got byte-identical typed-question bundles ^[18]. Of its 12,000 responses, 69 were permanently malformed ^[4]. If the 97.8% coverage figure is measured on the same 741 rows, the high bucket holds about 725 rows and roughly 72 wrong labels ^[5]. That is about 3.4 times Jev's 21 ^[6].

Laya failed in a different way. It produced 705 distinct confidence values on 741 rows, but they were not ranked well enough to reach the target at any useful cutoff ^[12]. Qwen offered three places to cut and none worked. Laya offered hundreds, poorly ordered. "A classifier that is slightly more accurate but badly calibrated still leaves you without a defensible automation boundary," the author wrote ^[13].

I'd accept that verdict for this reseller's queue. Whether it carries anywhere else depends on the gold. The gold covers quoted opportunities only, which are a selected subset, and Hardware is 77% of the single-class rows ^[7]. A rule that always answered Hardware would score about 77%, roughly 571 of 741 ^[7]. Laya's shipped defaults sit about one point above that ^[8]. The 634 fulfillment labels include an unvalidated heuristic ^[7]. The public repository holds aggregate results only. Gold labels and per-row predictions stay private ^[3].

The selection rule is the part I would copy. Nine Jev variants changed the question structure and the input, and one added a 1,500-character attachment excerpt ^[14]. All nine scored between 91.2% and 91.9%, or 676 to 681 correct ^[14]. The whole spread is five rows ^[9]. The author picked the winner, A-S2, by a rule written before the experiments: accuracy first, then coverage at the bounded precision cutoff, then the cheaper input ^[15]. The excerpt existed on only 5.6% of rows, and the question bundle itself consumed most of the tokens ^[16]. "That is not proof that attachments never help. It is evidence that this particular excerpt strategy did not," the author wrote ^[17].