GitHub's ModernBERT classifier replaces an LLM in secret scanning at over 98% lower serving cost
GitHub says a fine-tuned ModernBERT classifier cut median post-push secret-scan time by 86% and model serving costs by over 98% against its LLM pipeline. The speed also let it extend generic secret detection to push protection.
The Watch · Security desk
Three figures from GitHub. Median post-push scan execution time cut by 86% compared with the previous LLM pipeline. Precision 13% higher in an offline evaluation of post-push scanning on the same public traffic. Repository owner-weighted alert acceptance up 11.5% in a customer A/B experiment.
| Figure | Value | Note | Claim |
|---|---|---|---|
| Cut in median post-push scan time vs LLM pipeline | 86% | 1 | |
| Higher precision than LLM pipeline, post-push | 13% higher | offline evaluation, same public traffic | 9 |
| Increase in alert acceptance, customer A/B test | 11.5% increase | weighted by repository owner | 10 |
What happened
- The pipeline starts with regexes that flag credential-shaped strings, then a fine-tuned ModernBERT model scores each one using file path, credential type, variable names and nearby code or text.
- In an offline test of post-push scanning on the same public traffic, GitHub measured 13% higher precision and 19% higher recall than the LLM pipeline.
- Training data came from product feedback, regex-flagged repository examples and synthetic cases for thin or difficult categories, with human validation and annotation.
Why it matters
- capability Generic secrets that can resemble build identifiers or sample values can now be scored before a push is accepted, as well as after the push lands.
- exposure Developers who push credential-shaped strings face a block whenever the classifier's score meets the threshold, so a false positive now stops work in the moment.
- constraint The published accuracy figures cover post-push scanning only, and all of them are GitHub's own, so block precision at push protection has no public number.
Push protection checks changes before a push is accepted [4], so a model at that stage sits in the developer's path. We think that is why speed decided the design. Help Net Security's summary of GitHub's write-up says the classifier's speed is what let GitHub extend generic secret detection to that stage [3]. The speed comes from size: ModernBERT classifies short sections of code and text [18], and GitHub exported the model to ONNX with 8-bit quantization for faster inference [14].
An 86% cut leaves post-push scans at 14% of the old median time, about one-seventh [19]. A cut of more than 98% leaves serving cost under 2% of the LLM pipeline's, a factor of more than 50 [20]. Both figures are GitHub's, relayed by Help Net Security [1][2].
Generic secrets are the hard case. Structured credentials often have a recognizable prefix, length, character set or checksum, while generic ones can resemble build identifiers, asset hashes and sample values [7]. Batu Ozturkler and Shengyu Fu said, "Adding more rules can suppress some of them, but each exception risks hiding a real secret." [8]
The threshold decides what happens next. A candidate that meets it triggers an alert after a push or a block during push protection [6]. A false positive at push protection therefore interrupts a developer in the moment, where after a push it is an alert someone triages later. GitHub selected thresholds in offline evaluation and validated them on traffic resembling production before rollout [15].
The evidence has limits. The offline gains were measured for post-push scanning [9]. The write-up does not say whether they are relative or percentage points, and it includes no separate figures for push protection. GitHub grouped rows sharing a repository, candidate value or example before splitting the data, so near-duplicates stay out of both training and evaluation [13].
The customer experiment counts alert acceptance, weighted by repository owner so customers who resolve large batches at once do not dominate [11]. Acceptance shows which alerts customers act on. It cannot show secrets the scanner missed. The authors said, "Higher precision means fewer distracting alerts. Higher recall means more real secrets are caught," [17]
What to watch
- Whether GitHub publishes precision, recall or block-rate figures for push protection, which the current figures do not cover.
- GitHub's planned feedback loop: production misses and false positives going into training data, and any updated metrics that follow.
- GitHub's work on a single-step architecture that combines secret detection and extraction, which would simplify the regex-then-classifier pipeline.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence55
- Adoption50
- Hype gap+10
- Incentives60
- Confidence60
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
GitHub's new credential classifier pipeline cut median post-push scan execution time by 86% compared with its previous LLM pipeline.
- [2]
The new pipeline cut model serving costs by more than 98% compared with the previous LLM pipeline.
- [3]
The classifier's speed enabled GitHub to extend generic secret detection to push protection.
- [4]
Push protection checks changes before a push is accepted; post-push scanning examines repository content and creates alerts for potential secrets that need investigation and remediation.
- [5]
The pipeline starts with regular expressions that identify strings resembling passwords, access tokens and other credentials; a fine-tuned ModernBERT model then evaluates each candidate using its file path, credential type, variable names and surrounding code or text.
- [6]
The classifier assigns a score estimating the likelihood that a candidate is a real secret; candidates that meet the scanner's threshold can trigger an alert during post-push scanning or a block during push protection, and likely false positives are filtered out.
- [7]
Structured credentials often have a recognizable prefix, length, character set or checksum, while generic secrets can resemble build identifiers, asset hashes, sample values and other non-sensitive strings.
- [8]
"Adding more rules can suppress some of them, but each exception risks hiding a real secret."
- [9]
In an offline evaluation of post-push scanning on the same public traffic, the new pipeline achieved 13% higher precision and 19% higher recall than the previous LLM pipeline.
- [10]
A customer A/B experiment recorded an 11.5% increase in repository owner-weighted alert acceptance.
- [11]
GitHub weights alert acceptance by repository owner to limit the influence of customers who resolve large batches of alerts at once.
- [12]
GitHub built its training and evaluation datasets from product feedback, repository examples identified through regex scanning, and synthetic examples covering cases with limited data or difficult distinctions, with human validation and annotation supporting labeling.
- [13]
Before splitting the data, the team grouped rows sharing a repository, candidate value or example to prevent near-duplicates from appearing in both training and evaluation.
- [14]
The trained model was exported to ONNX format with 8-bit quantization to support faster inference.
- [15]
The team used offline evaluation to select classification thresholds, then validated performance on traffic resembling production use before rollout.
- [16]
GitHub plans to use production feedback to identify missed secrets and false positives and feed them into training and evaluation data, and is investigating architectures that combine secret detection and extraction in one step.
- [17]
"Higher precision means fewer distracting alerts. Higher recall means more real secrets are caught,"
- [18]
ModernBERT handles classification over short sections of code and text, supporting the throughput and response times needed for scanning at GitHub's scale.
- [19]
An 86% reduction in median post-push scan time leaves the new pipeline at 14% of the old median time, roughly one-seventh (about 7.1x faster).
- [20]
A serving-cost cut of more than 98% leaves the new pipeline at under 2% of the old cost, a factor of more than 50.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- helpnetsecurity.comGitHub adopts cheaper AI model to catch secrets developers leave in code
1 article · October 14, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- Developer securityFollow
- Replacing LLMs with small modelsFollow
- Secret ScanningFollow
Entities
- Shengyu FuFollow
- Help Net SecurityFollow
- Batu OzturklerFollow
- ModernBERTFollow
- ONNXFollow
- GitHubFollow