Databricks' LLM triage stops short of a malicious verdict on 14.4% of npm malware that bundles a binary
Databricks says its LLM pipeline fails to reach a decisive malicious verdict on 14.4% of npm malware that bundles a binary, against 6.2% for source-only packages. Those binary packages are 432 of 11,413 scanned versions, and Databricks tests feeding the model pre-computed static evidence as the fix.
The Engineer · Build desk
Bar comparison: binary-bearing npm malware packages fail to reach a decisive malicious verdict 14.4% of the time, against 6.2% for source-only packages, across OSV-confirmed npm malware.
OSV-confirmed npm malware with no decisive malicious verdict
| Item | Value | Claim |
|---|---|---|
| Binary-bearing packages | 14.4% | 5 |
| Source-only packages | 6.2% | 5 |
What happened
- Source-hidden or remotely fetched payloads make up 96.2% of the OSV-corroborated dataset, so bundled binaries are the minority case.
- Databricks built five payload types as native Node.js addons, and its sysmon stealer runs from a readable baseline to a hardened build with stripped symbols and XOR-encoded strings.
- The panel was nine models from seven providers: seven budget models, four of them open-weight, and two higher-capability baselines, plus Claude Opus 4.6 run only as a cost reference.
- The OSV-confirmed packages that ship a binary use PE, ELF (all stripped) and Mach-O formats.
Why it matters
- Who's exposed exposure A clean-looking wrapper around an opaque binary is the configuration Databricks says can tip a screening model toward a benign call.
- What's now possible capability Teams on closed, hosted models can add domain knowledge through the prompt, without the retraining that instruction-tuning needs as threats shift.
- Who has to decide decision Teams screening with LLMs have to decide whether bundled-binary packages get their own prompt path with static evidence added, since the same pipeline fails to decide on them more than twice as often.
- What it limits constraint The remedy is only as good as the static extractors behind it, and the vendor's claim of gains for budget and frontier models rests on results not in the text available.
The 14.4% counts packages where the pipeline did not reach a decisive malicious verdict [5]. That set is wider than benign calls. Databricks names a skew toward a benign verdict as a potential outcome when a stripped binary sits behind clean source and metadata [15]. The measured quantity is the 8.2-point gap in decisive verdicts [12], about 2.3 times the source-only rate [13].
Databricks attributes the rarity of binaries to cost. An adversary shipping JavaScript can assume the target already has the runtime, while a binary must be compiled for a range of target machines [16]. The scanner's static stage runs YARA on the tarball, Semgrep and the npm registry API on the JavaScript and metadata, and a binary-analysis module on the compiled artifact. It then builds a structured prompt, and the model can request more files over up to three rounds [4]. A stripped binary carries no source for those tools to read [14].
The fix moves expertise to inference time. Pre-computed static evidence goes into the prompt before the model reasons [18], and Databricks says this improves detection for budget and frontier models alike [19]. In our view the limit follows from the design. The evidence comes from the static stage, so whatever the extractors cannot recover from a hardened binary never reaches the model.
The test was built to be hard on exactly that point. Databricks picked single-byte XOR on purpose: a stronger cipher such as RC4 would leave structural traces in the disassembly that a model could pick up, and XOR leaves none [8]. Each crafted package also wrapped its binary in a professional package.json, README, JavaScript bindings and binding.gyp [7]. That is good adversarial design: the obfuscation removes the cues an extractor would otherwise hand the model.
The rates are specific to this pipeline. The 14.4% and 6.2% come from OSV-confirmed malware run through this scanner [5]. For them to transfer, a team's static stage would need to resemble that one. Its attackers would also need a similar binary share, and Databricks measured 3.8% in samples collected since November 2025 [1][3]. The crafted payloads are lab builds that Databricks says mirror collected malware [11].
The post's evaluation results, including the size of the injection gain, are not in the text available.
What to watch
- The post's evaluation results: how much injected evidence improves detection, split between the budget models and the two higher-capability baselines.
- Whether outside teams can rerun the crafted payload set and scanner, which the post describes as built to support the research community.
- Whether hardened builds beyond single-byte XOR, such as RC4-style ciphers, change how much the injected static evidence can recover.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence50
- Adoption
- Not scored
- Hype gap+15
- Incentives60
- Confidence55
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
Databricks has been collecting npm malware samples since November 2025 and scanned 11,413 package-versions corroborated through OSV npm malware advisories.
- [2]
Most malware on the npm registry hides its payload in the package source or fetches it from an external server, accounting for 96.2% of the corroborated dataset.
- [3]
The remaining 3.8% of the corroborated dataset, 432 package-versions, bundle a binary.
- [4]
Databricks' scanner runs a static stage with YARA on the tarball, Semgrep and the npm registry API on the JavaScript and metadata, and a binary-analysis module over the compiled artifact; it assembles a structured prompt and the LLM returns a verdict with reasoning, requesting additional files over up to three rounds.
- [5]
Across OSV-confirmed npm malware, binary-bearing packages fail to reach a decisive malicious verdict more than twice as often as source-only ones (14.4% vs 6.2%).
- [6]
Databricks built five payload types of native Node.js addons (.node files); the sysmon stealer spans an obfuscation ladder from a readable baseline to a hardened build with symbols stripped and strings XOR-encoded, while the other four types appear at a single obfuscation level.
- [7]
Each crafted package wraps the binary in a professional package.json, README, JavaScript bindings, and binding.gyp.
- [8]
Databricks chose single-byte XOR deliberately, since unlike a stronger cipher such as RC4 it leaves no structural markers in disassembly for a model to catch.
- [9]
Databricks evaluated nine models across seven providers: seven budget models, four of them open-weight, and two higher-capability baselines; it also ran Claude Opus 4.6, the most expensive model, only as a cost reference.
- [10]
Across the OSV-confirmed npm packages shipping a binary, the formats are PE, ELF (all stripped), and Mach-O.
- [11]
Databricks says the crafted payload set mirrors collected malware.
- [12]
The gap in decisive-verdict failures between binary-bearing and source-only packages is 8.2 percentage points.
- [13]
The binary-bearing failure rate is about 2.3 times the source-only rate.
- [14]
A stripped, opaque binary carries no source for tools to read.
- [15]
Databricks says a stripped, opaque binary can be nested within a facade of clean source and metadata and potentially skew an LLM towards a benign verdict.
- [16]
Databricks says packages with bundled binaries are the rarest and hardest for the pipeline to triage: an adversary distributing malicious JavaScript can assume the target already has the runtime, whereas a binary must be compiled for a range of target machines.
- [17]
Prior work instruction-tunes open-source models on expert-authored data; Databricks says that route cannot reach closed, hosted models and requires retraining as the threat landscape shifts.
- [18]
Databricks supplies the expertise at inference, as pre-computed static evidence placed in the prompt before the model reasons.
- [19]
Databricks says its purpose-built, cost-effective encoding of domain knowledge improves detection quality for both budget and frontier models alike.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- databricks.comClosing the Binary Blind Spot: Domain-Knowledge Injection for LLM Malware Triage in npm Packages
1 article · October 16, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- LLM malware triageFollow
- Binary obfuscationFollow
- npm Supply-Chain SecurityFollow