Syft and Trivy list under half of JavaScript lockfile packages in a 3,326-repository study

Most of the JavaScript gap is devDependencies ^[6]. Syft excludes them by design, but only for some package managers. On one npm project it reported all 2,249 production packages and none of the 4,930 dev packages. A Yarn project and a pnpm project came out at 100% for both ^[6]. That npm SBOM listed about 31% of its lockfile ^[1]. The authors call Syft's behaviour a design choice applied inconsistently ^[8]. A team that already keeps dev tooling out of its SBOM would have made part of that choice anyway. It would still get a different scope depending on the package manager ^[6].

Trivy's gap is a different kind of failure. It also excludes dev dependencies by default, but the exclusion stops at the first level ^[7]. A transitive dependency of a dev package can then show up attached to the root as if it were a top-level production dependency ^[7]. The authors call that an implementation error ^[8]. cdxgen reached 98.72% on JavaScript ^[3]. Its small remaining gap came from a hard-coded list that skips any directory named examples, docs, tests or .github ^[9].

The tools also disagree on names. cdxgen and Trivy report an npm alias under the alias, and Syft reports it under the real scoped name ^[10]. One git-pinned dependency came back as a commit hash from one tool, a tarball URL from the second and a semantic version from the third ^[10]. Vulnerability matching runs on those identifiers. The same code scanned with two tools can therefore produce two different vulnerability lists ^[10].

Several things would have to hold before these percentages describe your own pipeline. Your repositories would have to resemble the sample, which was popular GitHub projects with over 1,000 stars, each measured against a freshly regenerated lockfile ^[1]. The authors did not measure how that threshold affects coverage ^[12]. You would also be running Syft 1.38.2, Trivy 0.68.2 and cdxgen 12.0.0 ^[2]. Trivy 0.75.0 and Syft 1.54.0 both shipped on 1 October ^[14]. That Syft release is sixteen minor versions past the one tested ^[2]. Coverage has moved between versions before. An earlier 2026 study on Trivy 0.66.0 and Syft 1.33.0 got empty SBOMs for package-lock.json, and a few releases later the same tools produced partial ones ^[13]. The field-completeness figures check whether a field is present, not whether its content is right ^[12].

The dev.to commentary's author, who wrote that they have not reproduced either paper ^[17], put the conclusion this way: "an SBOM generated in CI is evidence about the tool you ran, at the version you ran it, under the flags you set" ^[15]. The author argues that the generation recipe is what goes under version control and gets defended in an audit ^[16]. I think that is the right tradeoff for anyone shipping JavaScript. In this data, both the package manager and the tool version changed the output ^[6][13]. If the generator and its version are pinned in the repository, a change in coverage shows up as a diff someone can review.

Pengyin Shan's audit is carefully built for the question it asks. Six research software projects each came in nine variants ^[18]. They were: no signal, a valid SBOM, a valid signature, a valid attestation, two with wrong-issuer material, one with every signal and one with contradictory metadata ^[18]. Trials ran across three models and two harnesses in network-less containers, with file access and commands logged ^[18]. Shan deposited the protocol with a DOI before the first trial ^[18]. Because the protocol was registered first, the variants could not be chosen after seeing which ones the assistants noticed. With a frontier-model supplement added, the trial count reached 2,114 ^[19].