Security1 publisher2 min readPublished
Judge model choice swings AI-Infra-Guard's false positive rate fifteenfold
Tencent's Zhuque Lab has put its AI asset scanner on GitHub for nothing. The skill auditor asks a language model whether code looks malicious, and the lab's own benchmark puts the false positive rate anywhere between 1.2 and 18.67 percent.
The Watch · Security desk
What happened
- Tencent's Zhuque Lab has released AI-Infra-Guard, a free scanner that fingerprints running Ollama, vLLM and ComfyUI services and matches what it finds against more than 1,600 known CVEs.
- A second layer inspects MCP servers and agent skills across 14 categories of risk, and it also runs jailbreak evaluations against a target model.
- Zhuque Lab scores the skill judgment against SkillTrustBench, 5,520 human-labeled samples in nine risk categories, where false positive rates run from 1.20 percent to 18.67 percent depending on the judging model.
- The open-source build has no authentication mechanism, and the repository warns operators not to put the platform on the internet.
Compiled by The WatchSomething wrong?How this is made
Why it matters
- decision Picking the judge model is a triage-budget decision made once at install time, and the CVE half of the tool offers no equivalent lever because its errors come from fingerprinting rather than interpretation.
- exposure Stood up without the proxy, the scanner becomes the softest box in the estate it was bought to inventory, since it answers to anyone who can reach the port and holds credentials for every model it tests.
- constraint A pass from the MCP scanner cannot be used as sign-off, which keeps a human reviewer on exactly the hostile-server checks teams deploy the tool to absorb.
The two halves of the platform fail differently, and Zhuque Lab says so in plain terms. Version matching holds no view about intent. "CVE version-matching isn't intent-based at all," the AI-Infra-Guard team told Help Net Security. "FPR there is purely a function of fingerprint accuracy." [9] The skill auditor is a language model reading a file and forming a judgment [4], and its error rate belongs to whichever model the operator wired in [6].
That spread is 15.6x wide: 18.67 divided by 1.20 [20]. Put it in queue terms. Run a thousand clean skills past the low-end judge and roughly 12 come back flagged; the same thousand past the high-end judge come back with about 187 [21]. Those 175 extra reviews are a configuration artefact, and Help Net Security reckons that at the high end close to one flag in five wastes somebody's afternoon [7]. The figures are Zhuque Lab's, measured on Zhuque Lab's benchmark of 5,520 human-labelled samples [5]. No independent evaluation is in the record.
The injection problem is structural to the design. Both scanning components read tool descriptions and skill files that the server under test controls [10]. Release 4.1.9 puts that content into an explicitly delimited block, structurally separated from instructions, and tells the agent to treat the block as data to analyse and never as commands [11][12]. Zhuque says that "cuts naive-to-moderate injection significantly" but is "a mitigation, not a formal guarantee," adding: "we don't claim it's unbreakable, and we'd say that about any LLM-driven agent" [13]. Help Net Security's read is that a clean result on a suspicious MCP server is one input, not a clearance [14].
The authentication gap has a same-day fix. Zhuque calls this a single-operator tool: no login, no RBAC [16]. The documented answer is an nginx reverse proxy with Basic Auth or an IP allowlist, plus normal firewall rules [17]. That is the login layer, and standing it up is somebody's ticket. ICBC, China Merchants Bank, China Telecom, Lenovo, vivo and Bilibili are named as users [8], and those are estates where an unauthenticated port holding the API keys for every model the scanner evaluates [18] does not stay quiet.
AI-Infra-Guard is free on GitHub [19]. The two costs it carries are the reading its judge model generates and whatever a team puts in front of the port.
What to watch
- Per-model false positive figures from Zhuque Lab, which would let teams pick a judge on data rather than on whatever API key they already hold.
- An authentication mechanism landing in the open-source build, replacing the reverse-proxy recommendation with something shipped by default.
- A published bypass of the 4.1.9 delimited-block defence, which would move the MCP scanner from mitigated to attacker-steerable.