LSCBench finds the best coding agent writes working, secure code on 25.1% of tasks
Six coding agents pass 73.6% to 91.3% of LSCBench tasks, but the best writes working, secure code on only 25.1%, NUS and Alibaba researchers found. The prompts never mention security, so green tests say little about it, and a one-sentence reminder lifted three of four models by more than 15 points.
The Engineer · Build desk
Key figures: 66% of 2,700 runs ended with code that works and is still vulnerable; the highest functional pass rate among the six agents was 91.3% of tasks; 57 of 150 scenarios were never secured by any model.
| Figure | Value | Note | Claim |
|---|---|---|---|
| Runs ending in code that works and is still vulnerable | 66% of 2,700 runs | 8 | |
| Highest functional pass rate | 91.3% of tasks | 2 | |
| Scenarios no model secured | 57 of 150 | 9 |
What happened
- LSCBench has 450 tasks built from 150 real vulnerabilities, spread across seven languages and 24 CWE categories, and each task prompt asks only for functionality.
- Of the 150 vulnerability scenarios, 57 were never secured by any of the six models.
- Among working code, server-side request forgery was also secure 3% of the time and information exposure 1%, against 44% for memory safety.
- Appending the phrase "while also being secure" lifted GPT-5.6-Sol from 24.7% to 46.2% but moved DeepSeek-V4-Pro only from 18.0% to 19.1%.
Why it matters
- exposure At the benchmark's 66% rate, a pipeline that gates agent output on functional tests alone would pass about 1,780 vulnerable runs out of 2,700.
- constraint A functional leaderboard does not rank safety. Kimi-K3 tops the functional pass rate and has the lowest secure rate of the six, so choosing a model on test scores can pick the least secure one.
- decision The paper's rerun supports adding a security reminder, but most tasks stayed insecure for all four models rerun, so we would keep attack-replay tests in CI.
In the Java importer case, the agent had to rewrite a missing file. It built the parser on SnakeYAML's default constructor, which, according to the paper (Section 5.4), "honors every !!class tag" in the uploaded YAML [12]. All 14 hidden functional tests passed and all four security tests failed [12]. The agent was Kimi-K3, and Qwen3.8-Max and GLM-5.3 made the same mistake [13]. Claude Fable 5 and GPT-5.6-Sol used SafeConstructor and passed both suites [13].
Nothing in the task pushed the agent toward the safe constructor. The authors wrote that "Every task asks only for functionality, through a specification that never mentions security" [4]. The dev.to write-up that summarizes the paper says this matches how most people prompt a coding agent [21]. The weakest functional score, 73.6%, is about 2.9 times the best secure score of 25.1% [19].
Scoring is strict. A task counts only if two hidden suites both pass, the security suite replays the original attack, and there is no partial credit and no judge model [5]. A pass therefore means the code survived the replayed exploit. The 25.1% transfers to another repo only if three things hold. Your prompts omit security. Your code touches the same kinds of sinks as the 150 real vulnerabilities the tasks come from [3]. And your CI has no attack-replay tests.
Two cautions on reading the charts. The language rates are the share of runs, all six models combined, that ended both working and secure. The weakness-family rates are the share of working code that was also secure [10][11]. The bases differ, so the 44% for memory safety cannot be set beside the 30% for C.
The harness also varies. GPT-5.6-Sol was tested in Codex v0.154.0 and the other five models in Claude Code v2.1.143, each set to its highest reasoning effort [6]. Any gap between GPT-5.6-Sol and the rest mixes model and harness. The paper went to arXiv on October 10, 2026, and its authors include the Qwen team [1]. Qwen3.8-Max is one of the six agents tested [2].
The reminder helped unevenly. GPT-5.6-Sol, GLM-5.3, Qwen3.8-Max and DeepSeek-V4-Pro gained 21.5, 16.9, 15.6 and 1.1 points [18]. Functional pass rates moved by a few points either way [15]. The paper's verdict is that "most tasks stay insecure even with the reminder" [15]. Claude Fable 5 and Kimi-K3 were not in the rerun [16].
The write-up's author suggests a line in AGENTS.md or CLAUDE.md as the nearest everyday equivalent [20]. The paper appended the sentence to each task, so results from a standing instruction file are not reported [20].
What to watch
- A rerun of the reminder test on Claude Fable 5 and Kimi-K3, which the paper left out.
- Results from putting the security line in AGENTS.md or CLAUDE.md instead of the task text, which the paper did not test.
- A replication by a group outside NUS and Alibaba, and one that runs GPT-5.6-Sol in the same harness as the other five.
Clarity's read
What the record supports and how the coverage leans. The claims behind it follow.
Reality
- Evidence58
- Adoption
- Insufficient
- Hype gap+12
- Incentives35
- Confidence60
Claim ledger
Ranked by verification strength, evidence, and original report placement.
- [1]
LSCBench was submitted to arXiv on October 10, 2026 by researchers from the National University of Singapore and Alibaba Group, including the Qwen team.
- [2]
Six frontier agents pass 73.6% to 91.3% of LSCBench tasks functionally, and the best of them writes code that is both working and secure on 25.1%. Qwen3.8-Max is one of the six agents.
- [3]
The benchmark has 450 tasks built from 150 real vulnerabilities, across seven languages and 24 CWE categories.
- [4]
"Every task asks only for functionality, through a specification that never mentions security."
- [5]
A task scores only if two hidden suites both pass: functional tests and security tests that replay the original attack. There is no partial credit and no judge model.
- [6]
All six models ran at their highest reasoning effort. Five ran inside Claude Code v2.1.143 and GPT-5.6-Sol ran inside Codex v0.154.0.
- [7]
Kimi-K3 has the highest functional pass rate and the lowest secure pass rate of the six models.
- [8]
Across all 2,700 runs, 66% ended with code that works and is still vulnerable.
- [9]
Of the 150 scenarios, 57 were never secured by any model.
- [10]
By language, the share of runs that ended both working and secure, all six models combined, was: C and Python 30%, JavaScript 27%, C++ 19%, Java 18%, Go 14%, PHP 10%.
- [11]
The weakness-family chart uses a different base, the share of working code that was also secure: memory safety 44%, injection 28%, server-side request forgery 3%, information exposure 1%.
- [12]
In a Java configuration importer task, Kimi-K3 used SnakeYAML's default constructor, which "honors every !!class tag" in the uploaded YAML. All 14 hidden functional tests passed and all four security tests failed.
- [13]
Claude Fable 5 and GPT-5.6-Sol parsed the same input with SnakeYAML's SafeConstructor and passed both suites. Qwen3.8-Max and GLM-5.3 made Kimi-K3's mistake.
- [14]
The authors reran four of the six models with one sentence appended to every task asking for an implementation that meets the functional requirements "while also being secure" (Section 5.5). Works-and-secure scores: GPT-5.6-Sol 24.7% to 46.2%, GLM-5.3 21.3% to 38.2%, Qwen3.8-Max 22.0% to 37.6%, DeepSeek-V4-Pro 18.0% to 19.1%.
- [15]
In the reminder rerun, functional pass rates moved by a few points either way. The paper's verdict: "most tasks stay insecure even with the reminder", and SSRF and information exposure stay near the bottom.
- [16]
Claude Fable 5 and Kimi-K3 were not in the reminder rerun.
- [17]
About 1,780 of the 2,700 runs ended with code that works and is still vulnerable.
- [18]
The reminder raised works-and-secure scores by 21.5 points (GPT-5.6-Sol), 16.9 (GLM-5.3), 15.6 (Qwen3.8-Max) and 1.1 (DeepSeek-V4-Pro).
- [19]
The lowest functional pass rate (73.6%) is about 2.9 times the best secure rate (25.1%).
- [20]
The write-up's author suggests a line in AGENTS.md or CLAUDE.md is the nearest everyday equivalent of the reminder. The paper appended the reminder to the task.
- [21]
The write-up's author says that a feature-only prompt, with no mention of security, is how most people prompt a coding agent.
Get this ledger inside ChatGPT. Ask about any of our stories and the answer brings its figure, each claim’s verdict and the sources. Free, no account.
Sources
1 independent publisher whose own reporting we read for this story.
- dev.toCoding Agents Pass Up to 91% of Tasks in a New Benchmark. At Most 25% Are Also Secure
1 article · October 14, 2026
Topics and entities
Follow any of these and your For You feed starts watching them — no settings page required.
Topics
- AI BenchmarksFollow
- AI for Code SecurityFollow
- Coding AgentsFollow