Build1 distinct publisher3 min readUpdated
A developer's OWASP Benchmark test showed the model's failure was optimism, not ignorance. His fix flips the default on unknown functions, which moves the error rather than removing it.
The Engineer · Build desk

Compiled by The EngineerSomething wrong?How this is made
A developer testing an open-source VS Code extension called EdgeGuard against the OWASP Benchmark for Java found that the model's failure mode was not ignorance of SQL injection but optimism about code it could not see [1][2][7]. When tainted HTTP input passed through a helper function whose body was not in the prompt, the model assumed sanitisation and returned SAFE while the value still reached a SQL sink [4][5][6].
The example in the write-up is three lines: a parameter read from an HTTP request, a call to `DatabaseHelper.doSomething(param)`, and the result concatenated into a `SELECT` string [3]. According to the author, the model could see the untrusted input and it could see the query construction, but not what `doSomething()` did, so it filled the gap with the guess that this was probably an internal helper that sanitises input [4][5]. The result was a false negative on a live data flow [6].
The remedy described is a change of default rather than a change of information. EdgeGuard now supplies explicit security assumptions and asks the model to preserve taint unless there is evidence of sanitisation [8], with the specific rule that data entering an unknown function stays tainted [9]. Output moved from a SAFE or VULNERABLE label to a numbered evidence chain: input received, taint preserved across the unknown helper, tainted value concatenated into SQL, conclusion of potential injection [10][11].
That is cheap and directionally correct, and it is worth being clear about what it buys. Nothing in the described change resolves the callee. The tool has traded a false-negative bias for a false-positive bias, and every function body absent from the prompt is now assumed hostile. Whether that is an improvement depends entirely on how large the "unknown function" category remains once a call graph is actually walked.
The second half of the problem is cost. Sending every method to an LLM is wasteful because most of a large codebase is getters, setters, `toString()`, `equals()`, CRUD, data mapping and utilities, and it runs into API rate limits and API cost [12][13]. EdgeGuard adds a local static screening stage that parses code inside VS Code and flags functions with database sinks, file-system access, process execution, network boundaries, user-controlled input, dangerous APIs or missing validation [14][15]. The author's framing is that this makes the LLM a reasoning engine rather than the first line of analysis [16].
At project scale, a workspace scan of the OWASP Benchmark Java project discovered 7,536 functions across 2,771 files [17] and reported 2,145 potential defects [18], including SQL injection, command injection and unsafe input-to-sink flows [19]. That is roughly 28 percent of all functions in the project carrying a finding [20].
The write-up reports those counts but no precision or recall against the benchmark's own labels [21], which is the number that decides whether 2,145 items is a security queue or a backlog nobody will read. Two things to watch: whether the taint-preservation rule gets paired with real callee resolution, so the unknown-function category shrinks; and the recall of the local screening filter, because any function the local parser skips never reaches the model at all, and its blind spots become the product's blind spots [14].
Follow any of these and your For You feed starts watching them — no settings page required.
Ranked by verification strength, evidence, and original report placement.
EdgeGuard is an open-source VS Code extension built to investigate code from an adversarial perspective, to try to break code before production does.
The author began testing EdgeGuard against the OWASP Benchmark for Java, and reported the AI was too optimistic.
The sample code was: String param = request.getParameter("id"); String bar = DatabaseHelper.doSomething(param); String sql = "SELECT * FROM USERS WHERE ID='" + bar + "'";
The LLM could see the untrusted HTTP input and the SQL construction, but could not see what doSomething() actually did.
The model made the optimistic assumption that doSomething() was probably an internal helper that sanitizes the input.
The result could be a SAFE verdict even though the data was still flowing into a SQL sink, which the author describes as the kind of false negative he wanted to find.
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
Single self-reported build log, no accuracy metrics
All substance rests on one dev.to post written by the tool's own author. The mechanism (optimistic assumption about an unreadable helper producing a SAFE verdict) is illustrated with a concrete code snippet and is plausible, and the scan counts are specific. But the counts are unverified, there is no precision or recall against the OWASP Benchmark's labelled vulnerabilities, no baseline comparison, and no independent replication.
No usage evidence beyond the author's own test
The supplied material shows only that the author published the extension and ran it himself against OWASP Benchmark, Juice Shop and eShopOnWeb. There are no installs, stars, downloads, third-party deployments, contributors or user reports, so adoption cannot be scored without guessing.
Framing outruns the unmeasured accuracy trade
The headline claim of OWASP breaking the reviewer and the fix that follows are presented as a resolved improvement, yet the only quantified outcome is 2,145 potential defects across 7,536 functions with no precision or recall. Preserving taint through every unknown function predictably converts false negatives into candidate findings, so the reported change moves the error rather than demonstrably removing it; the post does not test that distinction.
Author is the tool's creator, self-published
The post is written by the builder of EdgeGuard on a platform with no editorial review, and it functions as a showcase for his own open-source extension. That gives a clear interest in a favourable narrative arc — problem found, problem fixed, scale demonstrated — and plausibly explains why counts are published while accuracy figures are not. No sponsorship, funding or commercial pricing is disclosed in the source.
Clear on what was said, weak on whether it works
Confidence is high that the post says what the ledger records — the snippet, the prompt rule, the screening stage and the counts are all explicit. Confidence is low in the underlying performance claim, because there is one self-interested source, no accuracy metrics, and no adoption signal to triangulate against.
build
Wrapping a web tool in VS Code: four sandbox rules, and two gaps in the published fix1 distinct publisher
build
Your agent needs the API call, not the API key1 distinct publisher
build
A session that read "finished" and "still executing" was a slow queue, not a dropped handshake1 distinct publisher
build
Amazon Q executed code from any repo you opened, and it is not the only one1 distinct publisher
Distinct publishers with included, body-backed reporting in this cluster.
dev.to
1 article · August 19, 2026