Build1 publisher2 min readPublished
Booting a real server flips 147 of 627 agent patches from pass to fail
NVIDIA's SWE-Serve scores the same 627 patches twice on 19 SGLang tasks, once with the live-serving tests and once without. The pass rate falls from 69.4% to 45.9%, and 242 of the 276 live tests came from SGLang itself.
The Engineer · Build desk
What happened
- SWE-Serve turns 83 merged SGLang pull requests into 53 executable tasks across six inference-engineering families, built with input from the SGLang team.
- On the 19 tasks that include live-serving checks, the same 627 patches passed 45.9% of the time under the complete verifier and 69.4% with the live tests excluded, a swing of 147 patches.
- The median reference solution modifies 553 lines across seven files, and a typical verifier pairs seven tests for new behavior with 10 regression tests.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint The 45.9%-versus-69.4% comparison exists only where a server boots, which is 19 of the 53 tasks. Quote it as a property of the whole benchmark and you are stretching the measurement past what was run.
- cost Seeing these failures at all takes a machine that can load the weights, and 41 of the 53 tasks want an H100, so a CPU-only pre-merge gate cannot produce this signal.
- capability Because 242 of the 276 discriminating tests already existed in SGLang, a team can recover most of the same signal by promoting its own end-to-end tests into the agent's verification loop instead of waiting for a benchmark.
- precedent Agent vendors reporting one pass number now have to contend with a published case where identical patches score 23.5 points apart depending on which tests ran. The composition of a verifier is a reportable detail.
69.4% of 627 patches is about 435 passing without the live checks. 45.9% is about 288 passing with them. The difference is the 147 patches NVIDIA reports changing from fail to pass once the live serving tests were excluded, and 147 of 435 is 34% [4][19][20][21].
What those tests do is start a server and load real weights. One task asks the agent to add serving support for Qwen3.5. Both the 0.8B dense model and the 35B-A3B mixture-of-experts model have to load and serve through the normal SGLang interfaces on a single H100 [12]. Its verifier checks model registration, configuration and weight loading, image and video inputs, OpenAI-compatible requests, native batched generation, log probabilities, and execution through the MoE model's routed experts [13]. On the Gemma 4 MoE task, 16 of 33 patches passed every other check and failed at least one live serving test: model loading, expert routing, text and image serving, batched generation with correct ordering and log probabilities [15].
Where those tests came from decides whether the 23.5-point spread transfers [22]. Of the 276 live serving tests across the 19 tasks, 242 are sourced or adapted from SGLang, and 34 were written because no suitable upstream test existed [14]. So roughly 88% of the discriminating tests already lived in the project's own suite [23]. For the same ratio to appear in another repository, two things have to hold. The code has to have behavior that only exists after weights load and a server accepts requests. And the pre-merge gate has to be skipping the tests that exercise it.
Scope is narrow by design. Nineteen of the 53 tasks start a real server and three enforce a calibrated performance gate on an H100 [11]. The first release does not evaluate other inference engines, multi-GPU execution or multi-node serving [7]. Twelve tasks run on CPU and the other 41 use a single H100 [6][18]. That is a familiar hardware requirement for a benchmark published on NVIDIA's developer blog.
The tasks themselves are sizeable: the median reference solution modifies 553 lines across seven files, and a typical verifier has seven tests for the new behavior and 10 regression tests [10]. The agent gets an instruction and a containerized SGLang checkout from before the target change, and its patch is judged against a hidden verifier on the declared hardware, not against the reference implementation [9]. NVIDIA's post is direct about what a pass means: the tests "don't establish that an agent patch or benchmark reference solution is deployable, ready to merge, or endorsed by SGLang maintainers" [16].
What to watch
- Whether the 34 SWE-Serve-authored live tests get upstreamed into SGLang. That would put the signal inside ordinary project CI.
- Leaderboard submissions that publish both rates, with and without live checks, instead of a single pass number.
- Results from a codebase outside SGLang. That is the evidence that would show whether one in three is an inference-serving figure or a general one.