benchmark

SWE-Serve

Benchmark that scores coding agents on repository-scale changes to LLM inference-serving code, built from merged SGLang pull requests and verified with hidden test suites, some of which start a real server and load a model.

Known aliases

SWE-Serve benchmark

Relationships

No evidence-backed relationships are recorded.

Current clusters

build1 publisher

Booting a real server flips 147 of 627 agent patches from pass to fail

NVIDIA's SWE-Serve scores the same 627 patches twice on 19 SGLang tasks, once with the live-serving tests and once without. The pass rate falls from 69.4% to 45.9%, and 242 of the 276 live tests came from SGLang itself.

Publishers:developer.nvidia.com

Reality

Evidence58
Adoption30
Hype gap−10
Incentives70
Confidence60