SWE-Serve tests whether an agent’s patch works behind a live server

SWE-Serve evaluates coding agents on changes to SGLang, an inference engine for serving language models. Its checks include real server requests, not only local tests. NVIDIA reports that removing live-serving checks raised the measured pass rate from 45.9% to 69.4% across the examined patches. A patch can look correct locally while breaking the path users actually call.

NVIDIA · Sep 23, 2026

Why it matters

Agent evaluations should exercise deployment boundaries when those boundaries are part of the feature.

More context

Inspect the public benchmark before adopting its score. The first release contains 53 tasks: 12 use CPUs and 41 require a single H100. Try a task matching your serving changes and compute budget.

Keep in mind

The reported comparison covers 19 live-server tasks and 627 patches. It is vendor research, limited to SGLang and single-node scenarios; passing does not establish merge readiness.

Original read