SWE-Serve: Benchmarking Agentic Engineering for Production Inference Serving
SWE-Serve：首個針對「生產級推論服務」的 AI Agent 軟體工程基準測試
Traditional SWE benchmarks rarely address the unique complexities of inference serving. SWE-Serve addresses this with 53 repository-grounded tasks derived from production changes in SGLang. Across 11 models and 31 configurations, the best achieved a 75% mean pass@1. However, the benchmark exposes a massive "production correctness gap": end-to-end serving tests rejected approximately one-third of patches that had passed all local functional tests, emphasizing that local success does not guarantee production readiness.
Key points
Fills Inference Engineering Gap
Unlike general coding or kernel-specific benchmarks, SWE-Serve targets repository-scale engineering across the entire inference serving stack.
Grounded in Real SGLang Tasks
Features 53 tasks from actual SGLang production changes across 6 engineering families, executing on CPU or a single H100 GPU.
Quantifies the Production Gap
On 19 E2E tasks, about one-third of patches that passed local tests were rejected by production-level E2E serving tests, dropping pass rates from 69.4% to 45.9%.
Rigorous Multi-Dimensional Evaluation
Utilizes hidden functional and regression tests, calibrated performance gates, and adversarial verifiers to ensure robust evaluation integrity.
How it works
| 傳統基準測試 (Traditional Benchmarks) | SWE-Serve | |
|---|---|---|
| Engineering Scope | 通用程式碼修復 / 單一演算法任務 | 推論服務棧（模型支援、執行期、API） |
| Testing Depth | 單元測試與局部功能測試 | 隱藏測試、E2E 服務測試與校準效能門檻 |
| Task Source | 多個不同開源專案的歷史問題 | SGLang 生產環境中的實際變更 |
| Production Correctness Gap | 未明確區分本地與生產環境差異 | 量化端到端測試對程式碼的拒絕率 (約 1/3 遭拒) |
Why it matters
As AI agents scale to complex systems, local code execution is no longer the final standard. SWE-Serve is the first to measure the "production correctness gap" directly, pushing agents to understand system-level interactions across models, runtimes, and APIs. This shifts the focus from simple local code completion to shipping truly production-ready systems software.
Who it affects
- AI Developer
- AI Researcher
- Product Manager
How to use it
- 1Evaluating the capabilities of coding agents on high-concurrency, high-performance systems like inference engines.
- 2Developing and validating automated integration and regression testing workflows for inference frameworks like SGLang.
Limitations & caveats
- The benchmark tasks are currently grounded entirely in a single repository (SGLang), which may not reflect architectural patterns of other engines.
- Certain tasks require high-end hardware (H100 GPU) for execution, which may limit accessibility for some research teams.
Related
CliffCompaction: Cost-Efficient Context Compaction for Long-Horizon Coding Agents
CliffCompaction：長任務程式碼 Agent 的高效能 context 自動壓縮技術
CliffCompaction is an autocompaction technique for long-horizon coding agents that reduces token costs by up to 50% and prevents context drift by strictly truncating rather than rewriting original context.