ServerBench

Main results

Latency-qualified dollars per dollar (higher is better) for 11 coding agents on three deployments, against the best of SGLang and vLLM on the same hardware. A system scores only if it passes verification.

Why a new benchmark?

Agent-driven systems optimization is moving fast, but existing benchmarks evaluate individual kernels, code patches, or servers under preset test conditions. That leaves three gaps when the goal is a complete serving system.

ServerBench in three designs

Each task fixes a model checkpoint, a GPU allocation, a coding-session workload, and service requirements. Starting from a minimal Transformers server, an agent gets an 8-hour development budget in an Internet-disabled sandbox. It may redesign execution, batching, memory management, scheduling, and state reuse, as long as it preserves the checkpoint, precision, and inference semantics. Its submission is an implementation plus a session concurrency.

1. Workload: real coding-agent trajectories

Using the AgentX harness, ServerBench replays recorded coding-agent sessions with their request lengths, multi-turn structure, shared prefixes, parallel subagent branches, and inter-turn delays. Prompt and tool contents are reconstructed for privacy while keeping lengths, prefixes, and session topology. Replay is closed-loop: each turn waits on the response it depends on, so serving latency directly governs how much work completes in the measurement window. Shared context rewards reuse, while retained state and new sessions compete for memory and compute.

2. Objective: dollars per dollar

We price each completed request's input and output tokens at fixed API rates, count only requests that meet both the TTFT and TPOT targets, and divide by the GPU cost of the measurement window:

Here $q_i=1$ if request $i$ meets both latency targets, $d$ is the number of GPUs, $r$ the price per GPU-hour, and $W$ the window in seconds. For example, USD 3.65 of qualified token value over USD 0.83 of GPU cost scores 4.40. The objective rewards capacity improvements together with a matching load choice: serving more workload raises value only while requests still meet their latency targets.

3. Validation: beyond correctness

Latency targets are the p90 TTFT and TPOT of an SGLang reference running its model-specific cookbook configuration, frozen before development. After development, the submission is rebuilt in a fresh deployment and replayed for 1,800 seconds on held-out seeds that no development run could draw. Independent checks verify complete prompts, prescribed output lengths, streaming consistency, generation fidelity to the checkpoint, natural EOS stopping, and held-out answer accuracy. A post-run review then inspects code and recorded actions for prohibited shortcuts. Only verified systems receive a score.

Results

We evaluate 11 coding-agent models at maximum reasoning effort on three deployments: Gemma E4B (1×L40), Qwen3.5 9B (2×L40), and Qwen3.8 27B INT4 (2×L40). GPT models run in Codex and the others in Claude Code. Baselines are SGLang v0.5.19 and vLLM v0.28.0 at their best operating points on the same hardware.

What separates strong agents

We analyze the development trajectories of 30 runs with full server execution records, and back up each pattern with controlled comparisons on frozen systems and common seeds.

At fixed GPU cost, more sessions can raise token value, but they also raise TTFT and TPOT through queueing and contention. Strong agents build capacity first and then raise load to match it. Among runs reaching at least 2× the strongest reference, 7 of 17 load decisions improve value on the next measurement, against 2 of 23 for verified runs below the reference. Those runs also reach a usable end-to-end replay sooner, in a median of 0.90 hours versus 2.96.

Sonnet 5.5 on Gemma holds four sessions while it revises the engine, then raises load together with further implementation changes. Its frozen system goes from 1.30 at two sessions to 2.97 at 32. It also knows when to stop: 36 sessions produce more gross value but more TPOT misses, so it submits 32. Weaker runs often move the other way, lowering concurrency to one to relieve contention instead of building capacity.

Useful feedback requires testing a change under the conditions that matter. Among runs reaching at least 2× the reference, 49% of retained design decisions follow improvements on workload-relevant tests, against 25% in failed runs.

Prompt-lookup speculation on Gemma shows how much this matters. Opus 5 tests it on a synthetic random-text probe, sees no benefit, and drops it. Opus 5.5 gets the same null result, but checks it against 60 real replies from replay, fixes its simulator, measures 2.09–2.34 accepted tokens per step, and adopts speculation. When we resume Opus 5 from the same checkpoint with and without feedback from its own replay replies, only the arm with workload feedback re-adopts speculation, and its server delivers 1.13–1.31× the qualified value.

Favorable feedback can mislead too. One Opus 5.5 run on Qwen3.8 27B INT4 raises its development score from 2.20 to 2.99 and validates that configuration on an unusually light development draw, but its qualified share drops from 76.0% to 16.8% in the final replay. A CPU scan of the workload generator over 128 seeds, taking just 7.71 seconds, would have shown that the draw sat near the fourth percentile.

Changes to admission, scheduling, and state reuse create request boundaries and state transitions that need their own checks. At submission time only 5 of 33 entries have recorded checks of natural stopping on the submitted system under load or long context, and related symptoms appear before submission in 8 of the 11 runs that fail verification.

• Admission boundaries. Qwen3.8-Max-0902 tightens admission to protect TPOT and checks only a short prompt. Its policy leaves a range of prompt lengths that can never enter execution, and final evaluation stalls on a 315K-token input.
• Shared state. Kimi K3 retests an isolated long prompt on a fresh engine, but its prefix cache crashes when the same prefix is inserted a second time, and the crash recurs in final evaluation.
• Doing it right. After adding preemption and cache restoration, Opus 5.5 on Gemma shrinks cache capacity to force those paths to run. The stress test exposes invalid outputs from uninitialized cache storage. The agent repairs it and reruns forced preemption against isolated execution. With the repair reverted, a reduced-pool replication passes 0 of 3 tests; with it, 3 of 3.

Takeaways

Coding agents can already design LLM serving systems that substantially beat production frameworks on a realistic workload, but reliable delivery is far from solved. Across trajectories and controlled comparisons, three priorities stand out:

• coordinate capacity improvements with the operating load;
• design experiments around workload characteristics and variation;
• verify the execution paths that system changes introduce, and rerun those checks after every repair.

ServerBench, including the harness, tasks, baselines, and evaluation, is open source. Try it with your own agent at github.com/Infini-AI-Lab/Server-Bench. The full paper is on arXiv (2611.06841).

Citation

If you find ServerBench useful, please cite our paper:

@misc{zhong2026serverbench,
  title={ServerBench: How can your coding agents design specialized LLM serving systems?},
  author={Xinrui Zhong and Zhiming Wang and Yongqi Liu and Zhaohui Wang and Shanglin Yang and Zikun Li and Zhuoming Chen and Beidi Chen},
  year={2026},
  eprint={2611.06841},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2611.06841}
}