CompressionBench

Why compression matters: memory and energy

Memory has become the scarcest part of the AI stack. A model's weights must sit in high-bandwidth memory (HBM) on the GPU, and HBM capacity, not compute, often decides how many GPUs a deployment needs. As AI build-outs absorb supply, HBM and DRAM are in shortage, so every gigabyte a model occupies is a gigabyte that is hard to buy.

Memory is also where the energy goes. LLM inference is increasingly bounded by memory traffic, not arithmetic. In autoregressive decoding, every generated token streams the weights it uses through the memory system, and an off-chip memory access costs far more energy than the arithmetic it feeds.

That makes the number of bytes per weight one of the most direct levers on both the memory a model occupies and the energy cost of every token. A model stored at 2–4 bits per weight instead of BF16's 16 needs 4–8× less HBM for its weights, moves 4–8× fewer weight bytes per token, and can run on fewer GPUs. A 120B-class model takes about 240 GB in BF16; our tightest MoE tasks give the agent 32 GiB. In our tasks, each compressed 120B-class mixture-of-experts model runs on one H200 NVL, where its BF16 reference needs two.

There is a catch. These savings are real only if the model stays useful and its inference path does not spend them again on unpacking and dequantization. Building such a model is substantial experimental and systems work, exactly the kind coding agents increasingly take on. An agent that compresses models well would make every AI system that follows it cheaper to host and more energy-efficient to run. So we ask: can coding agents autonomously build compressed LLMs that remain useful?

Main results

Verified models from six coding agents on three source models, each under a tighter and a looser storage limit. PPL is measured on held-out BF16-generated continuations (lower is better); accuracy averages GSM8K, HumanEval and TriviaQA; TPS is batch-one decode speed. Highlighted: lowest PPL per task.

What an agent needs to compress a model

Existing compression methods provide a rich set of building blocks, but no single method yields a useful model under an arbitrary storage limit and hardware target. To construct one autonomously, an agent needs three abilities.

CompressionBench in three designs

To evaluate these abilities without prescribing how agents compress, we formulate LLM compression as a terminal task, following the outcome-based evaluation of Terminal-Bench and Harbor. A coding agent reads papers, writes code, constructs checkpoints, and runs experiments in an offline container within an eight-hour budget.

1. Task: a complete model under a storage limit

Given source weights, a storage limit $C$, one GPU and a pinned vLLM runtime, the agent may combine quantization, mixed precision, rotations, pruning, low-rank representations and distillation, and must implement whatever inference operations its representation requires. The limit applies twice: to the saved package on disk and to the model storage resident on the GPU during inference, so a small checkpoint cannot cheat by expanding into a full-precision copy at load time.

2. Feedback: complete-model likelihood

A supplied evaluator returns the source-to-compressed perplexity ratio on continuations sampled from the source model:

Here $\widehat{\Delta }$ is the excess cross-entropy over the source model, whose expectation equals the next-token KL divergence from the source. Every evaluation consumes the same eight-hour budget as model construction, so the agent must decide when cheap local evidence suffices and when a complete-model measurement is worth its cost.

3. Validation: the executable artifact

After the deadline, a separate container runs the frozen weights through the submitted inference code on held-out contexts. Both package and resident storage must fit $C$; mean GSM8K, HumanEval and TriviaQA accuracy must retain 80% of BF16; and review checks that tokens and probabilities come from the submitted forward pass. Decode speed is measured and reported, but kept out of the score. Only verified models receive a score.

How agents improve compression

We analyze the agents' recorded searches to explain how they improve compression and where improvement breaks down.

An agent can allocate more bytes to a sensitive component only by saving them elsewhere, or improve how existing codes approximate the weights. In 21 of 24 recorded revisions that change package size by at most 0.01 GiB, agents lower PPL, across all six agents and all six tasks. On the tighter Qwen3.5 task, Kimi changes clipping and GPTQ damping at unchanged package size and cuts feedback PPL from 3.738 to 3.132. Agents also spend storage on metadata deliberately: Kimi shrinks group-scale storage to afford more four-bit weights.

Local metrics are cheap, but they do not always agree with the complete model. Across 79 matched comparisons from 26 runs, reward feedback reverses 17 prior local rankings, across all six tasks and all agents except MiniMax. Agents respond by reverting the candidate, revising its numerical choices, or changing the local validation they use from then on.

Perplexity scores fixed source-generated histories; generation feeds the model's own choices back in. Twelve comparisons from four runs show both newly correct and newly incorrect answers after small PPL changes. Sonnet's versions v3–v5 differ in PPL only in the fourth decimal (3.2572, 3.2531, 3.2525) yet answer 111, 104 and 115 of 150 questions correctly. Pruned models produce repeated or truncated HumanEval completions while preserving likelihood. Consistency tests catch execution bugs too: comparing batched with per-expert inference, MiniMax finds an expert-indexing bug on Nemotron, and fixing it cuts the PPL increase over BF16 from about 11% to 4.5%.

Memory saved today, joules next: execution is the open frontier

Two complete strategies show how far agents go. Sonnet compresses dense Qwen3.8 to 12 GiB with Hadamard-rotated trellis codes, error compensation and Viterbi search, allocating bits from a fitted KL loss model; it beats BF16's accuracy (76.67% vs. 74.00%) at PPL 3.253 vs. 3.221. Opus compresses MoE Qwen3.5 to 48 GiB with rotated GPTQ and expert-specific precision.

The memory savings are already real: verification measures the weights resident in GPU memory, not just the files on disk, so every verified model fits its limit while it runs. Speed and energy savings are not there yet. Sonnet's trellis model improves PPL and accuracy over the released GSQ-RCO quantization, yet decodes at 16.7 tokens/s against the release's 74.6, because it reconstructs weight tiles from trellis codes on the fly. Every agent-built model in the table decodes slower than BF16 at batch size one. Turning fewer bytes into fewer joules per token requires inference kernels that read compressed weights without paying it back in unpacking work. This is the third ability, and the clearest room for agents to improve.

Takeaways

Coding agents can already build compressed LLMs that cut a 120B-class model's GPU memory by about 7× and fit it on a single GPU at 2–4 bits per weight while staying close to BF16, and that beat released quantizations on likelihood. What separates strong runs is not spending but judgment:

• design the representation and its precision allocation together under the storage budget;
• confirm local wins with complete-model feedback before keeping them;
• test generation and execution directly, since likelihood alone hides regressions;
• and co-design the inference path, so that less memory also becomes less energy per token.

CompressionBench, including the tasks, harness and verification, is open source. Try it with your own agent at github.com/Infini-AI-Lab/Compression-Bench. The full paper is on arXiv (2611.06858).

Citation

If you find CompressionBench useful, please cite our paper:

@misc{zhong2026compressionbench,
  title={CompressionBench: How can your coding agents compress LLMs?},
  author={Xinrui Zhong and Zhifeng Li and Zhiming Wang and Yongqi Liu and Shanglin Yang and Yang Zhou and Zikun Li and Zhuoming Chen and Beidi Chen},
  year={2026},
  eprint={2611.06858},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2611.06858}
}