A quantized model can be faster and still be worse, and a speed benchmark will never tell you. When we baked Qwopus3.8-27B-Coder-Flash down to INT4, tool calling (BFCL) had dropped, and it took a paired task comparison against the BF16 reference to catch it. The fix was in the calibration data. Finding it took a pile of shell scripts, three eval harnesses with clashing dependencies, and a few hard lessons about how they behave under load. QualityBench is that kit, cleaned up so anyone can run it with one command.
What it measures
| Task | Items | Scored as |
|---|---|---|
| GSM8K | 1319 | last number in the reply, 0-shot chat |
| IFEval | 541 | prompt-level strict |
| MMLU-Pro | 1000 | fixed subset over 14 subjects |
| HumanEval+ / MBPP+ | 164 / 378 | base + plus tests, sandboxed |
| BFCL | 400 | function calling through the server's own tool parser |
| LiveCodeBench v6 | 175 | fixed slice, sandboxed |
| Logprobs | 592 windows | perplexity; KLD and top-1 agreement in compare |
| Needle | 20 | retrieval at 32k and 128k context |
| Spec parity | 32 | greedy text with and without speculative decoding (optional) |
Every run answers the same items with greedy decoding and seed 608. The logprob windows are built once per tokenizer from public corpora and identified by a hash of their token ids. So two runs compare item by item, not as two headline numbers side by side.
Compare, don't eyeball
qualitybench compare ref.json test.json takes a reference run (say BF16) and a test run (your INT4 build) and reports what changed:
- KL divergence of the next-token distribution, top-1 agreement and the perplexity change, over the same token windows;
- a paired accuracy change per task, with an exact McNemar p and a 10,000-resample bootstrap interval, so “GSM8K dropped 1.2 points” comes with whether that is noise;
- a pass or fail per threshold, from a thresholds file you write before you measure.
The output is a self-contained HTML report that works offline, plus a results file that launch80 renders as a shareable page. Here is one, from our second Qwopus study.
Built from what went wrong
Abandoned requests are cancelled
evalplus kills a request after 100 seconds and retries it forever. Behind a thinking proxy, the abandoned generations kept running and filled the KV cache, and one run stalled for 3.5 hours. QualityBench gives each request an hour, makes at most three attempts, and closes the connection on anything it gives up on, so the server stops generating.
Runs resume, and deadlines are honest
Rerun the same command and finished tasks, code-generation shards and logprob chunks are skipped. With --deadline 6h, no new work starts after the deadline, and cut tasks are listed as partial instead of disappearing.
Model code never touches the network
HumanEval+, MBPP+ and LiveCodeBench solutions run in containers with no network, capped memory and process counts, and no capabilities.
Thinking on or off, verified
--thinking on|off sets the mode through a local proxy, then checks that reasoning text is really there or really absent before scoring anything.
The study is now the bench's regression test. Re-scoring its per-item files with QualityBench reproduces every published task number exactly.
Run it without L80
uv tool install git+https://github.com/launch80/QualityBench@v0.1.0 qualitybench setup # one time: harness environments, sandbox image, datasets qualitybench run --endpoint http://127.0.0.1:8000/v1 --model my-model \ --out results/bf16.json --quick qualitybench compare results/bf16.json results/int4.json
--quick runs a fixed tenth of every task, which is a few minutes on a small model. Drop it for the full suite. The logprobs task needs vLLM started with --max-logprobs 64, and BFCL needs the server's tool-call parser on. The README lists the rest.
Run it with L80
QualityBench support is coming to the L80 CLI in its next release. L80 qualitybench runs the suite and publishes the result as a launch80 page. The L80 console gets a QualityBench screen with full, quick or custom task scope, an estimated run time and live per-task progress. After that, rented GPUs: a baseline or a comparison on a Vast.ai box, configured in the console and torn down automatically.
Links
Have a model or a quant you want measured, or results to compare? Join the Launch80 Discord · support the server