A Better INT4 Qwopus We Decided Not to Ship

Nothing changed for users. Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked is still v1.1.1. The v2.0 candidate went live on the model repo for a short time before we decided against it; we put main back to v1.1.1 (byte-identical) and never tagged v2.0. The measurement pages stay up, because they are accurate records of what we measured.

Why we went back

The first study ended with v1.1.1: a GPTQ INT4 body with a baked INT4 lm_head and MTP head at 92.5–96.0 tok/s on one card. It passed every task bar, missed top-1 agreement by 0.16 points, failed spec-on-equals-off parity only at ties, and failed T6 outright: on our harness, Jackrong’s Q4_K_M GGUF stayed closer to BF16 (mean KLD 0.0250 against our 0.0412, top-1 agreement 94.92% against 92.84%). A newer build, YTan2000’s TQ_L, reports beating Q4_K_M at 14.63 GB on its own harness.

We can’t run TQ_L (its engine is NVIDIA-only) and its harness isn’t comparable to ours, so we didn’t rent hardware to measure it. Instead, before building anything, we wrote down a bar that scales our own Q4_K_M measurement by the ratios TQ_L reports against Q4_K_M:

It is an anchor, not a measurement of TQ_L, and we said so in the plan. Speed was not part of the bar.

Where the gap was

The first study’s ablation already pointed at two places. With a full-precision head, our INT4 body scored KLD 0.0343, so the body alone missed TQ1. Quantizing the head to INT4 g128 cost about 1.5 points of top-1 agreement. So the plan had three levers: a better body quantizer, a finer-grained head, and smaller embeddings to pay for the extra bytes. We changed one thing at a time against the shipped weights, so every gain is attributable.

First we checked how much of our 0.041 is the engine rather than the quantization. We served the shipped v1.1.1 weights on a rented H200 with CUDA vLLM and compared them with the same weights on the B65: KLD 0.0018 between the two. The Intel XPU stack adds almost nothing, so the gap is the weights.

What each change bought

All rows below use the same harness: top-64 log-probs on our fixed 512 × 2048-token set, against the BF16 reference, speculative decoding off, on the B65. Lower KLD is better. From the g32-head row down, every build also carries an INT4 g32 embedding, except the candidate.

• Body. AutoRound 0.16.1, which tunes rounding and scales with signed gradient descent, beat GPTQ on the same calibration text. Asymmetric quantization (a zero-point per group) helped again. Body plus head took KLD from 0.0412 to 0.0330, and the two gains added up almost exactly.
• Head. Same Hessian bake as v1.0, at group 32 instead of 128: output error on held-out rows fell from 4.68% to 3.72%. That one file gave back most of the top-1 the INT4 head had cost.
• Embeddings. INT4 or INT8 embeddings were almost free in quality (+0.0003 KLD) and saved 1.3–1.9 GB. But stock vLLM 0.30 builds this model’s embedding without its quantization config, so a quantized embedding needs a loader patch. We had a rule from the start that we don’t ship a model that needs a vLLM patch, so the candidate kept the embedding in BF16. That decision is why it misses TQ3: about 16.2 GB of text weights (our estimate) instead of ≤ 14.63.
• W4A8. Quantizing activations to 8 bits on every body linear raised KLD by 60%, so it was out. It wouldn’t have helped decode anyway, as shown below.

We got the bulk of the way to the bar and stopped short of it: the candidate’s KLD would need to fall another 28%. Since that falls short of what we set out to do, we report it as a miss.

The candidate, measured as it would ship

We ran the candidate through the first study’s full protocol, served exactly as it would ship with MTP k=6. That is what the v1.0-vs-v2.0 study page shows. Harness: the scripts in our public quality-eval dataset.

No task loses significantly against BF16. BFCL is the only significant difference (+2.5 points, CI +0.5 to +4.5, McNemar p = 0.03), and it is a gain. Treat the other above-BF16 scores as noise: these are greedy, single runs. The four speculative-decoding mismatches are all ties at the point of divergence, the same pattern as v1.x. On the first study’s bars, the candidate passes T4, which v1.0 missed. It still fails T6 against Q4_K_M, though the KLD gap halved.

Where it lost

Speed, in two ways. Quick BetterBench run, MTP k=6 at 262k context: 72.5 tok/s weighted decode against 75.8 for v1.x (page). That happened even though the candidate’s MTP head accepts more draft tokens: 4.36 tokens per verify step on the quality suite against 3.81 for v1.0. Each step simply takes longer, about 50 ms against 42 ms. The candidate loads through vLLM’s compressed-tensors INT4 kernel with a group-32 head. v1.x loads through the GPTQ path with group 128. We didn’t profile which part costs the time.

The bigger loss is the fast profile. v1.1.1’s headline speed, 92.5–96.0 tok/s at 128k context (page), comes from our INT4 DFlash2 drafter. On the candidate, vLLM refuses to start it: “A head_dtype different from the model dtype is only supported for an unquantized lm_head.” The BF16 drafter shares the target’s quantized output head. The same drafter starts on v1.x; we didn’t establish why the compressed-tensors head is treated differently.

The decision

Several decisions shaped this study:

• Anchor, don’t rent. We set the TQ_L bar from published ratios instead of renting NVIDIA hardware to run a different engine.
• No vLLM patches in a released model. This cost the candidate TQ3 and 1.8 GB, but anyone with stock vLLM can load it.
• Ship below the bar if it beats v1.0? At first, yes: we were going to release it as “beats v1.0, short of TQ_L”.
• Not without the fast profile. Once the measurements showed it couldn’t run the DFlash2 drafter, we stopped. A release that is better on paper but drops the profile most people use is not an upgrade. We reverted the model and dataset repos and kept v1.1.1.

One process note. The candidate reached the model repo’s main before the DFlash result was weighed, because we ran the release steps in parallel with reading the results. For about four minutes (21:13 to 21:17 UTC on 4 Nov), anyone pulling main got the candidate. It was never tagged, and main now matches v1.1.1 file for file. Next time, every gating result gets read before anything is published.

Open ends, if you want to pick this up

We’re closing this line of work for now. If you want to continue it, start here. The eval set, the BF16 reference logits, the calibration set and the scoring scripts are all in the dataset, so you can score your own build against exactly what we used.

1. Step time. Profile one verify step of the candidate against v1.1.1 on the same card. Try the AutoRound body with the g128 head, or the same body exported in GPTQ format (which can carry real zero-points), to separate the kernel path from the head.
2. DFlash2 on a compressed-tensors target. Try the drafter in fp16, a drafter with its own head, or both models in bf16.
3. The last 28% of KLD. Q4_K_M keeps a few dozen sensitive tensors at 6 bits. A selective higher-precision mix on our side is the obvious next lever, followed by more AutoRound iterations or calibration data.
4. Embeddings without a patch. If vLLM’s Qwen3.5-family loader passed the quantization config to the embedding upstream, an INT8 embedding would save about 1.3 GB at almost no cost and bring TQ3 within reach.
5. W4A8 for prefill only. In a single-GEMM microbenchmark on the B65 (a kernel rate, not end to end), W4A8 ran 1.1–1.7× faster at 512 rows or more but 0.4–0.95× at 1–8 rows, which is decode and MTP verify. Both kernels read the same packed weights, so routing by row count (A16 for decode, A8 for prefill) could cut time to first token without touching decode or memory. Nobody has built it.
6. A tie-aware parity bar. Every version fails “spec on = off” as written because batched verification breaks exact ties differently. A bar that tolerates ties would measure what we actually care about.

What it cost

Four rented H200 sessions, about $22 in total by our account balance. One was lost: the quantization finished, but the results couldn’t be downloaded through the provider’s SSH proxy (5–7 MB/s) before the time cap destroyed the box. We redid that run with direct SSH, a resumable download that checks hashes, and a network speed check that refuses slow hosts before any work starts. Everything else ran on the B65.

Every page from this study

• Quality study page, v1.0 vs the v2.0 candidate
• BetterBench, v2.0 candidate, MTP k=6 (quick)
• Quality-eval dataset: BF16 logits, eval set, calibration set, scripts
• Hugging Face collection for the whole study
• The first study, v1.0 to v1.1.1