Baking a 27B Coder Model Down to INT4 on One Arc Pro B65

How this started

Wrapzii, whose baked Swift-1.5 INT4 build we have benchmarked before, asked us to run Jackrong/Qwopus3.8-27B-Coder-Flash through our bake. The goal was to make it a fast local model on a single Intel Arc Pro B65 32 GB, served from the machine that runs our coding harness.

The model only exists in BF16: 18 safetensors shards, 55.6 GB, a Qwen3.8 hybrid with 64 layers plus one MTP (multi-token prediction) layer and a vision tower. That doesn’t fit in 32 GB, and nobody had published a GPTQ or AWQ build of the Coder variant. Every model we had baked before started from someone else’s INT4 body. This time we had to quantize the body ourselves, on a host with 30 GB of RAM, before the bake could even start.

What “the bake” is

A typical GPTQ INT4 checkpoint quantizes the transformer body and leaves two large pieces in 16-bit: the output head (lm_head, 248k × 5120) and the MTP draft head that speculative decoding uses. On a card limited by memory bandwidth, those two are a large share of the bytes read for every token. Our bake quantizes both to INT4 with GPTQ, using Hessians captured from the model’s own activations inside the serving engine. It then only keeps a layer if it beats plain round-to-nearest (RTN) quantization on held-out rows. The result is one checkpoint that stock vLLM serves with no runtime patches to the weights.

Building it on one card

The body was quantized with GPTQModel 7.3.2 on the B65 itself, inside an XPU container built on the same vLLM image we serve with. Settings: 4-bit, symmetric, group size 128, act-order off, damping 0.05, 512 calibration samples of 2048 tokens (C4, code and chat).

The first problem was host memory, not the GPU. Short 2-, 4- and 8-layer smoke runs showed resident memory growing by about 1 GB per layer. That would have run out of RAM halfway through a 64-layer model. It wasn’t a leak in the quantizer: glibc’s per-thread malloc arenas were fragmenting. Setting MALLOC_ARENA_MAX=2, calling malloc_trim(0) between layers and finalizing tensors synchronously cut the 8-layer peak from 23.7 GB to 17.5 GB and made the per-layer floor flat. The full body then quantized in 3 h 12 min (11,542 s), peaking at about 27.8 GB on the first layer and staying near 17 GB after that.

The bake itself followed. Hessians for the head and MTP layers took 46 minutes. The gate passed 9 of 9: every baked linear beat RTN on the same held-out rows. The INT4 lm_head’s output error was 4.66%, against 8.16% for RTN, and the MTP linears were about 40% better than RTN. The assembled checkpoint is 16 GB.

First speed check: a quick BetterBench run, MTP speculative decoding with k=6 at 262k context, gave 78.9 tok/s weighted decode and about 1,250 tok/s prefill (page). That was fast enough to be worth checking whether the model was still any good.

Setting the bar before measuring

We wrote the quality thresholds down and hashed the file before any quantized model was evaluated. We recorded each later change to the protocol as a dated addendum with its own hash. There were four, all made before any INT4 results existed: the LiveCodeBench window, the GSM8K answer extraction for chat models, the BFCL data version, and an exploratory BF16 thinking-on arm that is not gated. The bars:

The seven tasks are HumanEval+, MBPP+, LiveCodeBench v6, GSM8K, an MMLU-Pro subset, IFEval and a 400-item BFCL tool-calling subset. Every task compares the same questions item by item, so we report each one as a paired delta with a 95% bootstrap interval, not as two separate scores.

Renting the yardstick

A 55 GB BF16 model won’t run on the B65, so we rented the reference. Over three Vast.ai rentals of an RTX PRO 6000 96 GB, running the same vLLM 0.30, we captured the BF16 model’s top-64 next-token log-probabilities on a fixed set of 512 × 2048 tokens, ran all seven tasks with thinking off, ran the needle-in-a-haystack tests at 32k and 128k, and then ran the seven tasks again with thinking on. The three rentals cost roughly $35 in total, by our own estimate from the hourly rates.

Two of those rentals kept billing for hours after their work was done. The rental tool rightly won’t destroy a box without a human answering its prompt, and nobody was there to answer. Rented boxes now get a detached teardown watcher that ends them the moment their results are pulled back.

The thinking-on numbers aren’t gated, but they show how much of this model’s ability sits behind its reasoning: LiveCodeBench rises from 41.1% to 57.1% and MMLU-Pro from 71.8% to 79.2% (study page with both, BF16 baselines).

Three arms, one change at a time

To show which step costs what, the INT4 build was measured as a chain. Each arm adds one change:

• A: the INT4 body with the head and MTP left in full precision.
• B: A plus the baked INT4 head and MTP. This is the arm the thresholds gate.
• C: B served with MTP speculative decoding at k=6, for the T7 parity check.

Does it predict the same tokens?

Perplexity and KL divergence pass with room to spare. Top-1 agreement misses by a quarter of a point on B but not on A, so the shortfall comes from quantizing the output head, not the body.

Does it still get answers right?

Six of the seven tasks are within the bar or within chance. HumanEval+ and LiveCodeBench drop more than 2 points, but their intervals include zero, so by the rule we registered they don’t count. BFCL doesn’t pass: both INT4 arms lose 8–10 points on tool calling, far outside chance.

The tool-calling bug

Almost all of the BFCL loss was in two categories: parallel and parallel_multiple, where the right answer is several tool calls in one turn. On 24 of the 25 parallel items arm B lost, it made exactly one call where BF16 made two to four. Then it stopped. Its replies were less than half as long (median 54 tokens vs 134). The parser wasn’t dropping anything: the model was ending its turn.

The token right after the first </tool_call> showed why. BF16 strongly prefers a newline there, the start of the next call, with <|im_end|> (end of turn) about 2–3 nats behind. The INT4 model has them the other way round. That is a 2.5-nat shift in a single decision, not a near-tie flipped by rounding.

We ruled things out one at a time:

• The tool and reasoning parsers are the same on both sides.
• The tokenizer files the quantizer had re-saved gave the same probabilities as the original files. They will still be restored before upload.
• The FP8 KV cache: the same margins with the default cache.
• FP16 vs BF16 serving: the same margins.
• The baked head: arm A, with the full-precision head, fails worse.

That left the body, and its calibration set: 40% C4 web text, 40% code, 20% chat, and not one tool call or tools system prompt. GPTQ shapes its rounding with statistics of the activations it sees during calibration. The format of a multi-call reply was never among them.

The fix was to quantize the body again with 20% of the calibration replaced by Hermes function-calling conversations, rendered through the model’s own chat template with its tools, half of them parallel calls. Then we rebaked the head and MTP on the new body.

B2 recovers almost all of it and passes. Its next-token numbers barely moved: KL divergence 0.041, perplexity +1.7% on both corpora, top-1 92.84%. It still misses the 93% top-1 bar by a fraction of a point, as B did.

This recalibration is a post-hoc change: we made it after seeing the BFCL result, not as part of the plan we registered. Every page that shows B2 says so. B2 is the checkpoint we intend to publish, so all seven tasks are being rerun on it as shipped (see the log below).

Where it stands on the bars

T7 needs explaining. Six of the 32 prompts produce a different greedy output with speculative decoding on. At every one of those divergence points, the top two candidates were tied: five exactly, one within a single FP16 rounding step. Batched verification breaks a tie one way and plain decoding breaks it the other. Neither is wrong, but we wrote “identical”, so it’s reported as a fail with the evidence. The needle tests were 100% at 32k and 128k for both BF16 and the INT4 build.

The full study, with every chart and caveat, is on its QualityBench page.

Why it’s slower than Swift

With B2 in hand, a quick BetterBench run measured 75.8 tok/s weighted decode (page). On the same card and image, the Swift-1.5 bake with the same MTP drafter reaches 80.5, and with a DFlash2 drafter 96.7.

Speculative decoding speed comes down to two numbers: how long one verify step takes, and how many drafted tokens each step keeps. The step time is the same for both models, 42 ms, so the INT4 body and its kernels aren’t the difference. The difference is acceptance: Qwopus keeps about 3.0 tokens per step under BetterBench’s sampling, Swift 3.35. Under greedy decoding the MTP head’s guesses land 81% of the time at the first position, then 61%, 45%, 33%, 23% and 17% down the draft.

We checked the MTP head’s weights against the base Qwen3.8-27B. All 15 tensors are byte-for-byte identical. The fine-tune changed the model but never trained its draft head, so the drafter is still guessing what the base model would say, not what Qwopus says. Swift’s MTP head turned out to be stock as well.

Next: 115 tok/s

Our target is 115 tok/s weighted decode on a quick BetterBench run. Decode speed is tokens kept per step divided by step time. The step is already close to the bandwidth floor: about 27 ms for the target model against roughly 24 ms to read its bytes. So the gain has to come from the drafter. At the step time of the best drafter we have, reaching 115 means keeping about 5 tokens per step, which needs a per-position acceptance of about 0.88 at a draft length of 7. The plan, in order:

• Run the stock DFlash2 drafter, which gave Swift its 96.7, on Qwopus, and measure how acceptance and step time change with draft length.
• Retrain a drafter on Qwopus’s own outputs: the DFlash2 drafter first, the MTP head second, so that there is also a 262k-context option.
• If that isn’t enough, remove the measured +5 ms jump in step time at nine query tokens, so that longer drafts pay off.

None of this changes the target model’s weights. With greedy decoding, speculative decoding doesn’t change the output, so the quality study above stays valid. Each new drafter only needs the parity check.

We also have a W4A8 build planned, with INT8 activations on selected layers. We expect it to speed up prefill, not decode, because decode on this card is limited by memory bandwidth and W4A8 reads the same 4-bit weights. It also has little quality headroom: B2’s KL divergence is 0.041 against a 0.05 bar. So it will be its own study, later.

Things that went wrong

• A benchmark that cancelled itself. evalplus 0.3.1 cancels every request after 100 seconds and retries forever. With thinking on, the answers took longer than that, so each problem was sent again and again while the abandoned generations kept running upstream: 56 requests in flight and a KV cache 96% full. A wrapper with a one-hour timeout and three tries fixed it.
• vLLM’s default batch size on a hybrid model. vLLM 0.30’s default of 1,024 sequences exceeded the model’s 670 linear-attention cache blocks and crashed the BF16 server. Capping it at 256 fixed it.
• Calibration that depends on the library version. transformers 5.22 on the host builds the C4 calibration windows in a different order from 5.16 in the quantization image. Calibration data is now always built inside the image.
• Tokenizer files rewritten by the quantizer. They’re harmless for serving, but they don’t match upstream. The originals go back before upload.

Every page from this run

• BF16 vs INT4 quality study: arms A, B and B2 against BF16 (current).
• Quick BetterBench, B2 on the B65: 75.8 tok/s, MTP k=6.
• Quick BetterBench, first bake (B) on the B65: 78.9 tok/s.
• Study with complete thinking-on results, before B2 existed.
• BF16 quality reference on its own.
• BF16 task baselines, thinking off and on.

Pages on launch80 are self-reported until they are verified, and each one carries a badge saying which it is.

Log

• 1 NovRequest from Wrapzii. Thresholds registered and hashed. BF16 download (55.6 GB). Body quantization began after the malloc-arena fix.
• 2 NovBake passed its gate 9/9 and the first quick BetterBench ran at 78.9 tok/s. Three BF16 reference rentals. Arms A, B and C measured. BFCL regression found and traced to calibration. Body recalibrated with tool calls, giving A′ and B2; B2 passes BFCL. Study and B2 BetterBench published. MTP head found to be stock. The full seven-task rerun on B2 as shipped started.