One Model, Every Local Stack

A recurring question in our Discord: which local-AI setup is fastest for running a big coding model on Apple Silicon? People name five in the same breath — ctrl8d, vMLX, pi, hermes, and opencode — as if they were interchangeable. They aren't. Two of them are inference servers (the thing that loads the model and answers requests) and three are agents (the coding tool you actually type at, which sends requests to a server). You can't rank a server against an agent any more than you can rank an engine against a driver.

So we ran two separate benchmarks, both on the same M4 Max, both serving the same Qwen 35B. Then, because people asked, we went further and threw every other local inference server we could get running at the identical session. Here is what we found — including the mistakes we had to back out of to keep it fair.

Part 1 — The servers, on the exact same file

The model is an MLX 6-bit quant — a specific file. Only MLX-family servers can load that exact artifact, so this first group is a true apples-to-apples comparison. Each replays an identical recorded 18-turn coding session cold (empty cache, so none carried an unfair head start). ctrl8d is our fork of mlx-vlm; mlx_lm.server is the bare runtime everything here is built on; stock mlx-vlm is the upstream project ctrl8d forks; vMLX and LM Studio are polished desktop MLX apps. TTFT is “wait before the first word”; decode is streaming speed, weighted by tokens produced.

* vMLX's raw decode is from its own single-shot benchmark on a short prompt (which runs fine). On the full session it GPU-faulted every time. LM Studio's bundled MLX runtime doesn't recognize this model's architecture (qwen3_5_moe) at all — the load aborts at 20%.

Three things stand out. First, the floor: mlx_lm.server, the bare runtime with no caching or speculation, already does ~90 tok/s and finishes in 65 s. That's the honest baseline — every MLX server here is a wrapper around it, and it's genuinely fast on its own. The optimizations earn their keep at the edges (first token, and under load), not in raw single-stream decode.

Second, ctrl8d holds the speculation mode's decode rate (98.5 tok/s) at the caching mode's time-to-first-token (0.66 s) — stock mlx-vlm forces you to pick one or the other, and ctrl8d finishes the session in 42 s against its 70 s and 142 s. (Why that trade exists, and how the fork removes it, is the subject of our previous post.)

Third, and more bluntly: two of the polished desktop apps couldn't run the model at all. vMLX loaded it and then hit three hard Metal GPU faults across three attempts — a page fault at turn 9, a hang at turn 16, another at turn 5 — falling over past a few thousand tokens of context every time (its raw single-shot speed is fine, ~74 tok/s). LM Studio wouldn't even finish loading: its bundled MLX runtime doesn't support this model's architecture yet. Both are nice software; Qwen3.6 is just newer than their shipped runtimes. We are reporting what happened, not what we expected.

Under load, the gap widens

The realistic case for us is several agents hitting one server at once. Four concurrent sessions, same model:

vMLX and LM Studio are absent here — neither could run the session in the first place. mlx_lm.server runs fine but has no concurrency layer; its four-way wait balloons to ~10 s, which is exactly the gap ctrl8d's scheduler closes.

Part 1b — A different file: GGUF, and the ones that won't install

Everything above loads the same MLX file. The tools people ask about next — llama.cpp, ollama, vLLM — can't. They read a different format (GGUF), so we downloaded the closest match, a Q6_K quant of the same base model, and ran the identical session. This is a cross-format comparison, not the same artifact — read it that way.

Decode for the GGUF row is the engine's self-reported rate; llama.cpp and ollama share the same underlying engine, so the real gap between them is wrapper overhead, not raw speed.

Both GGUF servers work — which already beats two of the desktop MLX apps — but they land slower. llama.cpp's prefill is heavier (1.8 s to first token vs ~0.7 s for the MLX runtimes), and ollama, which is llama.cpp wrapped, adds overhead on top and handles concurrency poorly: even after we raised its parallelism to four, its four-way wait was 13–19 s against llama.cpp's 5–7 s. On Apple Silicon, for this mixture-of-experts model, the native MLX runtimes are simply the faster path — and that is the whole reason the MLX-first tools exist.

vLLM never got off the ground. It has no Metal backend for this kind of model, and its build on Apple Silicon fails outright on a dependency it can't satisfy (it demands an older PyTorch that isn't published for this platform). vLLM is superb on NVIDIA GPUs; on a Mac it is simply the wrong tool, and we're noting that plainly rather than pretending we measured it.

Part 2 — The agents

Now the other layer. pi, hermes, and opencode are coding agents; we added ctrl8 (the companion agent to the ctrl8d server) as the reference. All four were pointed at the same ctrl8d backend and given the same tiny task — write a correct fizzbuzz(n) function to a file — at temperature 0, so the runs are deterministic.

Because they share one backend and one model, their streaming speed is identical — all four decode at ~105 tok/s, because that's the server's number, not the agent's. What the agent controls is something else entirely: how many tokens of scaffolding it spends to set the task up, and whether it actually gets the job done.

The spread in that top bar is the story: hermes spends about 16× the prompt tokens that ctrl8 does for the identical task. That isn't waste, exactly — hermes carries a large system prompt and a broad toolset, which buys capability it didn't need here. But on a memory-bound Mac, every one of those tokens is prefill work and RAM, so the heavier the harness, the more the small local model has to chew through before it can start.

The result we sat with longest is opencode's 0/3. It isn't that opencode is broken — in a separate test, with a more forceful prompt, it wrote the file perfectly. But on this exact task phrasing, driving this exact local 35B, it described the solution in prose across three turns and never emitted the tool call to write the file — deterministically, all three runs identical. Stronger models make that failure vanish. It is a real, reproducible mismatch between one agent's scaffolding and one small local model, not a verdict on the tool.

What we had to back out of

A benchmark is only as honest as the errors you catch before publishing. Four we caught mid-run:

• ctrl8d looked impossibly fast at first — because it was serving from a disk cache warmed by earlier runs while the others started cold. We cleared it and re-ran every server from cold. That persistence is a real ctrl8d feature, but it doesn't belong in a head-to-head bar.
• vMLX and ollama both defaulted to serial handling, which would have made them look artificially bad under load. We raised each to four-way concurrency for a fair shot — vMLX crashed anyway, and ollama stayed slow.
• The GGUF servers run a different file. To compare them at all we matched the quantization as closely as we could (Q6_K against the 6-bit MLX) and labeled every one of those rows cross-format, so nobody reads them as the same artifact.
• Our first agent task was ambiguous. “A fizzbuzz function” can mean one value or a whole sequence; agents that guessed differently looked like failures when they weren't. We rewrote the task to pin the exact signature before trusting any result.

Two patterns held across all eight servers. On Apple Silicon, the native MLX runtimes are the fast path — the GGUF engines run, but slower, and the desktop apps were either behind on the architecture or unstable on it; the further a tool sits from Apple's own MLX stack, the rougher the ride. And for our actual workload — many agent sessions against one Mac — the combination that held up was a lean agent (ctrl8 or pi) on a server that doesn't make you trade latency for throughput (ctrl8d).

But the most durable lesson is the one that keeps recurring when you measure instead of assume: the tools people quote in one breath are often not doing the same job — and half of them may not even run what you think they run.

Method: recorded 18-turn agent session replayed at temperature 0, M4 Max in high-power mode, one server resident at a time, each cold-started. Agent runs measured server-side (turns and tokens read from the server's own counters) over three deterministic runs each. Raw result files are kept alongside the benchmark harness.