Two Optimizations That Were Never Allowed to Run Together

We run large language models on our own Apple Silicon hardware to power the coding agent our team builds with. The software that serves those models is mlx-vlm, an open-source inference server. We forked it. This is what the stock version does, why it wasn't enough, and what the fork bought us.

Two costs decide whether an agent feels fast

Every response carries two separate costs, and they are fixed by two entirely different mechanisms.

Coding agents punish the first one especially hard. Every turn re-sends the whole growing conversation, so the wait climbs turn over turn. In the 18-turn session we benchmark against, the conversation reaches 10,500 tokens — and re-reading it from scratch every turn means 96,500 tokens of work across the session to produce about 3,000 tokens of output.

Stock mlx-vlm fixes either cost. Not both.

The stock server ships two accelerations. Speculative decoding uses a small draft model to guess several tokens ahead, which the large model verifies in one pass — that fixes the stream. Prefix caching remembers text the server has already read, so repeated history isn't re-read — that fixes the wait. You choose one when you start the server, and you pay the other cost in full.

The exclusion is not architectural. Upstream simply switches the cache off whenever a draft model is loaded — a one-line precaution, never revisited.

There is a second, quieter gap. Stock caching only remembers text we sent. Everything the model wrote itself is discarded the moment a turn ends, then re-read from scratch on the next one. In an agent session, the model's own output is most of the conversation. Nothing survives a restart — every restart starts cold. And the cache collapses to zero hits once four sessions run at once, which is precisely the multi-agent case.

Three changes to the fork

Ranked by measured impact.

Same hardware, same model, same real session

Measured against the stock server in its best configuration for each metric — not its worst. Streaming speed is weighted by tokens produced, and session time is normalized to a common output volume, since each server picks its own reply lengths.

What this costs us to carry

The part that generalizes

There are two lessons here, and the second one cost us more than the first.

The first: the speedup didn't come from a clever algorithm. It came from a default nobody had revisited. Someone shipped a one-line safety measure, it was correct at the time, and every user downstream inherited it as a law of physics. Founders hit this constantly — the tool you build on states a limit in its docs, and you plan around the limit instead of testing whether it still holds. Reading the source took an afternoon and was worth more than a month of working around it.

The second: our first benchmark was wrong, in our own favor. We had measured the fork one way and the stock server another — letting the fork generate its own shorter replies while the baseline replayed a longer recorded transcript — and we compared a full-speed fork against baselines recorded while the machine was in Low Power Mode. Neither was deliberate. Both flattered us, and the published numbers came out roughly three times better than the truth.

Nobody caught it until someone re-derived the numbers from the raw run data and asked why the fork's session had processed a quarter as many tokens as the baseline's. If you are going to publish a benchmark, have someone try to break it first — and be suspicious when a result is exactly as good as you hoped.

Agent quality is still bounded by iteration speed, and the honest version of the win is a session dropping from 141 seconds to 38 against the configuration we had been running, plus 1.6× the throughput at a third of the wait when four agents work at once. Smaller than our first claim. Still the difference between staying in flow and context-switching away.

Benchmarks: replay of a recorded 18-turn agent session on an M4 Max in high power mode, one resident server at a time, all runs taken on a single day against a stock server restored specifically to measure against.