Decode throughput across three local runtimes
Single 35B model, one machine, identical prompts
Self-reported — not verified by launch80We pointed llama.cpp, vLLM and MLX at the same 35B model on one host and measured time-to-first-token and sustained decode rate over a 24-turn session replay.
- Best decode142tok/svLLM, batch 1
- Median TTFT0.98s
- Peak memory32.3GB
- Turns24
Method
Each runtime served the same quantized 35B checkpoint on the same host, with no other load.
We replayed a recorded 24-turn coding session against each server and recorded per-turn wall time, time-to-first-token, and reported token counts.
- Identical prompts and generation caps across runtimes
- Warm cache: one discarded priming run before measurement
- Server-reported token counts, not estimates
What surprised us
The spread in time-to-first-token was much wider than the spread in decode rate. Two runtimes within 4% of each other on sustained decode differed by more than 2x on TTFT, which is the number a user actually feels while typing.