We made this as an onboarding map for new inference engineers. When you join a team that runs vLLM, SGLang or llama.cpp, the hard part usually isn't any single layer. It's knowing which layer you're looking at, and what it hands to the next one. The video follows one request the whole way. This page walks the same route in writing, with a link to each moment in the video.
1. The request ▶ 0:07
A client sends a prompt to the API server, usually an OpenAI-compatible endpoint. The server validates it, gives it a request ID, and turns on streaming, so tokens can go back to the client as soon as they exist instead of after the whole answer is done.
2. Tokenize ▶ 0:18
Models don't read text. The tokenizer first wraps the prompt in the model's chat template, the special tokens that mark system, user and assistant turns. Then it splits the text into tokens and maps each one to an integer ID. From here until the very end, the engine works with numbers.
3. Schedule ▶ 0:29
The scheduler decides what the GPU works on next. Our request joins a batch that's already running. With continuous batching, requests join and leave at every step, so a long answer doesn't hold up a short one. The scheduler also reserves KV cache memory for the request in small fixed-size pages. That's paged attention: memory is handed out as it's needed instead of being reserved for the longest possible answer.
4. Prefill, through the whole stack ▶ 0:45
The forward pass is where software meets hardware. Each layer hands off to the one below it:
| Layer | What it does |
|---|---|
| Framework | Often PyTorch. Runs the model's layers in order. |
| Kernels | Each operation becomes a kernel: matrix multiplies, attention, norms. |
| Runtime | CUDA, ROCm or SYCL launches those kernels. |
| Driver | Turns the launches into commands for the hardware. |
| GPU | Thousands of cores do the math while the weights stream in from memory. |
This first pass over the whole prompt is prefill. Every prompt token is processed at once, so the GPU has plenty of math to do per weight it reads. Prefill is compute-bound.
5. The KV cache ▶ 1:16
Prefill also saves a key and a value for every token, in every layer. That's the KV cache. Later tokens reuse it instead of recomputing attention over the whole past. It's also usually what runs out first: the cache grows with every token of every user. Quantizing it, in the video to 8-bit keys and 4-bit values, fits more users into the same memory.
6. Multi-GPU ▶ 1:32
Big models don't fit on one GPU. With tensor parallelism, each GPU holds a slice of every layer. They compute their part at the same time, then combine results with an all-reduce over a fast link such as NVLink. The link matters: the all-reduce runs at every layer, for every token.
7. Sample ▶ 1:45
The last layer outputs logits, a score for every token in the vocabulary. The sampler applies temperature and top-p to those scores and picks one token. In the video, it picks “It”.
8. The decode loop ▶ 1:56
Now decode. The new token goes back in, and the model runs again, one token per step. Each step does very little math but reads every weight and the whole KV cache, so decode is limited by memory bandwidth, not compute. That's why batching many users together helps so much: they share the cost of reading the weights.
Speculative decoding attacks the same limit from the other side. A small draft model (or a draft head) guesses several tokens ahead, and the big model checks all of them in one pass. Every accepted guess is a token that didn't need its own full read of the weights.
9. Stream back ▶ 2:14
The detokenizer turns IDs back into text, and the server streams each piece to the client as it's made. When the model produces the end-of-sequence token, generation stops and the request's KV pages are freed for the next one.
The three numbers that measure it ▶ 2:26
Time to first token (TTFT)
Queueing, tokenizing and prefill. What a user feels as “is it working?”
Time per output token (TPOT)
One decode step. Set mostly by memory bandwidth. What a user feels as reading speed.
Throughput
Tokens per second across every request at once. What the hardware bill is measured against.
Every optimization in the video moves at least one of these, and often trades one against another. A bigger batch raises throughput and can slow each user's TPOT. A quantized KV cache fits more users but has to be checked for quality. You can't reason about those trades without measuring all three.
Benchmark your own engine ▶ 2:35
L80 betterbench measures decode, prefill, time to first token and throughput under concurrency on any OpenAI-compatible endpoint (vLLM, llama.cpp, SGLang, Ollama, llama-swap), and publishes the result as a shareable launch80 page.
curl -fsSL https://www.launch80.com/install.sh | sh L80 betterbench --help
The BetterBench docs cover the options, and the L80 console how-to shows the same run from the console.
Questions, results, or a model you want tested? Join the Launch80 Discord · support the server