How LLMs Actually Generate Text: The Full Inference Pipeline

Ask a model “What is gravity?” and it answers a word at a time, left to right, as if it were thinking out loud. It is not thinking out loud. Under the hood every single token you see streamed back has traveled the same fixed assembly line: tokenize, embed, run the transformer stack, project to a vocabulary, sample one token, then do the whole thing again for the next one. This piece walks that assembly line end to end, and explains the one fact that reframes everything you thought you knew about inference speed.

The structure here follows a genuinely excellent hand-drawn diagram by Brij Kishore Pandey, “How LLMs Actually Generate Text,” which lays out the full pipeline in nine numbered stages. I am going to use those same nine stages as the spine, then spend most of the words on the part the diagram flags in red down the right margin: the systems reality that decides how fast and how expensive any of this is.

The whole pipeline in one breath

Before we zoom in, here is the entire journey. A model does not do anything mysterious. It does these nine things, in order, and the last few repeat once per output token.

1. Input. Your raw text prompt.
2. Tokenizer. Split the text into tokens and map each to an integer ID.
3. Embedding. Turn each ID into a dense vector.
4. Transformer block (x N layers). Self-attention plus a feed-forward network, stacked dozens of times.
5. Linear plus softmax (LM head). Project the final vector to a probability over the whole vocabulary.
6. Sampling. Pick the next token from that distribution.
7. Speculative decoding. An optional speed trick that guesses several tokens ahead and verifies them at once.
8. Detokenizer. Turn the chosen token IDs back into text.
9. Streaming output. Show tokens to you as they land.

Stages 4 through 8 run once for every token in your prompt to “warm up,” and then again, one token at a time, for every word of the answer. That split is the whole story, so let us name it now: the warm-up is called prefill, and the one-at-a-time generation is called decode. Hold onto those two words.

What happens before the model thinks: tokenize and embed

A model cannot read characters. The tokenizer chops your prompt into subword pieces and looks up an integer for each one. “What is gravity?” might become five tokens, and notice that a common word like “gravity” can split into grav and ity. That is normal. Tokenizers are trained to break rare words into frequent fragments.

"What is gravity?"
  -> tokens:     ["What", "is", "grav", "ity", "?"]
  -> token IDs:  [2601, 318, 26110, 879, 30]

Each ID is then used to look up a row in an embedding matrix, producing a dense vector. In a GPT-4 class model that vector has d_model = 4096 dimensions. So a five-token prompt becomes a 5 x 4096 grid of numbers. Everything from here on is matrix math on that grid. Tokenization also quietly decides your bill, because you pay per token, and a tokenizer that fragments your text more aggressively means more tokens for the same words.

Inside the transformer block: attention, the KV cache, and the FFN

This is the engine, and it is stacked deep. A GPT-4 class model runs on the order of 96 identical layers, and every layer does the same two things: mix information across tokens with self-attention, then transform each token on its own with a feed-forward network.

Self-attention projects each token into three vectors: a Query, a Key, and a Value. To decide how much token i should care about token j, the model takes the dot product of i’s query with j’s key, scales it, and runs a softmax so the weights sum to one. Those weights then blend the value vectors together. The famous one-line version:

attention(Q, K, V) = softmax( (Q @ K^T) / sqrt(d_k) ) @ V

After attention, a residual connection adds the input back and a LayerNorm keeps the numbers stable. Then the feed-forward network expands each token vector to a wider hidden size, applies a nonlinearity like ReLU or SwiGLU, and projects it back down. That is where much of a model’s raw knowledge lives. (In the biggest models this FFN is sparse, and only a few expert subnetworks fire per token, which is the trick behind Mixture of Experts.) Add, normalize again, and hand off to the next layer.

The KV cache is the single most important object in LLM inference, and almost nobody outside the field has heard of it.

Here is the part that governs everything downstream. When the model generates token number 500, its attention needs the keys and values of all 499 tokens before it. Recomputing those from scratch every step would be insane, so the model stores them. That store is the KV cache: the key and value vectors for every token, at every layer, kept in GPU memory so they are never computed twice.

The catch is in the arithmetic. The KV cache grows linearly with sequence length. Every new token adds another slice of keys and values, at every one of the 96 layers, for every attention head. A long conversation or a large document does not just cost more compute, it eats gigabytes of memory, and reading that cache back on every single decode step is what actually caps your speed.

From hidden state to a word: the LM head and sampling

After the final transformer layer, one more linear projection, the LM head, maps the last token’s 4096-dimensional vector to a score for every word in the vocabulary. Vocabularies are large now, on the order of 128K entries, so this produces 128,000 raw numbers called logits. A softmax turns them into a probability distribution: this token 41% likely, that one 12%, and so on across the whole vocabulary.

Now the model has to actually pick one. This is sampling, and the strategy you choose is the difference between robotic and creative output.

Why is decoding so slow? Prefill versus decode

Now we can cash in those two words from earlier. Inference runs in two phases with opposite personalities, and confusing them is why so many people misjudge where the time goes.

This is the counterintuitive heart of the whole subject. During decode the GPU is not short on math, it is short on memory bandwidth. It spends most of each step waiting to shuttle weights and the growing KV cache in from memory, then does a tiny amount of arithmetic on one token, then waits again. Your expensive accelerator is mostly idle, tapping its foot. Every serious inference optimization is an attack on that specific wait.

The speed tricks: FlashAttention, quantization, speculative decoding

Once you accept that decode is memory-bound, the entire optimization playbook reads like a list of ways to move less data. Here are the three that matter most, each aimed at a different part of the bottleneck.

Notice the pattern. FlashAttention moves less data during attention, quantization moves fewer bytes per weight, and speculative decoding gets more tokens out of each memory-hungry pass. None of them make the arithmetic faster, because arithmetic was never the problem. They all fight the memory wall.

Detokenize and stream

The last two stages are the easy ones. Each token the model commits to is a number, so the detokenizer runs the tokenizer in reverse, mapping IDs like 2610, 879, 318 back into “Gravity is…”. And because the answer is built one token at a time anyway, there is no reason to make you wait for the whole thing. Streaming pushes each token to your screen the instant it is chosen, which is why watching a model type feels like watching it think, even though it finished deciding each word a fraction of a second before you saw it.

Why this mental model matters

You do not need to memorize nine stages. You need one load-bearing idea: inference is two phases, and the slow one, decode, is memory-bound. Everything else falls out of that. Long prompts cost you in prefill compute. Long conversations cost you in KV cache memory. Every latency optimization worth knowing is a scheme to move less data on each decode step.

Once you see it that way, the news in this space stops being a blur of unrelated tricks. Bigger context windows are really a KV cache memory problem. Cheaper serving is really a bytes-per-weight problem. And when a lab claims a large speedup with no quality loss, you already know which wall they must have found a way around. For where this is all heading, see our take on small language models for agentic AI, where the same economics push work toward models small enough to keep the whole cache close.