Academy

Prefill & Decode

The two phases of LLM text generation: "understanding" and "outputting"

Prefill Decode KV Cache TTFT

10 min read

Chapter 01

Two Phases

When you ask a large language model a question, it doesn't just "think and respond" -- it actually goes through two distinctly different phases. It's like taking an exam: first read the question, then write the answer.

The first phase is called Prefill, where the model "reads" all of your input at once. The second phase is called Decode, where the model begins "writing" its response one token at a time.

Interactive Demo: The Full Generation Process

Explain quantum computing in three sentences

Click "Play" to start the demo

KV Cache 0%

💡 Prefill is like a teacher quickly scanning your essay (skimming through it all at once). Decode is like the teacher writing feedback one word at a time.

Chapter 02

The Prefill Phase

During the Prefill phase, the model processes all your input tokens in parallel. This means whether your prompt has 10 tokens or 1,000 tokens, they are all "understood simultaneously" -- like a GPU performing thousands of matrix multiplications at once.

During this process, the model computes a pair of vectors for each token: K (Key) and V (Value), which are stored in the KV Cache. This cache is used repeatedly during the subsequent Decode phase to avoid redundant computation.

Interactive: Parallel Input Token Processing

Input tokens (all light up simultaneously after clicking play):

Explain quantum computing in three sentences

KV Cache construction:

K vectors

V vectors

Memory usage 0%

The computational cost of Prefill scales quadratically with input length (due to the Attention mechanism), so longer prompts significantly increase Prefill time. The good news: these computations are highly parallelizable, allowing GPUs to fully utilize their compute power.

💡 Prefill is like photographing an entire book at once -- no need to flip page by page; just capture everything in a single shot.

Chapter 03

The Decode Phase

After Prefill completes, the model enters the Decode phase. Here the model generates tokens one at a time: each newly generated token is fed back into the model as input to produce the next token. This is known as auto-regressive generation.

The key advantage: when generating a new token, the model doesn't need to reprocess all previous tokens -- it directly reads the information already stored in the KV Cache. This means only one forward pass is needed each time.

Interactive: Auto-regressive Generation

KV Cache (from Prefill)

6 tokens cached

Waiting to generate...

        Note: Each new token requires only one forward pass + KV Cache lookup, with no need to recompute all previous tokens. This is the core value of KV Cache.
      

The bottleneck of the Decode phase isn't computation -- processing just 1 token at a time requires very little compute -- but rather memory bandwidth. For each generated token, the entire model's weights must be read from memory into the compute units. The larger the model, the slower the read.

💡 Decode is like writing calligraphy -- each stroke must wait for the previous one to dry before writing the next. But with KV Cache, you don't have to re-grind the ink every time.

Chapter 04

TTFT vs TPS

Now that we understand the two phases, we can understand two key performance metrics:

TTFT (Time To First Token) -- the wait time from sending a request to seeing the first output token. This is primarily determined by the Prefill phase. The longer the prompt, the larger the TTFT.

TPS (Tokens Per Second) -- the speed at which the model outputs text. This is determined by the per-token generation time in the Decode phase and is largely independent of prompt length.

Interactive: Timeline Visualization

Drag the slider to change prompt length and observe how TTFT changes:

Prompt length 512 tokens

          12851220484096
        

■ Prefill (TTFT) ■ Decode (per token)

~200ms

TTFT

~50 tok/s

TPS

~20ms

Per-token latency

💡 TTFT is the time it takes for the first drop of water to come out after turning on the faucet. TPS is the flow rate after that. Longer prompt = longer pipe = longer wait.

Chapter 05

Why Is Prefill the Bottleneck?

Prefill and Decode face entirely different hardware bottlenecks. Understanding this explains why different hardware architectures perform differently across the two phases.

🔥

Prefill

Compute-bound

GPU compute utilization~90%

Memory bandwidth utilization~40%

Massive parallel matrix operations
GPU running at full capacity

💾

Decode

Memory-bound

GPU compute utilization~10%

Memory bandwidth utilization~85%

Only processes 1 token at a time
but must read all model weights

      This is why ANE is particularly well-suited for Prefill -- ANE excels at dense matrix operations, which is exactly what Prefill demands.
    

💡 Prefill is like moving all the furniture in a building at once (compute-intensive). Decode is like carrying one piece at a time but having to keep running back and forth (bandwidth-intensive).

Chapter 06

Summary

Prefill = Understanding

Processes all input tokens in parallel and builds the KV Cache. Compute-intensive, GPU running at full load.

Decode = Outputting

Auto-regressive token-by-token generation, reusing the KV Cache. Memory bandwidth-intensive, low compute utilization.

TTFT vs TPS

TTFT depends on Prefill speed, TPS depends on Decode speed. They require different optimization strategies.

KV Cache Is the Bridge

KV Cache connects the two phases, trading space for time and avoiding redundant computation during Decode.

"Understanding Prefill and Decode explains why LLMs 'wait first, then stream fast' -- and how to optimize them."

Next, we'll explore how ANE uses hybrid inference to accelerate Prefill by 11.3x

Next: ANE Hybrid Inference →