Why can some models read entire books while others can only read a page?
8 min readWhen you talk to an AI, the model doesn't remember all previous interactions. It can only "see" everything passed in the current request — this is the context window. The context window is the maximum number of tokens a model can process in a single request, including both input and output.
Think of it like a desk: the bigger the desk, the more papers you can spread out and reference at the same time. If the desk is too small, you can only look at one page at a time, unable to cross-reference multiple documents.
Context window = max input tokens + max output tokens. For example, a model with a 128K context window can read roughly 100,000 words of text and produce a response in a single request.
Drag the slider to feel how much information different context sizes can hold:
The context window is like your desk — the bigger it is, the more documents you can spread out and reference at once.
In just a few years, model context windows have grown by three orders of magnitude. From GPT-2's 1,024 tokens to today's million-token contexts, the amount of text an AI can "see at once" has expanded exponentially.
Key insight: 1,000x growth in 5 years. From processing a single paragraph to reading dozens of books in one go.
From a sticky note to an entire wall — AI's "workspace" is expanding exponentially.
If longer context is so useful, why not make it infinitely long? The answer: compute and memory costs.
The Transformer's attention mechanism requires every token to interact with every other token. Doubling the context length means 4x the compute. This is the quadratic complexity problem.
During inference, the model caches Key and Value vectors for each layer (the KV Cache). The longer the context, the more GPU memory the cache consumes. A 1M-token KV Cache can require several GB of memory.
Longer inputs mean the model needs to process more tokens during the prefill phase, increasing the time users wait for the first output character (TTFT).
Drag the slider to see how context length affects compute and memory:
The trade-off: longer context = more capability, but also slower and more expensive. This is why model providers offer different context size tiers.
The longer the context, the more the AI has to "recall" — like having more reference materials during an exam means more time flipping through pages.
The Transformer's attention mechanism is inherently order-agnostic — it treats the input as a set, not a sequence. Without additional information, "the cat chased the dog" and "the dog chased the cat" look identical to the model.
To help the model understand word order, we need positional encoding.
The original Transformer used fixed sine/cosine functions to generate position vectors, giving each position a unique encoding. Simple and intuitive, but difficult to extrapolate beyond lengths seen during training.
Rotary Position Embedding is the modern standard. It encodes position information as "rotation angles" in the vector space, naturally supporting relative position relationships and enabling length extrapolation through techniques like YaRN and NTK-aware scaling.
ALiBi (Attention with Linear Biases) adds a distance-proportional bias directly to attention scores, requiring no extra position vectors. Different approaches have their own trade-offs, but they share one goal: telling the model what comes before and what comes after.
Toggle position encoding on/off to see how it affects the attention pattern:
Each word has a clear position tag, so the model can distinguish word order.
Position encoding is like page numbers in a book — without them, even the best content is just loose pages.
The context window is everything the model can "see" in a single conversation, determining how much information it can process.
From GPT-2's 1K to today's 1M+ tokens, the growth rate of context windows has been staggering.
The O(N²) complexity of attention and linear KV Cache growth make ultra-long contexts come at a real cost.
Without positional encoding, Transformers can't distinguish word order. RoPE is the current go-to approach.
The context window defines how much an AI can "see" at once — from a paragraph to an entire book, the boundary of capability keeps expanding.