Nvidia researchers argue huge AI context windows are a dead end
Recent advances allow artificial intelligence models to process massive amounts of text at once.
2 min read, Saturday, 21 November

Language models can now process entire novels at once. An Nvidia research team says that is often overkill and expensive. In many tests, periodic summaries and targeted lookups worked about as well.
A language model's context window is its working memory, covering everything it holds in view when generating a reply. Over the past few years, that window has grown rapidly. GPT-4 launched in March 2023 with 8,192 tokens. Today, GPT 5.6 Sol handles around one million tokens, about 120 times as much and enough for Tolstoy's War and Peace.
Key facts
- GPT-4 launch date
- March 2023
- GPT-4 context window
- 8,192 tokens
- GPT 5.6 Sol context window
- 1 million tokens
- Launch Date
- GPT-4: March 2023
- GPT 5.6 Sol: —
- Context Window
- GPT-4: 8,192 tokens
- GPT 5.6 Sol: 1 million tokens
| Metric | GPT-4 | GPT 5.6 Sol |
|---|---|---|
| Launch Date | March 2023 | — |
| Context Window | 8,192 tokens | 1 million tokens |
The cost of long memory
Processing millions of tokens requires significant computational power and financial investment. Companies race to expand working memory, assuming that holding an entire book or codebase in view at all times is always the best approach.
Yet, real-world tasks often do not require examining every single word simultaneously. Targeted lookups and periodic summaries can retrieve necessary information without the steep performance tax of massive context windows.
Alternative strategies
Instead of loading entire libraries into a single prompt, developers can break tasks into smaller segments. Periodic summaries capture the main points of earlier text, while targeted lookups pull specific facts only when needed.
These methods keep operational costs low and speed up response times. As the artificial intelligence industry matures, efficiency may take precedence over raw capacity increases.
Source: Original announcement
Worth a read?



Comments