Recall is the primary bottleneck for factual accuracy in large language models
Recent research reveals that frontier large language models successfully encode nearly all facts, yet struggle to recall them reliably.
2 min read, Thursday, 13 August

When large language models get facts wrong, it is easy to assume they simply never learned them. However, a new behavioral framework called knowledge profiling suggests otherwise. Researchers at Google Research have introduced an evaluation approach that separates factual encoding—the storage of facts within model parameters—from factual recall, revealing that retrieval is the true bottleneck in modern systems.
To study this, researchers built WikiProfile, a benchmark comprising 2,150 Wikipedia-derived facts paired with tasks testing encoding, recall, and recognition. The evaluation covered 13 models, generating roughly 4.5 million responses to analyze how frontier systems handle factual retrieval.
Key facts
- Benchmark size
- 2,150 facts
- Frontier encoding rate
- 95-98%
- Direct recall failure rate
- 26-34%
- Models evaluated
- 13 LLMs
- Thinking recovery rate
- 40-65%

The state of encoding versus recall
Across frontier systems such as Gemini 3 Pro and GPT-5, factual encoding is near saturation, with models successfully storing 95 to 98 percent of facts. Despite this high encoding rate, these models fail to directly recall 26 to 34 percent of those same facts. Even when using thinking mechanisms to aid retrieval, the models still miss 11 to 12 percent of facts.
This gap indicates that scaling model size or data coverage improves what a model stores far more effectively than it improves what a model can access. Factual errors in advanced models stem primarily from knowledge that is stored but remains inaccessible.
Why recall fails
Recall is closely tied to how a model originally encountered a fact during training. When a query diverges from the training-time context or phrasing, retrieval breaks down. This dynamic explains two persistent challenges in language models:
- Rare facts: Low-popularity facts are encoded at rates comparable to high-popularity facts, but they suffer from much larger recall gaps.
- Reverse questions: The reversal curse—where a model knows that A is B but cannot answer what B is—disappears in multiple-choice recognition tasks, proving that the bidirectional knowledge is encoded but difficult to generate in open-ended recall.

The role of thinking mechanisms
Thinking-optimized models use intermediate computations before producing a final answer. The research shows that thinking acts primarily as a recovery mechanism for stored knowledge rather than a tool for acquiring new information. In thinking-optimized models, this process recovers 40 to 65 percent of encoded-but-not-directly-known facts, helping to narrow popularity and directionality gaps.
These findings suggest that future improvements in factuality will rely less on brute-force scaling and more on developing better retrieval and utilization strategies for knowledge already present within model parameters.
Source: Original announcement
Worth a read?



Comments