Recall is the primary bottleneck for factual accuracy in large language models

Recent research reveals that frontier large language models successfully encode nearly all facts, yet struggle to recall them reliably.

2 min read

A diagram mapping out five distinct states of knowledge retrieval in language models, from encoding failure to direct recall.
Diagram illustrating five knowledge retrieval states in language models, distinguishing between encoding failures and recall failures.

When large language models get facts wrong, it is easy to assume they simply never learned them. However, a new behavioral framework called knowledge profiling suggests otherwise. Researchers at Google Research have introduced an evaluation approach that separates factual encoding—the storage of facts within model parameters—from factual recall, revealing that retrieval is the true bottleneck in modern systems.

To study this, researchers built WikiProfile, a benchmark comprising 2,150 Wikipedia-derived facts paired with tasks testing encoding, recall, and recognition. The evaluation covered 13 models, generating roughly 4.5 million responses to analyze how frontier systems handle factual retrieval.

Key facts

Benchmark size
2,150 facts
Frontier encoding rate
95-98%
Direct recall failure rate
26-34%
Models evaluated
13 LLMs
Thinking recovery rate
40-65%
A stacked bar chart comparing encoding and recall performance across multiple language model families.
Stacked bar chart showing the distribution of knowledge retrieval states across various frontier language models., Source ↗

The state of encoding versus recall

Across frontier systems such as Gemini 3 Pro and GPT-5, factual encoding is near saturation, with models successfully storing 95 to 98 percent of facts. Despite this high encoding rate, these models fail to directly recall 26 to 34 percent of those same facts. Even when using thinking mechanisms to aid retrieval, the models still miss 11 to 12 percent of facts.

This gap indicates that scaling model size or data coverage improves what a model stores far more effectively than it improves what a model can access. Factual errors in advanced models stem primarily from knowledge that is stored but remains inaccessible.

Why recall fails

Recall is closely tied to how a model originally encountered a fact during training. When a query diverges from the training-time context or phrasing, retrieval breaks down. This dynamic explains two persistent challenges in language models:

  • Rare facts: Low-popularity facts are encoded at rates comparable to high-popularity facts, but they suffer from much larger recall gaps.
  • Reverse questions: The reversal curse—where a model knows that A is B but cannot answer what B is—disappears in multiple-choice recognition tasks, proving that the bidirectional knowledge is encoded but difficult to generate in open-ended recall.
Bar charts comparing encoding and recall metrics for high versus low popularity facts across models.
Bar charts demonstrating that language models encode and recall high popularity facts better than low popularity facts., Source ↗

The role of thinking mechanisms

Thinking-optimized models use intermediate computations before producing a final answer. The research shows that thinking acts primarily as a recovery mechanism for stored knowledge rather than a tool for acquiring new information. In thinking-optimized models, this process recovers 40 to 65 percent of encoded-but-not-directly-known facts, helping to narrow popularity and directionality gaps.

These findings suggest that future improvements in factuality will rely less on brute-force scaling and more on developing better retrieval and utilization strategies for knowledge already present within model parameters.

Source: Original announcement

Worth a read?

Report

Comments