Scaling just-in-time test generation for codebases
Meta shares results from deploying large language models to generate executable tests before code lands in production.
2 min read·Friday, 30 October
An abstract representation of automated bug catching using large language models in codebases.
Maintaining reliability across hundreds of millions of lines of constantly evolving code presents a major engineering challenge. Traditional testing methods struggle to keep pace with rapid development cycles. To address this, Meta deployed a novel system using large language models — computer programs trained on vast amounts of text to understand and generate human-like language — to catch bugs on the fly.
Traditional Code Review: Human review based on prior experience
Catching JiTTests: Generated executable tests
Computation steps
Traditional Code Review: Fewer
Catching JiTTests: More
Behavioral difference detection
Traditional Code Review: Manual
Catching JiTTests: Automated
Metric
Traditional Code Review
Catching JiTTests
Primary mechanism
Human review based on prior experience
Generated executable tests
Computation steps
Fewer
More
Behavioral difference detection
Manual
Automated
The approach
The testing methodology relies on generating executable tests that demonstrate concrete behavioral differences introduced by a code change, known as a diff. Instead of traditional code reviews that merely inspect text, this system automatically creates tests designed to fail, surfacing bugs before code reaches production.
Tandem workflows
The infrastructure runs two main workflows in parallel. The intent-aware workflow infers the purpose of a code change and targets test generation toward likely failure points. Meanwhile, the dodgy diff workflow treats every code change as a potential bug, generating tests to expose behavioral changes without reasoning about intent.
Architecture diagram illustrating the parallel dodgy diff and intent-aware workflows used for test generation. · Source ↗
Filtering false positives
Distinguishing real bugs from false alarms is critical at scale. The system combines a rule-based environment called RubFake with an LLM-as-Judge ensemble. This ensemble aggregates judgments from several state-of-the-art large language models to classify potential issues. When human input is needed, a human-on-the-loop approach allows engineers to review outcomes at an aggregate level rather than supervising every individual output.
Production impact
Deployed across Meta's infrastructure, the system has engaged 589 engineers with a 60 percent rate of true positives. In the first half of 2026, the approach helped catch 74 serious privacy bugs before they landed in production, lowering the overall occurrence of privacy issues by 20 percent.
The Ultimate 2026 Free AI API Engine: Top 10 Provider Blueprints & Setup GuideFree guide
Comments