Scaling foundation models for digital advertising systems
Recent advances in recommendation foundation models bring large-language-model-style scaling to digital advertising.
2 min read·Wednesday, 14 October
GEM architecture scaling overview for frontier ads AI.
Digital advertising systems have traditionally relied on isolated prediction models tailored to specific surfaces like feeds or video streams. A new approach shifts this paradigm by consolidating these separate models into a single, unified foundation model. This central model learns from data across all surfaces to deliver improved performance and more relevant ad placements.
Performance comparison chart showing NE gains across GFLOPs for the Kunlun architecture. · Source ↗
Overcoming scaling hurdles
Traditional recommendation systems have struggled to follow the predictable scaling laws seen in text-based language models. Because advertising data involves massive, highly diverse features and constantly shifting distributions, simply adding more compute does not always yield proportional improvements. To address this, developers created a specialized architecture known as Kunlun, which is co-designed with modern graphics processing hardware to handle long sequences of user engagement history efficiently.
This new architecture introduces several technical adjustments:
Block attention: Replaces standard full attention mechanisms with sparse alternatives to reduce computational complexity relative to window size.
Mixture-of-experts design: Employs expert parallelism, allowing different processing sub-networks to run simultaneously across hardware accelerators.
Component skip connections: Selectively deactivates certain sequence modules to reduce floating-point operations with minimal quality loss.
Efficient knowledge transfer
Running a massive foundation model in real-time for every user request is computationally prohibitive and introduces unacceptable latency. To solve this, developers use post-training techniques to transfer intelligence from the central brain down to smaller, downstream models without adding real-time latency. A structured representation framework, along with elastic pre-computation, helps harvest the gains of the upstream model efficiently.
Scaling up training also required major improvements in hardware utilization. By combining low-precision custom kernels with novel multi-dimensional parallelism strategies, end-to-end training efficiency reaches significantly higher hardware utilization rates compared to earlier generations.
The Ultimate 2026 Free AI API Engine: Top 10 Provider Blueprints & Setup GuideFree guide
Comments