Introducing K2-Horizon: a fleet of six open language models
K2-Horizon introduces six open language models built for general reasoning and agentic tasks.
2 min read, Wednesday, 18 November

The K2 Team has released K2-Horizon, a fleet of six fully open language models ranging from a 0.9B dense model to a 375B-A23B mixture of experts — an architecture that routes tokens to specialized sub-networks to increase capacity efficiently. These models are designed for general-purpose reasoning and agentic workflows across software engineering, math, science, and office automation.
Key facts
- Fleet Size
- Six open language models
- Model Range
- 0.9B dense model to a 375B-A23B MoE
- Synthetic Tokens
- Roughly 10T synthetic pre-training tokens
- Pre-training Corpus
- 22T-token pre-training corpus
- Decoding Speedup
- Up to 2.2x
Architectural Innovations
To improve reasoning capabilities, the team introduced two primary techniques. Mixture-of-Value Attention applies mixture of experts to attention value projections, while softplus attention gating provides a more flexible, theoretically grounded gating mechanism. The 36B-A4B model uses both techniques and nearly matches a 32B dense model trained on the same data while using only one-seventh of the active parameters.
Additionally, the release includes Uno, which are lossless diffusion adapters that speed up text generation decoding by up to 2.2 times without requiring a separate draft model.
Open Source Transparency and Data
The release follows the LLM360 definition of open source to ensure full transparency and reproducibility. The team published the full development history, including intermediate checkpoints from every training stage, pre-training data, training logs, and all infrastructure code. This includes the production-scale xLLM framework for dense and mixture of experts training, as well as the RL360 reinforcement-learning stack.
The project also documents every pipeline used to build roughly 10 trillion synthetic pre-training tokens, demonstrating that synthetic data can serve as a primary ingredient for large-scale pre-training. Furthermore, four of the models share a single 22T-token pre-training corpus and recipe, enabling controlled studies of scale and sparsity.
Source: Original announcement
Worth a read?



Comments