AMD announces Instella-MoE, a fully open mixture-of-experts model

AMD has introduced Instella-MoE, a fully open Mixture-of-Experts language model featuring 16 billion total parameters and 2.8 billion active parameters per token. Trained from scratch on AMD Instinct MI300X and MI325X GPUs using the AMD ROCm software stack, the model combines a sparsely activated MoE design with architectural innovations including Gated Multi-head Latent Attention and FarSkip-Collective connectivity.

Comprehensive Training Pipeline Transparency

What distinguishes the Instella-MoE release is its extensive transparency across the development process. AMD is publishing model weights and checkpoints from every stage of the training pipeline, covering pre-training, mid-training, long-context extension, supervised fine-tuning, direct preference optimization, and a final reasoning-focused reinforcement learning checkpoint named Instella-MoE-16B-A3B-Think. Additionally, the release provides training configurations, data mixtures, and both training and inference code.

Architectural Innovations and Efficiency

The model's architecture consists of 27 decoder layers with a hidden size of 2048, utilizing 2 shared experts and 6 routed experts selected from 64 available routed experts per token. By incorporating Gated Multi-head Latent Attention, the model applies a lightweight learned output gate to MLA outputs, increasing expressivity. Furthermore, FarSkip-Collective modifies standard MoE connectivity to overlap communication with computation during expert-parallel training, delivering significant training speedups and reducing Time to First Token during serving.

Ecosystem and Availability

The training process leveraged open-source frameworks such as Primus for training and Miles for reinforcement learning. While the codebase is made available under the MIT license, the model weights are released under a ResearchRAIL license intended strictly for academic and research use.