Olmo-core 3 introduces an open stack for large MoE training

On October 1, 2026, Olmo-core developers released version 3, an open training framework for large mixture-of-experts models and future Olmo development. It replaces repeated weight gathering with a design that keeps experts on GPUs and routes relevant data to them.

On eight NVIDIA B300 GPUs, developers report a preliminary result of 52,000 tokens per second per GPU for a 47-billion-parameter model, versus 19,400 on the prior stack. A separate benchmark grew the expert pool from 8 to 128, choosing four per token; total capacity rose from 4.6B to 47B while throughput fell less than 5%.

The system was also tested at 1.2 trillion parameters across 512 GPUs, using random routing to measure system performance rather than trained-model quality. The developers say researchers and developers can use the open stack to train their own MoEs and adapt it to different hardware.