Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
Fan Jiang · Yu Zhao · Chenyang Lyu · Tianqi Shi · Yichao Du · Feihu Jiang · Longyue Wang · Weihua Luo
A fully-transparent and efficient upcycling method to pre-train highly sparse multilingual MoE LLMs
摘要
We present Marco-MoE, a suite of fully open multilingual sparse Mixture-of-Experts (MoE) models. Marco-MoE features a highly sparse design in which only 5\% of the total parameters are activated per input token. This extreme sparsity, combined with upcycling from dense models, enables efficient pre-training on 5T tokens. Our models surpass similarly-sized competitors on English and multilingual benchmarks, achieving a best-in-class performance-to-compute ratio. Our analysis reveals that Marco-MoE learns structured expert activation patterns shared across related languages, while maintaining highly specialized utilization for linguistically isolated ones. We further show that Marco-MoE allows for scalable language expansion without the interference typical of dense models. Finally, we establish a robust post-training recipe to adapt Marco-MoE into Marco-MoE-Instruct, which consistently surpasses state-of-the-art open-weight instruct models of comparable scale. To support the community, we disclose our full training datasets, recipes, and model weights.