Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts
Jacob Morrison · Sanjay Adhikesaven · Akshita Bhagia · Matei Zaharia · Noah A. Smith · Sewon Min
A modular MoE approach lets you extend a post-trained LM with new independently trained domain experts (e.g. math, code, tool use, safety) and composed via lightweight router training
摘要
Extending a fully post-trained language model with new domain capabilities is fundamentally limited by monolithic training paradigms: retraining from scratch is expensive and scales poorly, while continued training often degrades existing capabilities. We present BAR (Branch-Adapt-Route), which trains independent domain experts, each through its own mid-training, supervised finetuning, and reinforcement learning pipeline, and composes them via a Mixture-of-Experts architecture with lightweight router training. Unlike retraining approaches that mix all domains and require full reprocessing for any update (with cost scaling quadratically), BAR enables updating individual experts independently with linear cost scaling and no degradation to existing domains. At the 7B scale, with experts for math, code, tool use, and safety, BAR achieves an overall score of 49.1, outperforming retraining without mid-training (47.8), and nearly matching retraining with mid-training (50.5). We further show that modular training provides a structural advantage: by isolating each domain, it avoids the catastrophic forgetting that occurs when late-stage RL degrades capabilities from earlier training stages, while significantly reducing the cost and complexity of updating or adding a domain. Together, these results suggest that decoupled, expert-based training is a scalable alternative to monolithic retraining for extending language models.