PreMoE: Proactive Inference for Efficient Mixture-of-Experts
Zehua Pei · Ying Zhang · Hui-Ling Zhen · Tao Yuan · Xianzhi Yu · Zhenhua Dong · Sinno Jialin Pan · Mingxuan Yuan · Bei Yu
This paper proposes PreMoE, an inference framework designed to improve the efficiency of MoE models during inference by predicting future expert usage patterns in advance and proactively loads or schedules experts.
摘要
Mixture-of-Experts (MoE) models offer dynamic computation, but are typically deployed as static full-capacity models, missing opportunities for deployment-specific specialization. We introduce PreMoE, a training-free framework that proactively compiles sparse MoE variants for targeted deployment scenarios. At its core is Predicted Expert Utility (PEU), a robust metric for estimating expert importance from router logits through high-confidence threshold filtering and logit transformation, which together stabilize utility estimation under aggressive sparsity. Using PEU scores computed on a small calibration set, PreMoE produces domain-aware expert rankings that can be used to compile either domain-specific specialists or high-efficiency multi-domain generalists, without any retraining. Across MoE models ranging from 30B to 718B parameters, PreMoE preserves near-full accuracy at model-dependent sparsity levels of up to 50\%; on DeepSeek-R1, 50\% sparsity halves the deployment NPU count and increases throughput by 23\%. Controlled studies across calibration sources, formats, random subsets, and domain mixtures confirm that these gains are stable rather than tied to a particular calibration set. PreMoE further exposes a practical deployment trade-off: specialists maximize in-domain efficiency, while synthesized generalists retain broader cross-domain capability at the same sparsity budget.