一作ACL 2026Poster· ACL Industry Track
Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design
Junzhuo Li · Peijie Jiang · Changxin Tian · Jia Liu · Zhiqiang Zhang · Xuming Hu
摘要
This paper presents a novel extension of neural scaling laws to Mixture-of-Experts (MoE) models, focusing on the optimal allocation of compute between expert and attention sub-layers. As MoE architectures have emerged as an efficient method for scaling model capacity without proportionally increa...