MoRE: Mixture of Reused Experts
Eric S. Qiu · Utku Umur ACIKALIN · Justin Lovelace · Christian Belardi · Arjun B. Mulchandani · Carla P Gomes · Kilian Q Weinberger
We propose MoRE, which shares expert weights across layers using depth conditioning. This expands the routing space without memory overhead, outperforming standard MoEs and recurrent baselines matched in parameters and compute.
摘要
Mixture-of-Experts (MoE) architectures decouple model capacity from computational cost, yet incur high memory footprints as parameters grow linearly with the number of experts. Recurrent Transformers achieve parameter efficiency by reusing layer weights, but typically lack the capacity for competitive language modeling. We propose Mixture of Reused Experts (MoRE), a hybrid that shares expert pools across groups of adjacent layers. Each layer retains its own router but selects from a larger shared pool, expanding the diversity of routing combinations without additional parameters. To enable shared experts to distinguish between layers, we introduce lightweight learnable depth embeddings that condition each layer's input before routing. Experiments across three model scales (114M–1.15B parameters) show that MoRE consistently achieves lower perplexity and stronger downstream performance than standard MoEs and state-of-the-art weight-sharing architectures at matched compute and parameter budgets, with only minimal modifications to existing MoE implementations.