作者位次 #3/5OSDI 2026Oral· Operating Systems & Distributed Systems
Achieving Cloud-Grade SLOs for Local Mixture-of-Experts Inference through CPU–GPU Hybrid Design
Wenxin Wang · Yule Hou · Yu Ji · Peng Qu · Youhui Zhang
摘要
Local deployment of large Mixture-of-Experts (MoE) models falls short of the service quality achieved in cloud-scale environments, even under low-concurrency workloads. We identify four key gaps in local MoE inference: reliance on capacity-reduced models (quantized, distilled, rerouted), inabilit...