一作OSDI 2026Oral· Operating Systems & Distributed Systems
Kairox: Adaptive GPU-CPU Hybrid LLM Inference via Online Neuron Balancing
Yapeng Jiang · Minghao Gan · Zicong Hong · Wuhui Chen · Junyuan Liang · Yue Yu · Meng Guo · Zibin Zheng
摘要
Hybrid LLM inference systems exploit both GPU and CPU for computation and memory but are bottlenecked by the CPU’s lower computational capabilities. Recent approaches leverage activation sparsity by offline partitioning FFN neurons into "hot" (frequently activated) and "cold" (rarely activated) s...