一作OSDI 2026Oral· Operating Systems & Distributed Systems
Simple Is Better: Multiplication May Be All You Need for LLM Request Scheduling
Dingyan Zhang · Jinbo Han · Kaixi Zhang · Xingda Wei · Sijie Shen · Chenguang Fang · Wenyuan Yu · Jingren Zhou · Rong Chen
摘要
High-quality LLM request scheduling requires meeting two key objectives: ensuring the routed instance has KV$ to accelerate request execution, and ensuring that the workload is balanced across instances. Achieving both objectives is challenging because pursuing one may compromise the other. Curre...