作者位次 #5/10OSDI 2026Oral· Operating Systems & Distributed Systems
BatchGen: An Architecture for Scalable and Efficient Batch Inference
Tairan Xu · Leyang Xue · Zhan Lu · Jinfu Deng · Hongyang Xiao · Yinsicheng Jiang · Congjie He · Matej Sandor · Le Xu · Luo Mai
摘要
Batch inference has become a central mode of AI computation, yet existing inference engines still rely on execution models designed for interactive serving. When scaled to millions of sequences, batch workloads reveal two fundamental requirements: the ability to handle extreme inter- and intra-se...