BatchGen: An Architecture for Scalable and Efficient Batch Inference
Tairan Xu · Leyang Xue · Zhan Lu · Jinfu Deng · Hongyang Xiao · Yinsicheng Jiang · Congjie He · Matej Sandor · Le Xu · Luo Mai
摘要
Batch inference has become a central mode of AI computation, yet existing inference engines still rely on execution models designed for interactive serving. When scaled to millions of sequences, batch workloads reveal two fundamental requirements: the ability to handle extreme inter- and intra-se...