一作ACL 2026Workshop· Natural Language Processing
Optimizing Packing and Shuffling Strategies for Enhanced Performance in Generative Language Models
Yanbing Chen · Ruilin Wang · Zihao Yang · Lavender Yao Jiang · Eric Karl Oermann
摘要
Packing and shuffling tokens is a common practice in training auto-regressive language models to prevent overfitting and improve efficiency. Documents are typically concatenated to chunks of maximum sequence length (MSL) and shuffled in chunks of tokens (atom-size chunk), possibly breaking contex...