作者位次 #3/8ACL 2026Poster· Natural Language Processing
Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training
Lei Liu · Hao Zhu · Xiaoyan Yang · YUE SHEN · Zhixuan Chu · Jian Wang · Jinjie Gu · Kui Ren
摘要
Continual Pre-training (CPT) serves as a fundamental approach for adapting foundation models to domain-specific applications. Scaling laws for pre-training define a power-law relationship between dataset size and the test loss of an LLM. However, the marginal gains from simply increasing data for...