作者位次 #9/15ACL 2026Poster· Natural Language Processing
Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training
Changhao Jiang · Ming Zhang · Yifei Cao · Junjie Ye · Xiaoran Fan · Shihan Dou · Zhiheng Xi · Jiajun Sun · Yi Dong · Yujiong Shen · Jingqi Tong · Baoyu Fan · Tao Gui · Qi Zhang · Xuanjing Huang
摘要
The GPT-4 technical report suggests that downstream performance can be predicted from pre-training signals, but offers little methodological detail on how to quantify this. This work address this gap by modeling knowledge retention, the capacity of a pre-trained language model to memorize factual...