Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training
Kailai Yang · Xiao Liu · Lei Ji · Hao Li · Xiao Liang · Zhiwei Liu · Yeyun Gong · Peng CHENG · Mao Yang
摘要
Continual pre-training on small-scale task-specific data is an effective method for improving large language models in new target fields, yet it risks catastrophic forgetting of their original capabilities. A common solution is to re-weight training data mixtures from source and target fields on...