RSDA : Restoring Stale Data Affinity via Dynamic Renovation Strategy for Mitigating Data Scarcity
Yidan Liang · Jia Zhu · Weijie Shi · Hanghui Guo · Yue Cui · Jiawei Shen · Guoqing Ma · Jingjiang Liu · Qingyu Niu · Yilin Wang · Shimin Di · Jiajie Xu
摘要
High-quality data is the cornerstone of advancing large language models. However, the field currently faces a critical dilemma: the supply of premium data is nearing depletion, while vast stale corpora remain underutilized. Our empirical analysis reveals that training models on such data directly...