VRPO : Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training
Dingwei Zhu · Shihan Dou · Zhiheng Xi · Senjie Jin · Guoqiang Zhang · Jiazheng Zhang · Junjie Ye · Mingxu Chai · Enyu Zhou · Ming Zhang · Yuhui Wang · Caishuang Huang · Chenhao Huang · Yunke Zhang · Yuran Wang · Tao Gui · Qi Zhang · Xipeng Qiu · Xuanjing Huang
摘要
Reinforcement Learning (RL) in real-world environments often suffers from ambiguous or incomplete reward supervision, which undermines policy stability and generalization. Such noise may cause models to ignore key information or even collapse in advantage estimation. We find that a strong value m...