Bias Fitting to Mitigate Length Bias of Reward Model in RLHF
Kangwen Zhao · Jianfeng Cai · Jinhua Zhu · Ruopei Sun · Dongyun Xue · Wengang Zhou · Li Li · Houqiang Li
摘要
Reinforcement Learning from Human Feedback (RLHF) relies on reward models to align large language models with human preferences. However, RLHF often suffers from reward hacking, wherein policy learning exploits flaws in the trained reward model to maximize reward scores without genuinely aligning...