$V_{0.5}$: Generalist Value Model as a Prior for Sparse RL Rollouts
Yi-Kai Zhang · Yueqing Sun · Hongyan Hao · Qi GU · Xunliang Cai · Long Chen · De-Chuan Zhan · Han-Jia Ye
$V_{0.5}$ adaptively fuses a generalist value model prior with sparse rollouts to create a low-variance, compute-efficient baseline, significantly accelerating RL convergence and boosting reasoning performance.
摘要
In Reinforcement Learning with Verifiable Rewards (RLVR), constructing a robust advantage baseline is critical for policy gradients, effectively guiding the policy model to reinforce desired behaviors. Recent research has introduced Generalist Value Models (such as $V_0$), which achieve pre-trained value estimation by explicitly encoding model capabilities in-context, eliminating the need to synchronously update the value model alongside the policy model. In this paper, we propose $V_{0.5}$, whose advantage baseline is a convex combination of the frozen value model's prior prediction and the empirical rollout mean. An adaptive weight automatically governs the balance based on the real-time agreement between the two estimators. When the prior is reliable, the baseline heavily leverages the value model for variance reduction. When a conflict is detected, the baseline safely reverts to the unbiased empirical mean, and $V_{0.5}$ dynamically allocates additional rollout budget to sharpen the estimate. This mechanism minimizes the baseline estimator's Mean Squared Error (MSE), guaranteeing stable policy gradients, even under extreme sparsity with a group size of 4. Extensive evaluations across six mathematical reasoning benchmarks demonstrate that $V_{0.5}$ significantly outperforms GRPO and DAPO, achieving faster convergence and up to 12\% accuracy improvement.