SoftmaxGRPO: Learning to Reason using Softmax Advantage Group Estimation
Jefferson Hernandez · Jaywon Koo · Zilin Xiao · Chen Wei · Vicente Ordonez
SoftmaxGRPO replaces GRPO's symmetric z-score advantages with a temperature-scaled softmax, creating an exponential tilt that asymmetrically rewards the best reasoning trajectories.
摘要
Group-based reinforcement learning objectives such as GRPO can allocate learning signal poorly across prompt difficulty: under binary rewards, group normalization induces a divergent weighting on easy prompts. We introduce Softmax Advantage Group Estimation (SoftmaxGRPO), a drop-in alternative that replaces z-score-normalized group advantages with temperature-scaled softmax advantages, keeping weights bounded regardless of prompt difficulty. For binary rewards, we derive the exact finite-group population objective and identify MaxRL as its low-temperature limit. For bounded scalar rewards, we show that the large-group update exactly optimizes a log-moment-generating-function objective, while a universal finite-group scalar objective cannot exist without additional assumptions on the reward distribution. Empirically, SoftmaxGRPO reallocates measured gradient budget away from near-solved prompts and consistently improves over GRPO under identical rewards. It reaches 51.8\% on DeepMath with verifiable rewards and improves a 1.5B instruction-tuned model from 35.0\% to 68.0\% on Poetry using only lightweight text-similarity rewards.