Jingren Zhou

We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models. While GRPO style training scales effectively, it typically relies on outcome-based rewards (ORM) that distribute a global advantage uniformly across every token in a trajectory. We argue that this coarse-grained credit assignment imposes a performance ceiling by failing to distinguish critical logical pivots from trivial tokens. FIPO addresses this by incorporating discounted future-KL divergence into the policy update, creating a dense advantage formulation that re-weights tokens based on their influence on subsequent trajectory behavior. Empirically, FIPO enables models to break through the length stagnation seen in standard baselines. Evaluated on Qwen2.5-32B, FIPO extends the average chain-of-thought length from roughly 4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0% (converging at approximately 56.0%). This outperforms both DeepSeekR1-Zero-Math-32B (∼ 47.0%) and o1-mini (∼ 56.0%). Our results suggest that establishing dense advantage formulations is a vital path for evolving ORMbased algorithms to unlock the full reasoning potential of base models. We open-source our training system, built on the verl framework.

We investigate the functional role of emergent outliers in large language models, specifically attention sinks (a few tokens that consistently receive large attention logits) and residual sinks (a few fixed dimensions with persistently large activations across most tokens). We hypothesize that these outliers, in conjunction with the corresponding normalizations (\textit{e.g.}, softmax attention and RMSNorm), effectively rescale other non-outlier components. We term this phenomenon \textit{outlier-driven rescaling} and validate this hypothesis across different model architectures and training token counts. This view unifies the origin and mitigation of both sink types. Our main conclusions and observations include: (1) Outliers function jointly with normalization: removing normalization eliminates the corresponding outliers but degrades training stability and performance; directly clipping outliers while retaining normalization leads to degradation, indicating that outlier-driven rescaling contributes to training stability. (2) Outliers serve more as rescale factors rather than contributors, as the final contributions of attention and residual sinks are significantly smaller than those of non-outliers. (3) Outliers can be absorbed into parameters or reduced by explicit gated rescaling, which in turn yield better training and quantization performance.

Inspired by the success of scaling language modeling through massive real-world corpora, we demonstrate that similar power-law patterns exist in preference modeling. We propose \textbf{World Preference Modeling (WorldPM)}, which model a robust consensus on intrinsic utility from real-world user interactions. We curate a 15M-scale preference dataset from online forum data and conduct large-scale experiments on models from 1.5B to 72B parameters. Across 10 objective preference test sets with consensus-based labels, we observe two distinct scaling patterns: (1) common-sense preference (e.g., identifying factual errors) exhibits stable log-linear improvement across all model sizes, while (2) domain preference (e.g., coding, math, knowledge QA) reveals an emergent phenomenon where only sufficiently large models achieve meaningful scaling. Through training dynamics analysis, we reveal the learning mechanism behind preference scaling: the model first learns to discriminate using surface preferences, but counter-surface examples progressively compel it to discover intrinsic utility beyond surface features as training scales. Further experiments validate WorldPM as an effective foundation for preference fine-tuning, broadly improving generalization across human preference datasets of varying sizes (7K, 100K, and 800K), with gains exceeding 5\% on many key subtasks.