Yoshua Bengio

Reinforcement learning (RL) has proven effective for fine-tuning large language models (LLMs), significantly enhancing their reasoning abilities in domains such as mathematics and code generation. A crucial factor influencing RL fine-tuning success is the training curriculum: the order in which training problems are presented. While random curricula serve as common baselines, they remain suboptimal; manually designed curricula often rely heavily on heuristics, and online filtering methods can be computationally prohibitive. To address these limitations, we propose \textit{Self-Evolving Curriculum (SEC)}, an automatic curriculum learning method that learns a curriculum policy concurrently with the RL fine-tuning process. Our approach formulates curriculum selection as a non-stationary Multi-Armed Bandit problem, treating each problem category (e.g., difficulty level or problem type) as an individual arm. We leverage the absolute advantage from policy gradient methods as a proxy measure for immediate learning gain. At each training step, the curriculum policy selects categories to maximize this reward signal and is updated using the TD(0) method. Across three distinct reasoning domains: planning, inductive reasoning, and mathematics, our experiments demonstrate that SEC significantly improves models' reasoning capabilities, enabling better generalization to harder, out-of-distribution test problems. Additionally, our approach achieves better skill balance when fine-tuning simultaneously on multiple reasoning domains. These findings highlight SEC as a promising strategy for RL fine-tuning of LLMs.

The reverse Kullback-Leibler (KL) divergence between the trained policy and a reference policy is commonly used as a regularizer when training large language models (LLMs) with reinforcement learning (RL). Since computing the exact sequence-level KL divergence is intractable, practical algorithms use sample-based estimators computed from on-policy rollouts, and incorporate them in the reward or the loss. Despite its ubiquity, the interaction between estimator choice, position of the regularization and downstream performance is poorly understood. Recent work also identifies that certain implementation choices can result in biased gradients. We further analyze these practices and examine the gradients of several estimator configurations, showing how the choices shape gradient bias. We substantiate these observations with empirical evidence by RL fine-tuning LLMs with different KL configurations, and evaluating their performance on both in- and out-of-distribution tasks. In on-policy settings, we find that biased-gradient estimator configurations can cause training instabilities, while unbiased-gradient configurations lead to better performance on in-domain as well as out-of-domain tasks. We also observe that KL regularization can help stabilize asynchronous training. Overall, our findings provide useful takeaways for using KL-regularized objectives during RL post-training of LLMs.