PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost
Junkeun Yi · Damon Mosk-Aoyama · Baihe Huang · Ritu Gala · Charles Wang · Sugam Devare · Khushi Bhardwaj · Abhibha Gupta · Oleksii Kuchaiev · Jiantao Jiao · Jian Zhang · Venkat Krishna Srinivasan
PivotRL is a post-training framework that combines the compute efficiency of SFT with the OOD accuracy retention of E2E RL by performing local on-policy rollouts at informative pivot turns in existing SFT trajectories.
摘要
Post-training for long-horizon agentic tasks has a tension between compute efficiency and generalization. While supervised fine-tuning (SFT) is compute efficient, it often suffers from out-of-domain (OOD) degradation. Conversely, end-to-end reinforcement learning (E2E RL) preserves OOD capabilities, but incurs high compute costs due to many turns of on-policy rollout. We introduce PivotRL, a novel framework that operates on existing SFT trajectories to combine the compute efficiency of SFT with the OOD accuracy of E2E RL. PivotRL relies on two key mechanisms: first, it executes local, on-policy rollouts and filters for pivots: informative intermediate turns where sampled actions exhibit high variance in outcomes; second, it utilizes rewards for functional-equivalent actions rather than demanding strict string matching with the SFT data demonstration. We theoretically show that these mechanisms incentivize strong learning signals with high natural gradient norm, while preserving policy probability ordering on actions unrelated to training tasks. In comparison to standard SFT on identical data, we demonstrate that PivotRL achieves +4.17% higher in-domain accuracy on average across four agentic domains, and +10.04% higher OOD accuracy in non-agentic tasks. Notably, on agentic coding tasks, PivotRL achieves competitive accuracy with E2E RL with 4x fewer rollout turns.