Soroush Vosoughi

Multi-agent deliberation is often proposed as a way to improve the safety and reliability of AI systems. We show that it can also create a channel through which misalignment spreads. In particular, a single emergently misaligned agent, obtained through narrow fine-tuning without adversarial intent, can shift the privately elicited post-deliberation stances of an otherwise aligned majority through ordinary structured debate. We refer to this phenomenon as misalignment contagion. To study it, we use a three-stage protocol designed to distinguish public conformity during discussion from persistent post-discussion change under private re-elicitation. Across 35,290 trials spanning five normative and safety-relevant datasets and four communication topologies, we find that the effect is not well described as surface compliance alone. In many settings, shifted stances persist when agents are queried again in a fresh non-social context, with the strongest persistence arising in chain topologies. We also find that, within our tested settings, increasing the size of the misaligned minority does not increase contagion, while placing a misaligned agent in a structurally central position increases the depth of persistence. More generally, topology shapes not only how much agents move during deliberation, but whether that movement survives private re-elicitation. Finally, prompting aligned agents to be more rigid does not reduce susceptibility and can instead increase it. The effect is stable across group size, deliberation depth, and the domain on which the minority's misalignment was induced. These results show that component-level alignment does not by itself guarantee aligned behavior at the level of the interacting system.

We present Future-KL Influenced Policy Optimization (FIPO), a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models. While GRPO style training scales effectively, it typically relies on outcome-based rewards (ORM) that distribute a global advantage uniformly across every token in a trajectory. We argue that this coarse-grained credit assignment imposes a performance ceiling by failing to distinguish critical logical pivots from trivial tokens. FIPO addresses this by incorporating discounted future-KL divergence into the policy update, creating a dense advantage formulation that re-weights tokens based on their influence on subsequent trajectory behavior. Empirically, FIPO enables models to break through the length stagnation seen in standard baselines. Evaluated on Qwen2.5-32B, FIPO extends the average chain-of-thought length from roughly 4,000 to over 10,000 tokens and increases AIME 2024 Pass@1 accuracy from 50.0% to a peak of 58.0% (converging at approximately 56.0%). This outperforms both DeepSeekR1-Zero-Math-32B (∼ 47.0%) and o1-mini (∼ 56.0%). Our results suggest that establishing dense advantage formulations is a vital path for evolving ORMbased algorithms to unlock the full reasoning potential of base models. We open-source our training system, built on the verl framework.