It Takes One to Bias Them All: Breaking Bad with One-Shot GRPO
Naihao Deng · Yilun Zhu · Naichen Shi · Clayton Scott · Rada Mihalcea
We show that one-shot training on a single biased example can produce systematic bias across datasets and categories.
摘要
*Warning: This paper contains several toxic and offensive statements.* Modern large language models (LLMs) are typically aligned through large-scale post-training to ensure fair and reliable behavior. In this work, we investigate how easily such guardrails can be broken by Group Relative Policy Optimization (GRPO). We show that one-shot GRPO training on a single biased example is sufficient to induce systematic bias, with stereotype-driven reasoning generalizing across attributes, categories, and benchmarks. We further find that models differ in their susceptibility based on the initial likelihood of producing biased outputs. Our results reveal a critical vulnerability in post-training: alignment can be overridden by a single example.