Junshan Zhang

Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values. While benchmarks for general response quality are prevalent, evaluating how well reward models account for individual user preferences remains an open challenge. To bridge this gap, we introduce **Personalized RewardBench**, a novel benchmark designed to rigorously assess reward models' capacity to model personalized preferences. We construct chosen and rejected response pairs based on strict adherence to (or violation of) user-specific rubrics, ensuring that preference distinctions are uniquely tailored to the individual. In particular, human evaluations confirm that the primary discriminative factor between pairs is strictly personal preference, with both responses maintaining high general quality (e.g., correctness and helpfulness). Extensive testing reveals that existing state-of-the-art reward models struggle significantly with personalization, peaking at an accuracy of just 75.94\%. Crucially, because an effective reward model benchmark should predict a reward model's performance on downstream tasks, we conduct experiments demonstrating that our benchmark exhibits a significantly higher correlation with downstream performance under both Best-of-N (BoN) sampling and Proximal Policy Optimization (PPO) compared to existing baselines. These findings establish Personalized RewardBench as a robust and accurate proxy for evaluating reward models' performance in downstream applications.

Reinforcement learning from human feedback (RLHF) is vulnerable to reward hacking: policies can exploit spurious correlations in learned reward models to increase proxy reward while degrading true alignment. Existing defenses are largely static. They regularize optimization, improve the reward model, or target known biases, but do not explicitly detect exploitation as it emerges during training. We propose \textbf{Adversarial Reward Auditing (ARA)}, a two-stage framework for detecting and mitigating reward hacking. In Stage 1, a Hacker policy searches for vulnerabilities in a frozen reward model while an Auditor learns to detect exploitative responses from the reward model's internal representations. In Stage 2, the trained Auditor guides RLHF by downweighting rewards for responses likely to be exploitative. Across sycophancy, length bias, and code gaming, ARA yields the strongest alignment--utility tradeoff among competitive baselines, reducing exploitation while preserving or improving task performance. We also find cross-domain transfer in both attack and defense: a Hacker trained on code gaming increases sycophancy despite receiving no direct reward for that behavior, while an Auditor trained in one domain helps suppress exploits in others. These results suggest that reward hacking is a dynamic, transferable phenomenon and that auditing optimization in reward-model representation space is an effective way to control it.