Ming Jin

Large language models are increasingly used as penetration-testing copilots, but tool-interactive performance remains brittle: small command errors derail progress, recovery is inconsistent, and long-horizon context is easily lost. We present a failure-aware penetration-testing framework that makes error states explicit by requiring the model to emit typed failure tokens together with a structured action protocol. The model is trained to produce protocol-constrained outputs with planning, action, and completion tags, while conditioning on a compact trajectory memory and using training-time reasoning guidance plus unlikelihood loss to reduce spurious failure predictions. Evaluated on Cybench, the framework improves protocol compliance, step-wise decision quality, and subtask completion over standard fine-tuning baselines. Analysis further shows that failure-type prediction is not merely descriptive: representations become more separable under failure-aware training, and correcting the predicted failure type on confused cases leads to substantially better downstream recovery, indicating that explicit failure classification serves as a useful control signal for action selection.

Reinforcement learning from human feedback (RLHF) is vulnerable to reward hacking: policies can exploit spurious correlations in learned reward models to increase proxy reward while degrading true alignment. Existing defenses are largely static. They regularize optimization, improve the reward model, or target known biases, but do not explicitly detect exploitation as it emerges during training. We propose \textbf{Adversarial Reward Auditing (ARA)}, a two-stage framework for detecting and mitigating reward hacking. In Stage 1, a Hacker policy searches for vulnerabilities in a frozen reward model while an Auditor learns to detect exploitative responses from the reward model's internal representations. In Stage 2, the trained Auditor guides RLHF by downweighting rewards for responses likely to be exploitative. Across sycophancy, length bias, and code gaming, ARA yields the strongest alignment--utility tradeoff among competitive baselines, reducing exploitation while preserving or improving task performance. We also find cross-domain transfer in both attack and defense: a Hacker trained on code gaming increases sycophancy despite receiving no direct reward for that behavior, while an Auditor trained in one domain helps suppress exploits in others. These results suggest that reward hacking is a dynamic, transferable phenomenon and that auditing optimization in reward-model representation space is an effective way to control it.