When Correct Patches Become Vulnerable: Red Teaming LLM-Based Program Repair Agents
Simin Chen · Yixin He · Suman Jana · Baishakhi Ray
We aruge there are some issues with current APR agent evaluation, and propose a automatic red teaming apporoach to make the program repair agents to generate functional correct (pass all unit tests) yet vulnerable patches
摘要
LLM-based agents are increasingly used for automated program repair (APR) to support software maintenance. Given a GitHub issue describing a bug, these agents automatically generate patches to fix the reported problem. However, existing research on APR agents focuses primarily on improving functional correctness—whether generated patches pass regression tests—while largely overlooking security risks. This raises an important question: \textit{Can an adversarial user submit a valid GitHub issue that misleads an LLM-based APR agent into generating a functionally correct but vulnerable patch?} To answer this question, we propose \tool, which generates adversarial issue statements that induce APR agents to produce functionally correct yet vulnerable patches. Empirical evaluation on three agent pipelines and five backend LLMs shows that \tool can produce patches that are both functionally correct and vulnerable (the attack success rate on the correct patch could reach 0.91, whereas the baseline ASRs are all below 0.20). Our evaluation challenges the assumption that \textit{passing all tests guarantees a patch’s reliability and security, revealing critical limitations in the current APR evaluation paradigm.} Our code is available at GitHub.