一作ACL 2026Poster· Natural Language Processing
Lost in Simulation: LLM -Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations
Preethi Seshadri · Samuel Cahyawijaya · Ayomide Odumakinde · Sameer Singh · Seraphina Goldfarb-Tarrant
摘要
Agentic benchmarks increasingly rely on LLM-simulated users to scalably evaluate agent performance, yet the robustness, validity, and fairness of this approach remain unexamined. Through a user study with participants across the United States, India, Kenya, and Nigeria, we investigate whether LLM...