作者位次 #3/10ACL 2026Poster· Natural Language Processing
B ench M arker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks
Nishant Balepur · Bhavya Rajasekaran · Hyunjin Jane Oh · Michael Xie · Atrey Desai · Vipul Gupta · Steven James Moore · Eunsol Choi · Rachel Rudinger · Jordan Lee Boyd-Graber
摘要
Multiple-choice question answering (MCQA) is standard in NLP, but benchmarks lack rigorous quality control. We present BenchMarker, an education-inspired toolkit using LLM judges to flag three common MCQ flaws: 1) contamination—items appearing exactly online; 2) shortcuts—cues in the choices that...