Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration
Eliya Habba · Itay Itzhak · Asaf Yehudai · Yotam Perlitz · Elron Bandel · Michal Shmueli-Scheuer · Leshem Choshen · Gabriel Stanovsky
We introduce a psychometric framework using IRT and fixed-anchor calibration that enables efficient and continuous LLM evaluation while maintaining comparable scores across evolving benchmark suites.
摘要
The rapid release of both language models and benchmarks makes it increasingly costly to evaluate $\textit{every}$ model on $\textit{every}$ dataset. In practice, models are often evaluated on different samples, making scores difficult to compare across studies. To address this, we propose a framework based on multidimensional Item Response Theory (IRT) that uses anchor items to calibrate new benchmarks to the evaluation suite while holding previously calibrated item parameters fixed. Our approach supports a realistic evaluation setting in which datasets are introduced over time and models are evaluated only on the datasets available at the time of evaluation, while a fixed anchor set for each dataset is used so that results from different evaluation periods can be compared directly. In large-scale experiments on more than $400$ models, our framework predicts full-evaluation performance within 2--3 percentage points using only $100$ anchor questions per dataset, with Spearman $\rho \geq 0.9$ for ranking preservation. This shows that benchmark suites can grow over time while preserving score comparability, since adding a new dataset requires running existing models only on that dataset’s anchors.