Rob Fergus

Multimodal Large Language Models (MLLMs) can ace many multimodal benchmarks *without* strong visual understanding, by exploiting linguistic priors, answer patterns, and skewed label distributions. This is particularly problematic for *vision-centric* benchmarks, which explicitly aim to require visual inputs to be solved. If a benchmark *can* be gamed, it *will* be. We take this as the starting principle for robust benchmark design. Extending established partial-input auditing to contemporary multimodal evaluation, we argue that designers should stress-test the exact artifact they release, i.e., directly "train on the test set" to probe the *specific test set* for its intrinsic, exploitable patterns. We instantiate this as a "Test-set Stress-Test" (TsT), applying $k$-fold cross-validation to text-only diagnostics within the test set. This yields held-out solvability at the benchmark level and a sample-level bias score $s(x)$ that drives "Iterative Bias Pruning" (IBP). The primary diagnostic fine-tunes an LLM with no feature engineering, while a Random Forest variant adds an interpretable designer-side audit. Across VSI-Bench, CV-Bench, MMMU, and VideoMME we find vulnerability to be heterogeneous. Template-based benchmarks show large within-Qwen gains under identical scoring (up to +33 points), while knowledge-intensive MMMU shows essentially none despite high blind scores from pretrained knowledge. Pruning tracks these diagnoses. On VSI-Bench, a designer-in-the-loop pilot cuts the fine-tuned blind score far more than matched random pruning; on MMMU, automated pruning moves the downstream vision-blind gap no more than random removal, as expected where there is no learnable shortcut to remove. TsT is a pre-release audit of an evaluation instrument, not a recipe for training deployed models on test answers.