Hyunjung Shim

Large Vision-Language Models (LVLMs) demonstrate a promising direction for assisting individuals with blindness or low vision (BLV). Yet, measuring their true utility in real-world scenarios is challenging because evaluating whether their descriptions are BLV-informative requires a fundamentally different approach from assessing standard scene descriptions. While the “VLM-as-a-metric” or “LVLM-as-a-judge” paradigm has emerged, existing evaluators still fall short of capturing the unique requirements of BLV-centric evaluation, lacking at least one of the following key properties: (1) High correlation with human judgments, (2) Long instruction understanding, (3) Score generation efficiency, and (4) Multi-dimensional assessment. To this end, we propose a unified framework to bridge the gap between automated evaluation and actual BLV needs. First, we conduct an in-depth user study with BLV participants to understand and quantify their navigational preferences, followed by collecting complementary sighted human annotations assessing the quality of the descriptions that affect BLV user safety. The process leads to curating VL-GUIDEDATA, a large-scale BLV user-simulated preference dataset containing image-request-response-score pairs. Our framework shows how preferences from a target user population can be systematically distilled through human studies and scaled into a reward model. We then leverage the dataset to develop an accessibility-aware evaluator, VLGUIDE-S, which outperforms existing (L)VLM judges in both human alignment and inference efficiency. Beyond BLV evaluation, VL-GUIDE-S also generalizes across diverse image-text alignment benchmarks, suggesting that user-grounded reward modeling can benefit both specialized and general-purpose evaluation. We hope our work lays a foundation for automatic AI judges that advance safe, barrier-free navigation for BLV users while providing a practical framework for building evaluators tailored to diverse user populations.