Aritra Dutta

Eye gaze offers multimodal language models a grounding signal for understanding user interfaces, such as inferring which element a user intends to interact with. But natural gaze on dense interfaces rarely lands on a single element box or semantic group. Users scan between distant regions and revisit earlier elements, and their attention often spreads across different parts of the layout. Existing research either studies UI eye tracking without aligned language supervision or uses gaze to guide models in natural image and egocentric settings. We introduce GLAZE, the first benchmark for gaze-language grounding on web user interfaces. GLAZE is built from a think-aloud study in which 27 participants described real-world webpages while an eye tracker recorded their gaze. The dataset pairs more than 10,000 UI elements across 176 webpages with time-aligned gaze and verbal descriptions. The benchmark provides a model with a screenshot and a gaze visualization and evaluates whether it can identify the interface content or function the user is focusing on. We evaluate seven open and proprietary multimodal language models across five gaze visualizations. Current models often identify the general region being viewed but miss the precise attended elements and their function. Heatmaps and scanpaths are most effective for different UI element types, yet overall performance remains limited across visualization methods. Our benchmark can be useful for future research on UI agents, gaze-based accessibility tools, and automated interface usability analysis.