Improving GUI Grounding with Explicit Position-to-Coordinate Mapping
Suyuchen Wang · Tianyu Zhang · Ahmed Masry · Christopher Pal · Spandana Gella · Bang Liu · Perouz Taslakian
RULER tokens provide explicit position-to-pixel mapping and I-MRoPE balances spatial encoding, transforming GUI grounding from regression to referencing with strong high-resolution generalization.
摘要
GUI grounding, the task of mapping natural-language instructions to pixel coordinates, is crucial for autonomous computer-use agents (CUA) yet remains difficult for current VLMs. The core bottleneck is reliable patch-to-pixel mapping: current approaches generate coordinates as text tokens directly from visual features, forcing the model to infer complex position-to-pixel mappings implicitly. This implicit regression is unstable and breaks when extrapolating to high-resolution displays unseen during training. We address this with RULER tokens, explicit coordinate markers that share positional embeddings with image patches, letting the model reference known positions and adjust within a bounded range rather than regress coordinates from scratch. We also propose Interleaved MRoPE (I-MRoPE), which corrects a frequency imbalance in standard positional encodings so that width and height dimensions receive equal representational capacity. Experiments on ScreenSpot, ScreenSpot-V2, and ScreenSpot-Pro show consistent grounding improvements, with the largest gains on high-resolution displays unseen during training (+2.6% on ScreenSpot-Pro), achieved with less than 1% additional tokens.