Bang Liu

GUI grounding, the task of mapping natural-language instructions to pixel coordinates, is crucial for autonomous computer-use agents (CUA) yet remains difficult for current VLMs. The core bottleneck is reliable patch-to-pixel mapping: current approaches generate coordinates as text tokens directly from visual features, forcing the model to infer complex position-to-pixel mappings implicitly. This implicit regression is unstable and breaks when extrapolating to high-resolution displays unseen during training. We address this with RULER tokens, explicit coordinate markers that share positional embeddings with image patches, letting the model reference known positions and adjust within a bounded range rather than regress coordinates from scratch. We also propose Interleaved MRoPE (I-MRoPE), which corrects a frequency imbalance in standard positional encodings so that width and height dimensions receive equal representational capacity. Experiments on ScreenSpot, ScreenSpot-V2, and ScreenSpot-Pro show consistent grounding improvements, with the largest gains on high-resolution displays unseen during training (+2.6% on ScreenSpot-Pro), achieved with less than 1% additional tokens.

We study continual skill acquisition in open-ended embodied environments where an agent must construct, refine, and reuse an expanding library of executable skills. We introduce the Programmatic Skill Network (PSN), a framework in which skills are executable symbolic programs forming a compositional network that evolves through experience. PSN defines three core mechanisms instantiated via large language models: (1) REFLECT for structured fault localization over skill compositions, (2) progressive optimization with maturity-aware update gating that stabilizes reliable skills while maintaining plasticity for uncertain ones, and (3) canonical structural refactoring under rollback validation that maintains network compactness. We show that PSN's learning dynamics exhibit structural parallels to neural network training. Experiments on MineDojo and Crafter demonstrate robust skill reuse, rapid adaptation, and strong generalization across open-ended task distributions.