Ziqiao Ma

Large language model (LLM) agents are increasingly used for goal-directed tasks, yet collaboration under asymmetric information remains underexplored. We adapt Einstein Puzzles into a tabletop game in which two agents hold complementary constraints and control different parts of the workspace. Solving each game requires agents to reason, communicate, and manipulate objects toward a shared goal. We fine-tune agents under four communicative action spaces and evaluate an environment-based reasoning verifier that filters candidate actions using physical affordances, communication history, and inferred constraints. In the strongest Llama 3.1 configuration, agents that can both provide and seek information achieve the highest success rate, while mismatched communication capabilities reduce performance. Verifier-assisted inference improves success rates across all evaluated action spaces. Some variants also achieve high success without communication, but their trajectories contain more rule-inconsistent actions and rejected placement attempts. In a human study with 30 participants, non-communicative agents require a similar number of steps as fully communicative agents but are rated as more confusing. These findings show that task success alone does not capture the quality of collaborative behavior and highlight the value of bidirectional communication and environment-derived verification.

LLM agents now perform strongly in software engineering, deep research, GUI automation, and various other applications, while recent agent scaffolds and models are increasingly integrating these capabilities into unified systems. Yet, most evaluations still test these capabilities in isolation, which leaves a gap for more diverse use cases that require agents to combine different capabilities. We introduce CocoaBench, a benchmark for unified digital agents built from human designed, long horizon tasks that require flexible composition of vision, search, and coding. Tasks are specified only by an instruction and an automatic evaluation function over the final output, enabling reliable and scalable evaluation across diverse agent infrastructures. We also present CocoaAgent, a lightweight shared scaffold for controlled comparison across model backbones. Experiments show that current agents remain far from reliable on CocoaBench, with the best evaluated system achieving only 45.1% success rate. Our analysis further points to substantial room for improvement in reasoning and planning, tool use and execution, and visual grounding.

LLMs and VLMs are increasingly deployed as embodied agents, yet existing benchmarks largely revolve around simple short-term tasks and struggle to capture rich, realistic constraints that shape real-world decision-making. To close this gap, we propose DeliveryBench, a city-scale embodied benchmark that simulates the real-world profession of food delivery. Food couriers naturally operate under long-horizon objectives (maximizing net profit over hours) while managing diverse constraints, e.g. delivery deadline, transportation expense, vehicle battery, and necessary interactions with other couriers and customers. DeliveryBench instantiates this setting in procedurally generated 3D cities with diverse road networks, buildings, functional locations, transportation modes, and realistic resource dynamics, enabling systematic evaluation of constraint-aware, long-horizon planning. We benchmark a range of VLM-based agents across nine cities and compare them with human players. Our results reveal a substantial performance gap to humans, showing that these agents are short-sighted and frequently violate basic commonsense constraints. Additionally, we observe distinct personalities across models (e.g. adventurous GPT-5 vs. conservative Claude), highlighting both the brittleness and the diversity of current VLM-based embodied agents in realistic, constraint-dense environments. Our code, data, and benchmark are publicly available at https://deliverybench.github.io.