Julian McAuley

Large Language Models (LLMs) exhibit strong multi-step reasoning, but their inference-time performance depends critically on the quality of available reasoning traces. In many realistic settings, such traces are generated by multiple clients holding private and heterogeneous data, motivating the problem of federated reasoning, where a server aggregates client-generated reasoning without centralized training or data sharing. A key challenge is that client reasoning quality is highly query-dependent, and naive aggregation can amplify miscalibrated confidence and correlated reasoning errors. We propose Uncertainty-Aware Federated Reasoning (FERA), a training-free framework that aggregates client reasoning using query-dependent trust weights derived from uncertainty signals, rather than treating all client outputs equally. We instantiate FERA with Uncertainty-Aware Self-Critique Aggregation (UA-SCA), which leverages structured self-critique and cross-client verification to identify unsupported reasoning steps and downweight high-confidence but weakly justified traces. We provide theoretical guarantees on the convergence of the resulting trust-weighted aggregation and demonstrate on multiple reasoning benchmarks that FERA consistently improves accuracy over standard ensembling baselines, while requiring minimal communication and no model updates.

LLM agents now perform strongly in software engineering, deep research, GUI automation, and various other applications, while recent agent scaffolds and models are increasingly integrating these capabilities into unified systems. Yet, most evaluations still test these capabilities in isolation, which leaves a gap for more diverse use cases that require agents to combine different capabilities. We introduce CocoaBench, a benchmark for unified digital agents built from human designed, long horizon tasks that require flexible composition of vision, search, and coding. Tasks are specified only by an instruction and an automatic evaluation function over the final output, enabling reliable and scalable evaluation across diverse agent infrastructures. We also present CocoaAgent, a lightweight shared scaffold for controlled comparison across model backbones. Experiments show that current agents remain far from reliable on CocoaBench, with the best evaluated system achieving only 45.1% success rate. Our analysis further points to substantial room for improvement in reasoning and planning, tool use and execution, and visual grounding.