Xiaokun Chen

Recent advances in prompt learning allow large language model agents to acquire task-relevant knowledge from inference-time context without parameter changes. For example, existing methods (like ACE or GEPA) can learn system prompts to improve accuracy based on previous agent runs. However, these methods primarily focus on single-agent or low-parallelism settings. This fundamentally limits their ability to efficiently learn from a large set of collected agentic traces. It would be efficient and beneficial to run prompt learning in parallel to accommodate the growing trend of learning from many agentic traces or parallel agent executions. Yet without a principled strategy for scaling, current methods suffer from quality degradation with high parallelism. To improve the quality delay tradeoff of prompt learning, we propose Combee, a plug-and-play framework that scales parallel prompt learning on top of existing methods such as ACE and GEPA. Combee speeds up learning and enables running many agents in parallel while preserving most of the quality of sequential learning from their aggregate traces. To achieve this, Combee leverages parallel scans and employs an augmented shuffle mechanism; Combee also introduces a dynamic batch size controller to balance quality and delay. Evaluations on AppWorld, Terminal-Bench, Formula, and FiNER demonstrate that Combee substantially improves the quality delay tradeoff of prompt learning, achieving up to 17× speedup over previous methods at equivalent cost while matching or approaching their best accuracy.

Large language model (LLM) agents are increasingly deployed to automate productivity tasks (e.g., email, scheduling, document management), but evaluating them on live services is risky due to potentially irreversible changes. Existing benchmarks rely on simplified environments and fail to capture realistic, stateful, multi-service workflows. We introduce ClawsBench, a benchmark for evaluating and improving LLM agents in realistic productivity settings. It includes five high-fidelity mock services (Gmail, Slack, Calendar, Docs, Drive) with full state management and deterministic snapshot/restore, along with 44 structured tasks covering single-service, cross-service, and safety-critical scenarios. We study two independently varied scaffolding factors: domain skills that inject API knowledge via progressive disclosure and a meta prompt that coordinates behavior across services. Experiments across 6 models, 4 agent harnesses, and 33 conditions show that with full scaffolding, agents achieve task success rates of 39–64% but exhibit unsafe action rates of 7–33%. On OpenClaw, the top five models fall within a 10 percentage-point band on task success (53–63%), with unsafe action rates from 7% to 23% and no consistent ordering between the two metrics. Trajectory analysis identifies eight behavioral patterns spanning policy violations and process failures, including sandbox escalation, unauthorized contract modification, over-refusal, and hallucination. Code is available at https://github.com/benchflow-ai/ClawsBench.