ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces
Xiangyi Li · Kyoung Whan Choe · Yimin Liu · Xiaokun Chen · Chujun Tao · Bingran You · Wenbo Chen · Zonglin Di · Jiankai Sun · Shenghan Zheng · Jiajun Bao · Yuanli Wang · Weixiang Yan · Yiyuan Li · Hanchung Lee
ClawsBench evaluates LLM productivity agents across five realistic, stateful mock services, revealing that stronger task performance does not consistently correspond to safer behavior.
摘要
Large language model (LLM) agents are increasingly deployed to automate productivity tasks (e.g., email, scheduling, document management), but evaluating them on live services is risky due to potentially irreversible changes. Existing benchmarks rely on simplified environments and fail to capture realistic, stateful, multi-service workflows. We introduce ClawsBench, a benchmark for evaluating and improving LLM agents in realistic productivity settings. It includes five high-fidelity mock services (Gmail, Slack, Calendar, Docs, Drive) with full state management and deterministic snapshot/restore, along with 44 structured tasks covering single-service, cross-service, and safety-critical scenarios. We study two independently varied scaffolding factors: domain skills that inject API knowledge via progressive disclosure and a meta prompt that coordinates behavior across services. Experiments across 6 models, 4 agent harnesses, and 33 conditions show that with full scaffolding, agents achieve task success rates of 39–64% but exhibit unsafe action rates of 7–33%. On OpenClaw, the top five models fall within a 10 percentage-point band on task success (53–63%), with unsafe action rates from 7% to 23% and no consistent ordering between the two metrics. Trajectory analysis identifies eight behavioral patterns spanning policy violations and process failures, including sandbox escalation, unauthorized contract modification, over-refusal, and hallucination. Code is available at https://github.com/benchflow-ai/ClawsBench.