Jiaxin Pei

Large-scale public deliberations generate thousands of free-form contributions that must be synthesized into representative and neutral summaries for policy use. While LLMs have been shown as a promising tool to generate summaries for large-scale deliberations, they also risk underrepresenting minority perspectives, raising fairness concerns in high-stakes contexts. Studying and fixing these issues requires a comprehensive evaluation at a large scale, yet current practice often relies on LLMs as judges, which show weak alignment with human judgments. We introduce DeliberationBank, a large-scale, human-grounded benchmark for deliberation summarization that contains (1) 3,000 participant-generated opinions across ten deliberation questions and (2) 4,500 human annotations evaluating summaries along four dimensions: representativeness, informativeness, neutrality, and policy approval. Using this benchmark, we train DeliberationJudge, a domain-aligned evaluator that provides more reliable and efficient assessments than general-purpose LLM judges. With this evaluation framework, we benchmark 18 LLM summarizers and uncover consistent weaknesses, including systematic underrepresentation of minority viewpoints. Our benchmark and evaluator offer a scalable and reliable foundation for assessing deliberation summarization systems, supporting the development of more representative, equitable, and policy-relevant AI tools.

Resolving team conflicts requires not only task-specific competence, but also social intelligence to find common ground and build consensus. Similarly, as AI agents increasingly collaborate on complex work, they must develop coordination capabilities to function as effective teammates. We hypothesize that current agents lack these capabilities. To test it, we introduce CooperBench, a benchmark of 652 collaborative coding tasks spanning 12 libraries and 4 languages. Each task assigns agents independently implementable features that may conflict without coordination. Tasks are grounded in real open-source repositories with expert-written tests, which makes the cooperation outcomes fully verifiable. Evaluating SOTA coding agents, we observe the curse of coordination: at least 30\% average drop in success when agents work together versus a single agent completing both tasks, especially when the tasks are not extremely easy or hard. Critically, even on task pairs where each agent can individually solve its assigned feature, the best models still fail to coordinate roughly half the time, a conditional failure rate that cannot be explained by coding difficulty alone. We also find that human developer pairs solve 9 out of 10 tasks under the same coordination constraints that agents handle at only 3 out of 10, establishing that the overhead of splitting work is not the root cause. We identify three failure modes: (1) communication channels become jammed with vague, ill-timed, and inaccurate messages; (2) even with good communication, agents deviate from their commitments and hold incorrect expectations about others; and (3) agents have troubles using version control tools to perform real-time coordination. Beyond an open-source benchmark, this work contributes a novel understanding of what agents need to learn to become effective teammates.