Vishakh Padmakumar

When posed with prompts that permit a large number of valid answers, comprehensively generating them is the first step towards satisfying a wide range of users. In this paper, we study methods to elicit a comprehensive set of valid responses. To evaluate this, we introduce **diversity coverage**, a metric that measures the total quality scores assigned to each **unique** answer in the predicted answer set relative to the best possible answer set with the same number of answers. Using this metric, we evaluate $18$ LLMs, finding no single model dominates at generating diverse responses to a wide range of open-ended prompts. Yet, per each prompt, there exists a model that outperforms all other models significantly at generating a diverse answer set. Motivated by this finding, we introduce a router that predicts the best model for each query. On *NB-WildChat*, our trained router outperforms the single best model baseline ($26.3$% vs $23.8$%). We further show generalization to an out-of-domain dataset (*NB-Curated*) as well as different answer-generation prompting strategies. Our work lays foundation for studying generating comprehensive answers when we have access to a suite of models.

AI coding agents are being adopted at scale, yet we lack empirical evidence on how people actually use them and how much of their output is useful in practice. We present SWE-chat, the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild. The dataset currently contains 13,000 sessions, comprising more than 138,000 user prompts and 893,000 agent tool calls. SWE-chat is a living dataset; our collection pipeline automatically and continually discovers and processes sessions from public repositories. Leveraging SWE-chat, we provide an initial empirical characterization of real-world coding agent usage and failure modes. We find that coding patterns are bimodal: in 37% of sessions, agents author virtually all committed code ("vibe coding"), while in 26%, humans write all code themselves. Despite rapidly improving capabilities, coding agents remain inefficient in natural settings. Only 56% of all agent-produced code survives into user commits, and agent-written code introduces more security vulnerabilities than code authored by humans. Furthermore, users push back against agent outputs – through corrections, failure reports, and interruptions – in 49% of all turns. By capturing complete interaction traces with human vs. agent code authorship attribution, SWE-chat provides an empirical foundation for moving beyond curated benchmarks towards an evidence-based understanding of how AI agents perform in real developer workflows.