Sanmi Koyejo

Benchmarks play a central role in the development and governance of models, yet it is often unclear whether they actually measure the concepts they purport to measure (e.g., reasoning, refusal). We adapt the lenses of convergent and discriminant validity from the social sciences into an approach for interrogating AI benchmarks, which we use to analyze 56 capability and safety benchmarks using 53 models. We assign benchmarks with substantively similar \textit{purported concepts} to a shared \textit{assigned concept}, and ask whether model rankings on benchmarks with the same assigned concept correlate strongly with one another, and whether model rankings on benchmarks with different assigned concepts correlate less strongly. We ask analogous questions using model scores at the item-level, drawing on item response theory (IRT) models. We find that correlations between model rankings on benchmarks with the same assigned safety concepts are often weak, suggesting that these assigned concepts may be conceptualized inconsistently across benchmarks. For benchmarks with assigned capability concepts (e.g., reasoning, knowledge), model rankings are often as strongly correlated among benchmarks with the same assigned concept as between benchmarks with different assigned concepts, suggesting that different assigned capability concepts may not discriminate that well from one another. In some cases, benchmarks that share benchmark design elements (e.g., task structure, score format) correlate more strongly with one another than benchmarks with the same assigned concept. Finally, model rankings on some individual benchmarks correlate more strongly with model rankings on benchmarks with a different assigned concept than with model rankings on benchmarks sharing their own assigned concept, suggesting that these benchmarks may measure a different concept than they purport to. For example, model rankings on BBQ-accuracy correlate more strongly with model rankings on benchmarks assigned with reasoning than with model rankings on benchmarks that share its assigned concept, bias. To support future empirical work on the validity of benchmarks, we release our extensive dataset of model outputs and scores at the item- and benchmark-level.

LLM fairness should be evaluated through in-situ behavioral pattern rather than standardized-test Q\&A benchmarks. We show that the standardized-test paradigm can be structurally unreliable: surface-level prompt construction choices, although entirely orthogonal to the fairness question being tested, account for the majority of score variance, shift fairness conclusions in both the direction and the magnitude, and result in severe discordance in model rankings. We develop MAC-Fairness, a framework that embeds controlled variation factors into in-situ behavioral evaluation, examining how models' disparate-treatment behaviors shift when identity is varied as part of natural multi-agent conversation. Repurposing standardized-test questions as conversation seeds rather than as the evaluation instrument, we evaluate within-model variance in position persistence (how they hold positions, from the self-perspective) and peer receptiveness (how receptive they are to peers, from the other-perspective) across 8 million conversation transcripts spanning multiple models and identity presence configurations. In-situ behavioral evaluation reveals stable, model-specific, disparate-treatment behavioral signatures that could generalize across different fairness benchmarks, a form of evidence the standardized-test paradigm does not offer.

AI coding agents are being adopted at scale, yet we lack empirical evidence on how people actually use them and how much of their output is useful in practice. We present SWE-chat, the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild. The dataset currently contains 13,000 sessions, comprising more than 138,000 user prompts and 893,000 agent tool calls. SWE-chat is a living dataset; our collection pipeline automatically and continually discovers and processes sessions from public repositories. Leveraging SWE-chat, we provide an initial empirical characterization of real-world coding agent usage and failure modes. We find that coding patterns are bimodal: in 37% of sessions, agents author virtually all committed code ("vibe coding"), while in 26%, humans write all code themselves. Despite rapidly improving capabilities, coding agents remain inefficient in natural settings. Only 56% of all agent-produced code survives into user commits, and agent-written code introduces more security vulnerabilities than code authored by humans. Furthermore, users push back against agent outputs – through corrections, failure reports, and interruptions – in 49% of all turns. By capturing complete interaction traces with human vs. agent code authorship attribution, SWE-chat provides an empirical foundation for moving beyond curated benchmarks towards an evidence-based understanding of how AI agents perform in real developer workflows.

Large language models exhibit safety degradation in non-English languages. Standard evaluation relies on Jailbreak Success Rate (JSR), which confounds several safety-driving factors into one, obscuring the specific cause(s) of safety failure. We introduce a latent variable model, a Multi-Group Item Response Theory (IRT) framework, that decouples language-agnostic safety robustness ($\theta$), intrinsic prompt hardness ($\beta$), global language processing difficulty ($\gamma$), and a prompt-specific cross-lingual safety gap ($\tau$). Using the MultiJail dataset, we evaluate the safety robustness of 61 model configurations across 5 closed-model families and 10 languages of varying resource, aggregating a dataset of 1.9 million responses. Exploratory Factor Analysis shows safety is primarily unidimensional: models refuse different harm types mainly through a shared mechanism. Contrary to the expected trend that safety degrades largely in low-resource languages, 22 model configurations are more vulnerable in English than in low-resource languages. Low-resource languages produce more uncertain responses (high entropy) than high-resource languages. Also, high-$\tau$ prompts cluster in physical harm categories like Theft and Weapons and lower-resource languages, trends validated through cross-dataset generalization. While global translation quality shows low correlation with $\tau$, severe mistranslations drive high-bias outliers, as validated by native speakers. Cultural and conceptual grounding mismatches may also contribute to $\tau$. In predictive validation, the IRT framework achieves $\mathrm{AUC} = 0.940$, and unlike rate baselines stays predictive when a whole language is held out ($0.875$). Our framework reveals concept-language vulnerabilities that aggregate metrics obscure, enabling fairer cross-lingual safety evaluation and targeted improvements in dataset construction.