Tongshuang Wu

As NLP evaluation shifts from static benchmarks to multi-turn interactive settings, LLM-based simulators have become widely used as user proxies, serving two roles: generating user turns and providing evaluation signals. Yet, these simulations are frequently assumed to be faithful to real human behaviors, often without rigorous verification. We formalize the Sim2Real gap in user simulation and present the systematic study running the full $\tau$-bench protocol with real humans (451 participants, 165 tasks), benchmarking 31 LLM simulators across proprietary, open-source, and specialized families using the User-Sim Index (USI), a metric we introduce to quantify how well LLM simulators resemble real user interactive behaviors and feedback. Behaviorally, LLM simulators are excessively cooperative, stylistically uniform, and lack realistic frustration or ambiguity, creating an "easy mode" that inflates agent success rates above the human baseline. In evaluations, real humans provide nuanced judgments across eight quality dimensions while simulated users produce uniformly more positive feedback; rule-based rewards are failing to capture rich feedback signals generated by human users. Overall, higher general model capability does not necessarily yield more faithful user simulation. These findings highlight the importance of human validation when using LLM-based user simulators in the agent development cycle and motivate improved models for user simulation.

As large language models (LLMs) increasingly shape how users form, refine, and extend their goals, attributing contributions in human–AI collaboration becomes critical—both for users calibrating their own reliance and for evaluators assessing AI-assisted work. Yet existing methods focus on final artifacts, missing the process through which goals themselves are jointly shaped. We introduce a goal-level attribution framework, COTRACE, that decomposes explicit goals into verifiable requirements and traces both direct contributions and indirect influences across dialogue turns. Applying COTRACE to 638 real-world collaboration logs, we find that while models account for only ~10% of goal-shaping at the macro level, they contribute substantially more on introducing lower-level concrete requirements, and make various kinds of indirect contributions. Through controlled simulations, we show that interaction design choices significantly affect model goal-shaping behavior, with downstream effects on output diversity. In a user study, exposing participants to goal-level analyses shifts their perceived contributions by nearly 2 points on a 5-point scale, revealing systematic miscalibration in how users understand their own AI-assisted work.

LLMs are increasingly used as judges in code applications, yet their efficacy remains untested in realistic code settings that often feature partial context and ambiguous intent. We present $\textbf{TRACE}$ ($\textbf{T}$ool for $\textbf{R}$ubric $\textbf{A}$nalysis in $\textbf{C}$ode $\textbf{E}$valuation), a framework that evaluates LLM judges’ ability to predict human preferences and automatically extracts rubric items to highlight systematic differences in how humans and models weight these criteria. Across three modalities---IDE autocompletion, chat-based programming, and instructed code editing---we benchmark 13 judge models spanning general-purpose LLMs, specialized judge models, and reward models. Among these models, even the best judges align with human preference only $\textbf{6-18}$% above random chance. $\textbf{TRACE}$ identifies 8 significantly misaligned rubric items across interaction modalities. For example, in chat-based coding, judges prefer longer code explanations while humans prefer shorter ones. Many of these gaps occur within existing code quality dimensions, showing that current LLM judges remain misaligned with human evaluation in realistic coding workflows.

Understanding how developers interact with AI coding assistants requires more than chat logs or git histories in isolation; it requires reconstructing the full context: which prompt led to which edit, what the developer tried and discarded, and how their strategy evolved over time. We present REC...

Prompt underspecification is a common challenge when interacting with LLMs. In this paper, we present an in-depth analysis of this problem, showing that while LLMs can often infer unspecified requirements by default (41.1%), such behavior is fragile: Under-specified prompts are 2x as likely to re...

Current evaluations of agents remain centered around one-shot task completion, failing to account for the inherently iterative and collaborative nature of many real-world problems, where human goals are often underspecified and evolve. We argue for a shift from building and assessing task complet...