Valerie Chen

LLMs are increasingly used as judges in code applications, yet their efficacy remains untested in realistic code settings that often feature partial context and ambiguous intent. We present $\textbf{TRACE}$ ($\textbf{T}$ool for $\textbf{R}$ubric $\textbf{A}$nalysis in $\textbf{C}$ode $\textbf{E}$valuation), a framework that evaluates LLM judges’ ability to predict human preferences and automatically extracts rubric items to highlight systematic differences in how humans and models weight these criteria. Across three modalities---IDE autocompletion, chat-based programming, and instructed code editing---we benchmark 13 judge models spanning general-purpose LLMs, specialized judge models, and reward models. Among these models, even the best judges align with human preference only $\textbf{6-18}$% above random chance. $\textbf{TRACE}$ identifies 8 significantly misaligned rubric items across interaction modalities. For example, in chat-based coding, judges prefer longer code explanations while humans prefer shorter ones. Many of these gaps occur within existing code quality dimensions, showing that current LLM judges remain misaligned with human evaluation in realistic coding workflows.

Academic benchmarks for coding agents tend to reward autonomous task completion, measured by verifiable rewards such as unit-test success. In contrast, real-world coding agents operate with humans in the loop, where success signals are typically noisy, delayed, and sparse. How can we bridge this gap? In this paper, we propose a process to learn a ``critic'' model from sparse and noisy interaction data, which can then be used for inference-time scaling and training-time data curation. Specifically, we introduce Critic Rubrics, a rubric-based supervision framework with 24 behavioral features that can be derived from human-agent interaction traces alone. Using a semi-supervised objective, we can then jointly predict these rubrics and sparse human feedback (when present). In experiments, we demonstrate that, despite being trained primarily from trace-observable rubrics and sparse real-world outcome proxies, these critics improve best-of-N reranking on SWE-bench (Best@8 +15.9 over Random@8 over the rerankable subset of trajectories), enable early stopping (+17.7 with 83\% fewer attempts), and support training-time data curation of trajectories.

Understanding how developers interact with AI coding assistants requires more than chat logs or git histories in isolation; it requires reconstructing the full context: which prompt led to which edit, what the developer tried and discarded, and how their strategy evolved over time. We present REC...