Ameet Talwalkar

LLMs are increasingly used as judges in code applications, yet their efficacy remains untested in realistic code settings that often feature partial context and ambiguous intent. We present $\textbf{TRACE}$ ($\textbf{T}$ool for $\textbf{R}$ubric $\textbf{A}$nalysis in $\textbf{C}$ode $\textbf{E}$valuation), a framework that evaluates LLM judges’ ability to predict human preferences and automatically extracts rubric items to highlight systematic differences in how humans and models weight these criteria. Across three modalities---IDE autocompletion, chat-based programming, and instructed code editing---we benchmark 13 judge models spanning general-purpose LLMs, specialized judge models, and reward models. Among these models, even the best judges align with human preference only $\textbf{6-18}$% above random chance. $\textbf{TRACE}$ identifies 8 significantly misaligned rubric items across interaction modalities. For example, in chat-based coding, judges prefer longer code explanations while humans prefer shorter ones. Many of these gaps occur within existing code quality dimensions, showing that current LLM judges remain misaligned with human evaluation in realistic coding workflows.

Scaling laws are used to plan multi-million-dollar training runs, but fitting those laws can itself cost millions. In modern large-scale workflows, assembling a sufficiently informative set of pilot experiments is already a major budget-allocation problem rather than a routine preprocessing step. We formulate scaling-law fitting as budget-aware sequential experimental design: given a finite pool of runnable experiments with heterogeneous costs, choose which runs to execute so as to maximize extrapolation accuracy in a high-cost target region. We propose $\mathrm{SL}^2$ (Scaling Laws, Spend Less), an uncertainty-aware method for sequentially allocating experimental budget toward the runs most useful for target-region extrapolation. Across a diverse benchmark of scaling-law tasks, $\mathrm{SL}^2$ outperforms classical design-based baselines, and often approaches the performance of fitting on the full experimental set while using only about 10\% of the total training budget. Our code is available at https://github.com/PlanarG/active-sl.

Current evaluations of agents remain centered around one-shot task completion, failing to account for the inherently iterative and collaborative nature of many real-world problems, where human goals are often underspecified and evolve. We argue for a shift from building and assessing task complet...