Micah Goldblum

Recent advances in depth-recurrent language models show that recurrence can decouple train-time compute and parameter count from test-time compute. In this work, we study how to retrofit existing pretrained non-recurrent language models into depth-recurrent models. We find that using a curriculum of recurrences to increase the effective depth of the model over the course of training preserves performance while reducing total computational cost. In our experiments, on mathematics, we observe that retrofitting pretrained models to recurrent ones results in better performance at a given compute budget than simply post-training the original non-recurrent language model.

In high-stakes decision-making applications of large language models (LLMs), practitioners require not only accurate LLMs but also uncertainty estimates for their predictions. Existing approaches to uncertainty estimation for LLMs require access to log-probabilities output by the model or require fine-tuning access. However, the vast majority of industrial LLM products use closed-source API models, and many such API models like GPT-5 do not return log-probabilities and may not allow fine-tuning. We introduce PINOCCHIO, a suite of 8B checkpoints for assigning uncertainty estimates to predictions made by popular API models. To this end, we curate a large collection of hard datasets, ranging from long-context to vision-language, that expose conditions under which frontier models output incorrect responses. We then train PINOCCHIO to recognize the patterns in prompts and model responses that indicate confidence or uncertainty. PINOCCHIO checkpoints significantly outperform existing baselines for black-box uncertainty estimation in calibration metrics and transfer to unseen model families, including OpenAI, Qwen, Meta, Anthropic, Google, and Moonshot, whose responses they were not trained on. We release code for adding uncertainty estimation to existing repos in only two additional lines of code.

A slew of recent works develop agents for solving research problems end-to-end, a paradigm increasingly referred to as autoresearch. Such agents have inspired large industry investment, motivated by their potential to automate time-consuming human labor and customize machine learning solutions for specialized applications. In this paper, we study the modeling pipeline at the core of these autoresearch systems and identify common failure modes when they are applied to tabular datasets: (1) they waste compute resolving the same bugs over and over again; (2) they often fail to tune hyperparameters even when they have a large remaining compute budget; (3) the tree-search algorithms that power them do not explore; and (4) they perform data analysis, mimicking the humans whose data they are trained on, but do not use that analysis to make downstream decisions. We explore targeted interventions and find that a global debug consultant that shares discovered runtime constraints across all branches of the search tree, prompt- and control-level enhancements, and refined tree-search algorithms successfully recover wasted compute. Our results show that large gains in autoresearch agent performance are achievable through agentic design alone, holding the underlying language model fixed.

Mathematical reasoning is widely regarded as a hallmark of human intelligence, and determining whether large language models (LLMs) can meaningfully perform it remains a central question in artificial intelligence and cognitive science. As LLMs are increasingly integrated into scientific workflows, rigorous evaluation of their mathematical capabilities becomes a practical necessity. Existing mathematical reasoning benchmarks, however, are often limited by synthetic problem settings and growing contamination from widely circulated datasets. We present LiveMathematicianBench, a dynamic multiple-choice benchmark for evaluating research-level mathematical reasoning using recent arXiv papers published after model training cutoffs. By grounding evaluation in newly published theorem statements, the benchmark provides a more realistic testbed for assessing whether models can reason about natural mathematical claims beyond memorized benchmark patterns. LiveMathematicianBench introduces a taxonomy of thirteen problem categories based on the logical form of theorem statements, enabling fine-grained evaluation across reasoning types such as implication, equivalence, existence, and uniqueness. It further introduces a proof-sketch-guided distractor generation pipeline, in which proof sketches are used to construct plausible but invalid answer choices that reflect misleading proof directions. This design makes the benchmark more sensitive to genuine mathematical understanding rather than surface-level answer matching. We additionally introduce a substitution-resistant evaluation mechanism designed to distinguish answer recognition from substantive mathematical reasoning. Evaluation of frontier models shows that the benchmark is far from saturated: the best-performing model, Gemini-3.1-pro-preview, achieves only 43.5\% accuracy in the standard setting. Under substitution-resistant evaluation, accuracy drops sharply across all models: GPT-5.4 achieves the highest score at 30.6\%, while Gemini-3.1-pro-preview falls to 17.6\%, below the 20\% random baseline. A dual-mode protocol comparing performance with and without proof-sketch access reveals that sketches yield consistent gains (e.g., +13.6 pp for Gemini-3.1-pro-preview), suggesting that models can leverage high-level proof strategies to improve reasoning. Overall, LiveMathematicianBench offers a scalable and contamination-resistant testbed for studying research-level mathematical reasoning in large language models.