Lyumanshan Ye

Recent progress in LLM agents has advanced the prospect of autonomous research. Yet whether AI can complete difficult long-horizon tasks, especially those that advance AI research itself, remains largely unexplored. We present ASI-Arch, a system for **AI-driven AI research** that autonomously conducts neural architecture research through a closed-loop *research-experiment-analyze-update* process. Applied to linear attention, ASI-Arch ran 1,773 iterative experiments and discovered 105 state-of-the-art architectures. Its best architecture improves over DeltaNet by nearly three times the gain achieved by Mamba2. Beyond the final performance gains, we analyze the contributions of different parts of the framework in this hard research setting, shedding light on what enables autonomous progress in complex AI research tasks.

Rubric-based Reinforcement Learning (RL) has emerged as a promising approach for aligning Large Language Models (LLMs) with complex, open-domain instruction following tasks. However, existing methods predominantly rely on response-level rewards, introducing severe reward sparsity and reward ambiguity problems. To address these issues, we propose Rubrics to Tokens (RTT), a novel rubric-based RL framework that bridges coarse response-level scores and fine-grained token-level credit assignment. RTT introduces a Token-Level Relevance Discriminator to predict which tokens in the response are responsible for a specific constraint, and optimizes the policy model via RTT-GRPO, which integrates response-level and token-level advantages within a unified framework. Furthermore, when transitioning from one-dimensional, outcome-level reward to three-dimensional reward space in the token-level rubric-based RL, we propose a novel group normalization method, called Intra-sample Token Group Normalization, to accommodate this shift. Extensive experiments and benchmarks demonstrate that RTT consistently outperforms other baselines in both instruction- and rubric-level accuracy across different models. We open-source our RTT framework to facilitate future research.

Deep research agents have emerged with capabilities extending from basic queries to sophisticated research tasks. However, existing benchmarks primarily evaluate their abilities on web retrieval and report generation, overlooking the potential for discovering, integrating and generating insights in AI research. To address this gap, we introduce ResearcherBench, the first benchmark focused on evaluating the capabilities of Deep AI Research Systems (DARS) on open-ended AI research tasks. We curated a dataset of 65 questions expertly selected from real-world AI research scenarios, spanning 35 different AI subjects and categorized into 3 different types. Our dual evaluation framework consists of two components: insight assessment which implements expert-designed rubrics to evaluate insight quality, and citation assessment measuring citation faithfulness and groundedness (citation density). We evaluated several leading commercial DARS and baseline systems. Our evaluation results reveal that these systems excel at open-ended consulting questions compared to technical implementation questions, demonstrating the potential for DARS to serve as AI research partners. We open-source ResearcherBench and evaluation framework to accelerate the development of AI research assistants capable of scientific collaboration.

AI agents are being deployed commercially faster than evaluation methods have adapted to production realities. Existing benchmarks commonly use retrospectively curated tasks with explicit requirements and deterministic metrics, whereas production work contains implicit constraints, heterogeneous documents, fragmented information, undeclared domain knowledge, and expert-defined success criteria. We present AlphaEval, a production-grounded benchmark of 94 tasks sourced from seven companies across six O*NET occupational domains. Unlike model-centric benchmarks, AlphaEval evaluates complete agent products—Claude Code, Cursor, Codex, and GitHub Copilot—across 14 scaffold–model configurations. Its evaluation combines reference metrics, formal verification, fixed expert-authored rubrics applied by an LLM judge, and automated UI testing. Beyond the benchmark, we contribute a two-layer requirement-to-benchmark framework: a structured, human-led playbook for eliciting and validating authentic requirements with partner experts, followed by a technical pipeline that turns a specified task package into an automated, reproducible evaluation. The best configuration scores 64.41/100, with large variation across domains and scaffolds.