Xiangkun Hu

Recent progress in LLM agents has advanced the prospect of autonomous research. Yet whether AI can complete difficult long-horizon tasks, especially those that advance AI research itself, remains largely unexplored. We present ASI-Arch, a system for **AI-driven AI research** that autonomously conducts neural architecture research through a closed-loop *research-experiment-analyze-update* process. Applied to linear attention, ASI-Arch ran 1,773 iterative experiments and discovered 105 state-of-the-art architectures. Its best architecture improves over DeltaNet by nearly three times the gain achieved by Mamba2. Beyond the final performance gains, we analyze the contributions of different parts of the framework in this hard research setting, shedding light on what enables autonomous progress in complex AI research tasks.

Deep research agents have emerged with capabilities extending from basic queries to sophisticated research tasks. However, existing benchmarks primarily evaluate their abilities on web retrieval and report generation, overlooking the potential for discovering, integrating and generating insights in AI research. To address this gap, we introduce ResearcherBench, the first benchmark focused on evaluating the capabilities of Deep AI Research Systems (DARS) on open-ended AI research tasks. We curated a dataset of 65 questions expertly selected from real-world AI research scenarios, spanning 35 different AI subjects and categorized into 3 different types. Our dual evaluation framework consists of two components: insight assessment which implements expert-designed rubrics to evaluate insight quality, and citation assessment measuring citation faithfulness and groundedness (citation density). We evaluated several leading commercial DARS and baseline systems. Our evaluation results reveal that these systems excel at open-ended consulting questions compared to technical implementation questions, demonstrating the potential for DARS to serve as AI research partners. We open-source ResearcherBench and evaluation framework to accelerate the development of AI research assistants capable of scientific collaboration.