Neural Architecture Discovery via Autonomous Evolution
Weixian Xu · Yixiu Liu · Yang Nan · Lyumanshan Ye · Xiangkun Hu · Zhen Qin · Pengfei Liu
We present ASI-Arch to study whether AI can autonomously tackle long-horizon research problems in AI itself, and show that it discovers 105 SOTA linear attention architectures, with the best achieving nearly 3x the gain of Mamba2 over DeltaNet.
摘要
Recent progress in LLM agents has advanced the prospect of autonomous research. Yet whether AI can complete difficult long-horizon tasks, especially those that advance AI research itself, remains largely unexplored. We present ASI-Arch, a system for **AI-driven AI research** that autonomously conducts neural architecture research through a closed-loop *research-experiment-analyze-update* process. Applied to linear attention, ASI-Arch ran 1,773 iterative experiments and discovered 105 state-of-the-art architectures. Its best architecture improves over DeltaNet by nearly three times the gain achieved by Mamba2. Beyond the final performance gains, we analyze the contributions of different parts of the framework in this hard research setting, shedding light on what enables autonomous progress in complex AI research tasks.