作者位次 #2/7ACL 2026Findings· Findings of ACL
HTM uon: Improving Muon via Heavy-Tailed Spectral Correction
Tianyu Pang · Yujie Fang · Zihang Liu · Shenyang Deng · Lei Hsiung · Shuhua Yu · Yaoqing Yang
摘要
Muon has recently shown promising results in LLM training. In this work, we study how to further improve Muon. We argue that Muon’s orthogonalized update rule suppresses the emergence of heavy-tailed weight spectra and over-emphasizes the training along noise-dominated directions. Motivated by th...