paper.dinq.meAI 与前沿科学顶会 · 学术新星雷达
登录 DINQ
返回新星榜
SZ

Shanghang Zhang

AwardCVPR 2026×7ICRA 2026×7ICLR 2026×6AAAI 2026×3COLM 2026多模态效率与系统推理强化学习与后训练社会与人本
OROpenReview
新星分资深
50
一作贡献0
核心/参与65
方向广度19
合作网络12
0 一作 · 24 篇 · CVPR 2026 / ICRA 2026 / ICLR 2026 / AAAI 2026 / COLM 2026

获取完整研究档案

首次获取该候选人的完整档案及 StartLux 评估共 20 积分；每位用户单独解锁，之后查看免费。主动刷新 20 积分。

联系方式

实时查找电子邮箱，费用不超过20积分；未找到邮箱不扣费。

2026 顶会论文

24
作者位次 #8/8COLM 2026Poster· Poster Session 6· Franciscan B· #84

MoANT: Mixture-of-Rank-One-Experts with Semantic-aware Intuition for Multi-task Large Language Model Finetuning

Rongyu Zhang · Yijiang Liu · Huanrui Yang · Shenli Zheng · Li Du · Dan Wang · Yuan Du · Shanghang Zhang

MoANT is a simple yet efficient PEFT method that mixes rank‑1 “experts” with a lightweight, semantics‑aware router to boost multitask LLM performance while keeping training and inference costs low.

摘要

Large language models (LLMs) encounter significant challenges when adapting to diverse multitask fine-tuning settings. Mixture-of-experts (MoE) offers a promising solution with a dynamic architecture that enables effective task decoupling. However, scaling up the number of MoE experts incurs substantial parameter and computational overhead and yields limited performance gains due to naive routing mechanisms. In this paper, we design a novel framework, Mixture-of-rAnk-oNe-experTs (MoANT), which leverages the combination of ultra-low-rank experts to construct LoRA weights on pretrained LLMs. The rank-1 experts enable a mix-and-match mechanism that quadratically expands the vector subspace of experts with linear parameter overhead, achieving approximate error reduction with optimal efficiency. In addition, MoANT offers implicit guidance to the router by leveraging the inherent semantic clustering of training embeddings as prior knowledge, enabling optimized feature allocation across experts for smoother convergence. Extensive theoretical and empirical results demonstrate that MoANT achieves superior efficiency and generalizability across a diverse range of tasks. Compared with traditional LoRA, MoANT achieves up to 1.78% mean accuracy improvement with around 30%-40% fewer trainable parameters across 14 public datasets.

效率与系统数据与预训练理论与学习动力学Mixture-of-ExpertFine-tuningMulti-taskOpenReview

常见合作者

PWPengwei Wang×7ZWZhongyuan Wang×6CCCheng Chi×6MLMing Lu×5JLJiaming Liu×4RZRongyu Zhang×3CHChengkai Hou×3GDGaole Dai×3HCHao Chen×3ZLZhuoyang Liu×3RZRenrui Zhang×3CGChenyang Gu×3