Huanrui Yang

Unlearning ensures LLM compliance by removing the influence of private or copyrighted training data. However, since LLM models typically undergo post-training compression, like quantization, in practical deployment, it has been observed that the unlearning effect can be substantially weakened, with the forgetting behavior degrading more severely than that of model utility. This paper proposes a quantization-robust unlearning framework that makes forgetting robust to quantization while maintaining overall model utility. We analyze this gap through the lens of loss landscape. Specifically, our analysis reveals a curvature-based criteria that pinpoints sensitive weights in the unlearned model that leads to both non-robust forgetting and reduced utility. We therefore propose sensitivity-guided noisy regularization, which is applied on the sensitive parameters to steer the model convergence towards a smoother minima of uniformly low forget and retain losses. Balancing unlearning and utility, we further propose forget-critical optimization, which updates only forget-critical layers, preserving most of the network to retain useful knowledge. Extensive experiments on the MUSE and TOFU benchmarks across multiple LLM unlearning algorithms show that our approach achieves substantially more quantization-resilient forgetting while maintaining utility.

Large language models (LLMs) encounter significant challenges when adapting to diverse multitask fine-tuning settings. Mixture-of-experts (MoE) offers a promising solution with a dynamic architecture that enables effective task decoupling. However, scaling up the number of MoE experts incurs substantial parameter and computational overhead and yields limited performance gains due to naive routing mechanisms. In this paper, we design a novel framework, Mixture-of-rAnk-oNe-experTs (MoANT), which leverages the combination of ultra-low-rank experts to construct LoRA weights on pretrained LLMs. The rank-1 experts enable a mix-and-match mechanism that quadratically expands the vector subspace of experts with linear parameter overhead, achieving approximate error reduction with optimal efficiency. In addition, MoANT offers implicit guidance to the router by leveraging the inherent semantic clustering of training embeddings as prior knowledge, enabling optimized feature allocation across experts for smoother convergence. Extensive theoretical and empirical results demonstrate that MoANT achieves superior efficiency and generalizability across a diverse range of tasks. Compared with traditional LoRA, MoANT achieves up to 1.78% mean accuracy improvement with around 30%-40% fewer trainable parameters across 14 public datasets.