Qifan Wang

Low-rank adapters (LoRAs) are a parameter-efficient finetuning technique that injects trainable low-rank matrices into pretrained models to adapt them to new tasks. Mixture-of-LoRAs models expand neural networks efficiently by routing each layer input to a small subset of specialized LoRAs of the layer. Existing Mixture-of-LoRAs routers assign a learned routing weight to each LoRA to enable end-to-end training of the router. Despite their empirical promise, we discover, both theoretically and empirically, that the routing weights often collapse to only one LoRA even when we activate $k>1$ LoRAs during finetuning. When one LoRA has a dominantly large weight, then the computation of the other $k-1$ LoRAs are essentially **wasted** because using $k>1$ would have similar accuracy to $k=1$. This essentially limits the number of effective LoRAs and thus severely hinders the realized expressive power of existing Mixture-of-LoRAs models. How can we address this critical weakness? In this work, we attribute this weakness to the nature of learnable routing weights and rethink the fundamental design of the router. To address this critical issue, we propose a simple yet effective router design that we call *Reinforcement Routing for Mixtures of LoRAs* (ReMix). Our key idea is using **non-learnable** routing weights to ensure all active LoRAs to be equally effective, with no single LoRA dominating the routing weights. However, such non-learnable routing weights make it infeasible to directly train routers via gradient descent. In response, we further propose an unbiased gradient estimator for the router and employ the reinforce leave-one-out (RLOO) technique to reduce the variance of the estimator. Our gradient estimator also enables to scale up training compute to boost the predictive performance of our ReMix. Extensive experiments demonstrate that our proposed ReMix significantly outperforms state-of-the-art parameter-efficient finetuning methods under a small number of activated parameters.

Reinforcement learning from human feedback (RLHF) is vulnerable to reward hacking: policies can exploit spurious correlations in learned reward models to increase proxy reward while degrading true alignment. Existing defenses are largely static. They regularize optimization, improve the reward model, or target known biases, but do not explicitly detect exploitation as it emerges during training. We propose \textbf{Adversarial Reward Auditing (ARA)}, a two-stage framework for detecting and mitigating reward hacking. In Stage 1, a Hacker policy searches for vulnerabilities in a frozen reward model while an Auditor learns to detect exploitative responses from the reward model's internal representations. In Stage 2, the trained Auditor guides RLHF by downweighting rewards for responses likely to be exploitative. Across sycophancy, length bias, and code gaming, ARA yields the strongest alignment--utility tradeoff among competitive baselines, reducing exploitation while preserving or improving task performance. We also find cross-domain transfer in both attack and defense: a Hacker trained on code gaming increases sycophancy despite receiving no direct reward for that behavior, while an Auditor trained in one domain helps suppress exploits in others. These results suggest that reward hacking is a dynamic, transferable phenomenon and that auditing optimization in reward-model representation space is an effective way to control it.

Distilling reasoning capabilities from strong to weak language models typically involves imitating specific solution trajectories, effectively transferring what to answer rather than how to reason. This trajectory-level imitation encourages memorization of instance-specific steps rather than acquisition of transferable problem-solving skills, limiting generalization to novel problems. We propose Strategy-Guided Policy Optimization (SGPO), which replaces instance-level trajectory imitation with reusable strategy distillation. SGPO extracts structured strategy descriptions from strong-model responses and, for each problem, constructs both autonomous and strategy-guided trajectories to enable direct comparison of the model's behavior with and without strategic guidance. The framework then addresses two key questions. For how to distill, a token-level forward-KL objective selectively transfers the distributional shift induced by strategy conditioning into the unguided policy, with proximal constraints ensuring stability. For when to distill, adaptive instance-level weighting strengthens guidance when autonomous exploration falls short and reduces it as the model's own competence grows. Experiments on four mathematical benchmarks across two model families show that SGPO consistently outperforms SFT, on-policy RL, and hybrid-policy baselines, improving the average score by 2.2 points over the strongest baseline on Qwen2.5-7B-Instruct. Analysis reveals that the forward-KL objective provides an inherently selective distillation signal that outperforms direct trajectory imitation, and that strategy distillation exhibits complementary scaling with base model capability.