Disentangling MLP Neuron Weights in Vocabulary Space
Asaf Avrahamy · Yoav Gur-Arieh · Mor Geva
A data-free method that disentangles MLP neuron weights into interpretable components via kurtosis-maximizing rotations, outperforming activation-based methods for neuron description.
摘要
Interpreting the information encoded in model weights remains a fundamental challenge in mechanistic interpretability. In this work, we introduce ROTATE (Rotation-Optimized Token Alignment in weighT spacE), a data-free method requiring no forward passes that disentangles MLP neurons directly in weight space. Our approach relies on a key statistical observation: neurons that encode coherent, monosemantic concepts exhibit high kurtosis when projected onto the model's vocabulary. By optimizing rotations of neuron weights to maximize their vocabulary-space kurtosis, our method recovers sparse, interpretable directions which we name vocabulary channels. Experiments on Llama-3.1-8B-Instruct and Gemma-2-2B-it demonstrate that ROTATE consistently recovers vocabulary channels that are faithful to the neuron's behavior; ablating individual channels selectively disables corresponding input activations or the promotion of specific concepts. Moreover, aggregating channel-level descriptions yields comprehensive neuron descriptions that outperform optimized activation-based baselines by 2–3× in head-to-head comparisons. By providing a fine-grained, data-free decomposition of neuron weights, ROTATE offers a building block for scalable weight-space interpretability.