When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs
Suchit Gupte · Xueru Zhang · Mohammad Mahdi Khalili
Activation-aware pruning preserves sparse autoencoder interpretability in compressed LLMs by controlling representational disturbance, with middle layers being most fragile, inspiring a layer-wise sparsity allocation strategy.
摘要
Sparse autoencoders (SAEs) are widely used to interpret the internal representations of large language models (LLMs), yet their reliability under post-hoc model compression remains poorly understood. We present a systematic study of how pruning affects SAE behavior and theoretically show that its impact is governed by perturbation energy, a covariance-weighted norm. This perspective exposes a key limitation of magnitude pruning: by ignoring activation geometry, it distorts the learned representation space and degrades SAE functionality. Activation-aware methods such as Wanda and SparseGPT, in contrast, implicitly control perturbation energy and are therefore substantially more robust at preserving SAE behavior. We further reveal a consistent structural vulnerability across all pruning methods: middle layers are significantly more sensitive to pruning than early or late layers. Guided by this insight, we propose a Layer-wise sparsity allocation strategy, achieving lower perplexity under the same average pruning sparsity. Experiments across four model architectures validate our theoretical findings. We have made the code to reproduce our experiments publicly available at