Sememe: Causal Modeling for Task-Aware Depth Pruning of Large Language Models
Charles Yu · Heng Ji
We introduce a depth-/layer-pruning approch for Large Language Models which models the impact of each transformer block on the LLM's performance to determine which transformer block to prune.
摘要
Large Language Models (LLMs) exhibit strong generalist capabilities, yet task-specific behavior is often supported by diffuse and redundant structures. While pruning has been widely explored as a means to reduce inference costs, few existing methods translate theoretical sparsity into tangible speedups without sacrificing commensurate performance. We introduce **Sememe**, a task-aware depth-pruning method that frames block-importance estimation as causal inference, treating block removals as interventions in a Structural Causal Model (SCM) and estimating importance scores as path coefficients from per-example calibration losses, with a provable unbiasedness guarantee. Sememe identifies redundancy using a small calibration set and only the forward pass, avoiding the memory overhead of gradients or activations. Across Llama2, Llama3.1, Mistral, and Qwen3 models, Sememe with 32 calibration examples matches or exceeds the downstream retention that other training-free depth-pruning methods reach with 256-512 examples, while remaining stable where other methods degenerate. Sememe's interventional formulation naturally extends to joint multi-block removals and to task-aware calibration, which prunes toward a target skill from a small set of in-task examples. At larger scales, we find this depth pruning to be highly effective: Sememe retains 93.3\% of downstream accuracy when removing 20\% of Llama3.1-70B's layers, with speedups proportional to the fraction of removed layers on any hardware.