Understanding Primacy Effects in Large Language Models with Sparse Autoencoders
Ikhyun Cho · Julia Hockenmaier
We provide novel mechanistic insights into the primacy effect in large language models using sparse autoencoders.
摘要
The *primacy effect* is a well-established cognitive phenomenon in which information presented early in a sequence is often weighted more strongly than information presented later. Recent work suggests that large language models (LLMs) exhibit analogous behavior across a range of settings, yet the representation-level mechanisms underlying this bias remain underexplored. In this work, we provide a representation-level account of primacy effects in LLMs using sparse autoencoders (SAEs). We propose the *Dedicated Primacy Subspace* (DPS) hypothesis: LLMs appear to preferentially recruit a particular representational subspace to encode the semantics of the *first* demonstration, which subsequently modulates the processing of later context. Using few-shot classification as our testbed, we show that this mechanism has two complementary forms: *DPS-Affirmative* features, which encode support for the first-demonstration label, and *DPS-Contrastive* features, which activate when subsequent inputs contradict that initial label, thereby emphasizing competing alternatives. We further show that a subset of these features exhibits primacy-driven causal effects under intervention, providing direct causal support for the proposed mechanism. Taken together, these results provide a concrete representation-level account of how primacy effects manifest in LLMs.