Julia Hockenmaier

Instruction hierarchies are a core safety assumption of language model deployment: higher priority inputs, such as system prompts, should override conflicting lower priority inputs from users or tools. Yet frontier LLMs often violate this hierarchy. We introduce V-Steer, a training-free inference time method that restores privileged influence by editing cached value vectors at prompt positions. Using direct logit attribution on the first next token prediction, V-Steer identifies heads where lower priority spans dominate privileged ones, then boosts privileged spans and suppresses conflicting lower priority spans through in-place multiplicative edits to cached $V$ tensors. Since the method acts only on cached values, it remains compatible with fused attention backends and adds only a one time prefill overhead. Across models from 7B to 70B, this attribution guided intervention raises primary constraint accuracy from under 18\% up to 92\% on controlled role conflict benchmarks, and on broader instruction hierarchy evaluations substantially outperforms prompt only baselines while matching or exceeding SoTA training based methods on 3 of 4 scales of LLMs, with negligible decoding-speed overhead.

The *primacy effect* is a well-established cognitive phenomenon in which information presented early in a sequence is often weighted more strongly than information presented later. Recent work suggests that large language models (LLMs) exhibit analogous behavior across a range of settings, yet the representation-level mechanisms underlying this bias remain underexplored. In this work, we provide a representation-level account of primacy effects in LLMs using sparse autoencoders (SAEs). We propose the *Dedicated Primacy Subspace* (DPS) hypothesis: LLMs appear to preferentially recruit a particular representational subspace to encode the semantics of the *first* demonstration, which subsequently modulates the processing of later context. Using few-shot classification as our testbed, we show that this mechanism has two complementary forms: *DPS-Affirmative* features, which encode support for the first-demonstration label, and *DPS-Contrastive* features, which activate when subsequent inputs contradict that initial label, thereby emphasizing competing alternatives. We further show that a subset of these features exhibits primacy-driven causal effects under intervention, providing direct causal support for the proposed mechanism. Taken together, these results provide a concrete representation-level account of how primacy effects manifest in LLMs.