Retrieval Heads are Dynamic
Yuping Lin · Zitao Li · Yue Xing · Pengfei He · Yingqian Cui · Yaliang Li · Bolin Ding · Jingren Zhou · Jiliang Tang
摘要
Recent studies have identified “retrieval heads” in Large Language Models (LLMs) responsible for extracting information from input contexts. However, prior works largely rely on static statistics aggregated across datasets, identifying heads that perform retrieval on average. This perspective ove...