Ziheng Wang

This paper explores attention attractors —tokens that draw significantly high attention—in large language models. We analyze them from three perspectives: (1) Functionality : We demonstrate their role in aggregating information from preceding contexts to facilitate future predictions. (2) Distrib...