Pontus Stenetorp

Prior work has shown that large language models (LLMs) often converge to accurate input embedding for numbers, based on sinusoidal representations.In this work, we demonstrate that these representations are in fact strikingly systematic, to the point of being almost perfectly universal: different...

Multilingual large language models achieve impressive cross-lingual performance despite largely monolingual pretraining. While bilingual data in pretraining corpora is widely believed to enable these abilities, details of its contributions remain unclear. We investigate this question by pretraini...