一作ACL 2026Poster· Natural Language Processing
Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet
Milan Miletić · Julie Kallini · Ekaterina Shutova
摘要
Multilingual language models often exhibit performance disparities across languages that can arise as early as the tokenization stage. Widely-used subword tokenization approaches favor high-resource languages, and tokenizer-free methods still yield longer sequences for scripts with a higher bytes...