一作ACL 2026Poster· Natural Language Processing
MUTANT : A Recipe for Multilingual Tokenizer Design
Souvik Rana · Ashish Kulkarni · Arul Menezes · Chandra Khatri · Shubham Agarwal
摘要
Tokenizers play a crucial role in determining the performance, training efficiency, and the inference cost of Large Language Models (LLMs). Designing effective tokenizers for multilingual LLMs is particularly challenging due to diverse scripts and rich morphological variation. While subword metho...