作者位次 #6/7ACL 2026Findings· Findings of ACL
Multilingual Tokenization through the Lens of I ndian Languages: Challenges and Insights
Maharaj Brahma · N J Karthika · Rajat Verma · Nagasai Saketh Naidu · Rohit Saluja · Maunendra Sankar Desarkar · Ganesh Ramakrishnan
摘要
Tokenization plays a pivotal role in NLP and is fundamental to training language models. However, existing tokenizers are often skewed towards high-resource languages, limiting their effectiveness for linguistically diverse and morphologically rich languages such as those in the Indian subcontine...