作者位次 #2/4ACL 2026Findings· Findings of ACL
M orph BPE : Morphology-Aware Tokenization for Efficient LLM Training
Ehsaneddin Asgari · Yassine El Kheir · MohammadAli SadraeiJavaheri · Ali Nazari
摘要
Tokenization fundamentally shapes NLP performance, affecting both efficiency and linguistic fidelity. While Byte Pair Encoding (BPE) underpins most Large Language Models (LLMs), its frequency-driven merges often disregard morpheme boundaries, yielding inconsistent and semantically opaque segmenta...