作者位次 #5/7ACL 2026Poster· Natural Language Processing
Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
Negar Foroutan · Clara Meister · Debjit Paul · Joel Niklaus · Sina Ahmadi · Antoine Bosselut · Rico Sennrich
摘要
Tokenization is the first—and often least scrutinized—step of most NLP pipelines. Standard algorithms for learning tokenizers rely on frequency-based objectives, which favor languages dominant in the training data and consequently leave lower-resource languages with tokenizations that are disprop...