一作ACL 2026Poster· Natural Language Processing
T ok D rift: When LLM Speaks in Subwords but Code Speaks in Grammar
Yinxi Li · Yuntian Deng · Pengyu Nie
摘要
Large language models (LLMs) for code rely on subword tokenizers, such as byte-pair encoding (BPE), learned from mixed natural language text and programming language code but driven by statistics rather than grammar. As a result, semantically identical code snippets can be tokenized differently d...