Maitrey Mehta

All languages are equal; when it comes to tokenization, some are more equal than others. Tokens are the hidden currency that dictates the cost and latency of contemporary LLM access. However, many languages written in non-Latin scripts observe a poor exchange rate: LLMs take several multiples of tokens to encode the same information in many languages as they do for English. We observe that this issue, known as token over-fragmentation, persists in modern open-weight LLMs. The standard remedy is vocabulary expansion that adds target language items missing from the model’s vocabulary. We focus on two core decisions in the vocabulary expansion process: What items should we add? and How should we initialize their corresponding input and output embeddings? We comprehensively study and advance the new research direction of interpretability-based vocabulary expansion along these two core decisions, making four key contributions: First, we question the conventional use of frequency-based methods to choose candidate vocabulary items to add (a decision long treated as settled), and show that interpretability-based methods offer a superior performance-token efficiency tradeoff. Second, we strengthen the case for interpretability-based embedding initialization by showing large gains (∼20 pts) over baseline initialization methods for several languages written in non-Latin scripts. Third, we identify the phenomenon of “subword detokenization,” where models progressively merge fragmented subword tokens into larger subwords across layers. Fourth, we propose FragMend, a method that leverages this phenomenon to push the efficiency ceiling of interpretability-based expansion, and validate its effectiveness.

While dense retrieval models have been the standard for state-of-the-art information retrieval, their deployment is often constrained by high memory requirements and reliance on GPU accelerators for vector similarity search at scale. Learned sparse retrieval offers a compelling alternative by ena...