English is Not All You Need: Systematically Exploring the Role of Multilinguality in LLM Post-Training
Mehak Preet Dhaliwal · Shashwat Ranjan Chaurasia · Yao Qin · Dezhi Hong · Thomas Butler
English-only post-training is suboptimal; adding more languages improves performance across both English and non-English settings while enabling strong cross-lingual transfer.
摘要
Despite the widespread multilingual deployment of large language models, post-training pipelines remain predominantly English-centric, contributing to performance disparities across languages. We present a systematic, controlled study of the interplay between training language coverage, model scale, and task domain, based on 220 supervised fine-tuning runs on parallel translated multilingual data mixtures spanning mathematical reasoning and API calling tasks, with models up to 8B parameters. We find that English-only post-training is typically suboptimal: incorporating even a single non-English language improves both English performance and cross-lingual generalization. Increasing language diversity during post-training generally yields further gains, particularly for low-resource languages, while performance on high-resource languages tends to plateau rather than degrade. Moreover, greater language diversity enables strong zero-shot transfer to unseen languages, reducing the need for direct inclusion, though gains remain limited for typologically distant, low-resource languages.