Leandro von Werra

Crowdsourced evaluation platforms such as Chatbot Arena enable real-time human assessment of model responses, but evaluating LLM-generated code remains challenging due to long outputs and the need to reason about execution behavior. We introduce BigCodeArena, an open human evaluation platform for code generation with an on-the-fly execution backend. BigCodeArena executes LLM-generated code and allows humans to interact directly with execution processes and outcomes. We collect over 14K code-centric conversations across 10 widely used LLMs, spanning 10 programming languages and 8 execution environments, including more than 4.7K multi-turn samples with pairwise human preferences. Our analysis reveals fine-grained preferences across tasks, languages, and frameworks. With the collected data, we construct two benchmarks: BigCodeReward, which measures alignment between reward models and human preferences, and AutoCodeArena, an automated Elo-style benchmark for code generation. Results show that execution feedback improves preference modeling, and that proprietary models such as GPT-5 and Claude-4 variants remain strong performers.

Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing strategy, generator model, and source data, remain absent. We conduct extensive controlled experiments, generating over one trillion tokens, to identify critical factors in rephrasing web text into synthetic pretraining data. Our results reveal that structured output formats, such as tables, math problems, FAQs, and tutorials, consistently outperform both curated web baselines and prior synthetic methods. Notably, increasing the size of the generator model beyond 1B parameters provides no additional benefit. Our analysis also demonstrates that the selection of the original data used for mixing substantially influences performance. By applying our findings, we develop FinePhrase, a 486-billion-token open dataset of rephrased web text. We show that FinePhrase outperforms all existing synthetic data baselines while reducing generation costs by up to 30 times. We provide the dataset, all prompts, and the generation framework to the research community.