Binyuan Hui

Crowdsourced evaluation platforms such as Chatbot Arena enable real-time human assessment of model responses, but evaluating LLM-generated code remains challenging due to long outputs and the need to reason about execution behavior. We introduce BigCodeArena, an open human evaluation platform for code generation with an on-the-fly execution backend. BigCodeArena executes LLM-generated code and allows humans to interact directly with execution processes and outcomes. We collect over 14K code-centric conversations across 10 widely used LLMs, spanning 10 programming languages and 8 execution environments, including more than 4.7K multi-turn samples with pairwise human preferences. Our analysis reveals fine-grained preferences across tasks, languages, and frameworks. With the collected data, we construct two benchmarks: BigCodeReward, which measures alignment between reward models and human preferences, and AutoCodeArena, an automated Elo-style benchmark for code generation. Results show that execution feedback improves preference modeling, and that proprietary models such as GPT-5 and Claude-4 variants remain strong performers.

Inspired by the success of scaling language modeling through massive real-world corpora, we demonstrate that similar power-law patterns exist in preference modeling. We propose \textbf{World Preference Modeling (WorldPM)}, which model a robust consensus on intrinsic utility from real-world user interactions. We curate a 15M-scale preference dataset from online forum data and conduct large-scale experiments on models from 1.5B to 72B parameters. Across 10 objective preference test sets with consensus-based labels, we observe two distinct scaling patterns: (1) common-sense preference (e.g., identifying factual errors) exhibits stable log-linear improvement across all model sizes, while (2) domain preference (e.g., coding, math, knowledge QA) reveals an emergent phenomenon where only sufficiently large models achieve meaningful scaling. Through training dynamics analysis, we reveal the learning mechanism behind preference scaling: the model first learns to discriminate using surface preferences, but counter-surface examples progressively compel it to discover intrinsic utility beyond surface features as training scales. Further experiments validate WorldPM as an effective foundation for preference fine-tuning, broadly improving generalization across human preference datasets of varying sizes (7K, 100K, and 800K), with gains exceeding 5\% on many key subtasks.

The dominant Fill-in-the-Middle (FIM) paradigm for code completion is constrained by its rigid inability to correct contextual errors and reliance on unaligned, insecure Base models. While Chat LLMs offer safety and Agentic workflows provide flexibility, they suffer from performance degradation a...