Marek Suppa

Crowdsourced evaluation platforms such as Chatbot Arena enable real-time human assessment of model responses, but evaluating LLM-generated code remains challenging due to long outputs and the need to reason about execution behavior. We introduce BigCodeArena, an open human evaluation platform for code generation with an on-the-fly execution backend. BigCodeArena executes LLM-generated code and allows humans to interact directly with execution processes and outcomes. We collect over 14K code-centric conversations across 10 widely used LLMs, spanning 10 programming languages and 8 execution environments, including more than 4.7K multi-turn samples with pairwise human preferences. Our analysis reveals fine-grained preferences across tasks, languages, and frameworks. With the collected data, we construct two benchmarks: BigCodeReward, which measures alignment between reward models and human preferences, and AutoCodeArena, an automated Elo-style benchmark for code generation. Results show that execution feedback improves preference modeling, and that proprietary models such as GPT-5 and Claude-4 variants remain strong performers.

Tool-creating LLM agents are widely reported to benefit from accumulating reusable tool libraries. We test this claim with the create-and-discard ablation: build the tool, then throw it away. Across two benchmarks (BigCodeBench-Hard, 148 tasks; $\tau^2$-airline, 50 tasks), four models, and multiple retrieval variants including embedding-based and dependency-aware retrieval, creating and discarding tools matches creating and keeping them. Over every Create+Reuse--Create+Discard contrast we report---ten in total, spanning both benchmarks---the effect of keeping the tool has median $+0.4$pp and mean $-0.4$pp (range $-9.5$ to $+6.0$), and every contrast carrying a paired significance test is null ($p \geq 0.09$). On BigCodeBench-Hard all eight conditions cluster within a 2pp band. On $\tau^2$-airline, a wrapper-matched decomposition splits the raw +18pp Direct-to-Discard gain into +10pp from the agent harness itself, +6pp from a simple ``use your tools'' instruction, and only +2pp from the tool-creation structure. Tool creation helps agents engage with available capabilities, not build reusable memory---and agent-framework details alone can inflate reported gains by 10pp. We recommend that tool-creation evaluations report a create-and-discard ablation and a harness-matched baseline.