Jaroslav Kopčan

Tool-creating LLM agents are widely reported to benefit from accumulating reusable tool libraries. We test this claim with the create-and-discard ablation: build the tool, then throw it away. Across two benchmarks (BigCodeBench-Hard, 148 tasks; $\tau^2$-airline, 50 tasks), four models, and multiple retrieval variants including embedding-based and dependency-aware retrieval, creating and discarding tools matches creating and keeping them. Over every Create+Reuse--Create+Discard contrast we report---ten in total, spanning both benchmarks---the effect of keeping the tool has median $+0.4$pp and mean $-0.4$pp (range $-9.5$ to $+6.0$), and every contrast carrying a paired significance test is null ($p \geq 0.09$). On BigCodeBench-Hard all eight conditions cluster within a 2pp band. On $\tau^2$-airline, a wrapper-matched decomposition splits the raw +18pp Direct-to-Discard gain into +10pp from the agent harness itself, +6pp from a simple ``use your tools'' instruction, and only +2pp from the tool-creation structure. Tool creation helps agents engage with available capabilities, not build reusable memory---and agent-framework details alone can inflate reported gains by 10pp. We recommend that tool-creation evaluations report a create-and-discard ablation and a harness-matched baseline.