The Format Tax: Separating the Cost of Requesting Structure from the Cost of Enforcing It
Ivan Yee Lee · Loris D'Antoni · Taylor Berg-Kirkpatrick
Requiring structured output (JSON, XML, etc.) degrades LLM reasoning and writing. The cause is prompt-induced distribution shift, not decoder bias. Letting models reason freely then reformat recovers most lost accuracy.
摘要
Asking a large language model to respond in JSON should be a formatting choice, not a capability tax. Yet we find that structured output requirements—JSON, XML, LaTeX, Markdown—substantially degrade reasoning and writing performance across open-weight models. The research response has focused on constrained decoding, but sampling bias accounts for only a fraction of the degradation. The dominant cost enters at the prompt: format-requesting instructions alone cause most of the accuracy loss, before any decoder constraint is applied. This diagnosis points to a simple principle: decouple reasoning from formatting. Across six open-weight models, four API models, four formats, and tasks spanning math, science, logic, and writing, generating freeform first and reformatting in a second pass recovers most lost accuracy; enabling extended thinking within a single generation recovers more on reasoning tasks but can cost writing quality. Notably, most recent closed-weight models show little to no format tax, suggesting the problem is not inherent to structured generation but a gap that current open-weight models have yet to close. Code is available at https://github.com/ivnle/the-format-tax.