The Format Tax

1. Problem Statement

The paper addresses a critical issue in the deployment of Large Language Models (LLMs): the degradation of reasoning and writing performance when models are forced to generate structured output (e.g., JSON, XML, Markdown, LaTeX).

• The Implicit Contract: The industry standard assumes that requiring structured output is merely a formatting constraint that should be invisible to the model's reasoning capabilities. The model should "think" the same thoughts, only expressing them in a valid syntax.
• The Reality: The authors demonstrate that this contract is broken for open-weight models. Enforcing structured output requirements causes a substantial drop in accuracy and quality compared to freeform generation.
• The Gap: While recent research has focused on Grammar-Constrained Decoding (GCD)—techniques that mask invalid tokens during generation to ensure syntactic validity—the authors hypothesize that the primary source of performance loss occurs earlier in the pipeline, at the prompt level, rather than during decoding.

2. Methodology

The authors conducted a systematic empirical study across 10 models (6 open-weight, 4 closed-weight API models), 4 tasks (MATH-500, GPQA, ZebraLogic, WritingBench), and 4 output formats (JSON, XML, Markdown, LaTeX).

Experimental Design

The study isolates the source of degradation by varying two dimensions while holding task and format constant:

1. Generation Strategy:
   • 1-Turn: Single-pass generation where the model must produce reasoning and the final structured output simultaneously.
   • 2-Turn: A two-step pipeline where Turn 1 generates a freeform answer (no format constraints), and Turn 2 reformats that answer into the target structure.
   • Thinking: Models with "extended thinking" capabilities generate a hidden reasoning trace (scratchpad) before producing the visible structured output.
2. Prompt Contents:
   • Variations included providing the grammar schema (G), few-shot examples (E), and task descriptions (T).
   • Constrained Decoding (C): Some configurations activated GCD, while others did not, allowing the authors to measure the specific cost of the decoder versus the prompt.

Evaluation Metrics

• Reasoning Tasks: Accuracy measured against ground truth. For MATH-500, an LLM judge was used to determine mathematical equivalence to avoid penalizing correct answers embedded in complex markup.
• Writing Tasks: Quality judged by an LLM, with controls to ensure the judge evaluated content rather than markup aesthetics.
• Format Tax Definition: $Tax = Perffreeform - Perfformat-constrained$. A positive value indicates performance degradation.

3. Key Contributions

1. Quantification of the Format Tax: The paper provides the first systematic measurement of the accuracy gap between freeform and structured generation across diverse models and tasks.
2. Source Attribution (Prompt vs. Decoder): The study decomposes the tax into two components:
   • Prompt-Level Degradation (Dominant): The cost incurred simply by the model receiving instructions to produce structured output.
   • Decoder-Level Sampling Bias (Minor): The additional cost incurred by grammar-constrained decoding (masking tokens).
3. Decoupling Strategy: The authors propose and validate a simple principle: decouple reasoning from formatting. They show that separating these concerns recovers most of the lost accuracy.
4. Closed-Weight vs. Open-Weight Gap: The paper highlights that recent closed-weight models (e.g., Claude Haiku 4.5, Grok-4.1, GPT-5.4) exhibit near-zero format tax, suggesting the issue is a solvable training gap rather than an inherent limitation of structured generation.

4. Key Results

A. The Tax Exists and is Widespread

• Across six open-weight models, requiring structured output consistently reduced accuracy compared to freeform generation.
• The degradation was observed in math, science, logic, and writing tasks.
• Writing Quality: Directly requesting LaTeX output significantly degraded writing quality, a finding confirmed even after stripping markup to isolate content.

B. The Source is the Prompt, Not the Decoder

• Comparison: When comparing a prompt requesting format (GET·) against the same prompt with grammar-constrained decoding active (GETC), the accuracy drop was nearly identical.
• Statistical Evidence: In 72 model-task-format cells, 92% of the statistically significant degradation was present before any decoder constraints were applied (i.e., just by reading the prompt). Only a small fraction (15 cells) showed additional degradation specifically due to GCD.
• Conclusion: The model's distribution is distorted by the instruction to format, not just the enforcement of the format.

C. Mitigation via Decoupling

• 2-Turn Strategy: Generating a freeform answer first, then reformatting it in a second pass, recovered most of the lost accuracy (average improvement of +6.8 percentage points).
• Thinking (Extended Reasoning): Models that generated a hidden reasoning trace before the final output also recovered significant accuracy, particularly on math and logic tasks.
• Prompt Engineering: Simply adding more few-shot examples or detailed schema descriptions to the prompt did not close the gap. The issue is structural, not a lack of instruction clarity.

D. The Closed-Weight Advantage

• Recent closed-weight models (e.g., claude-haiku-4.5, grok-4.1-fast) showed little to no format tax, often performing equally well in structured and freeform modes.
• This suggests the "tax" is not an inherent property of LLMs but a capability gap in current open-weight models that can be closed through better training or scaling.

5. Significance and Implications

• Paradigm Shift for Deployment: The paper challenges the assumption that "constrained decoding" is the primary solution for structured output. Instead, it argues that architectural or inference-time strategies (like 2-Turn pipelines or extended thinking) are necessary to maintain reasoning quality.
• Training Objectives: The fact that some models resist the tax implies that robustness to format constraints should be an explicit objective in the post-training (RLHF/SFT) of open-weight models.
• Cost-Benefit Analysis: While 2-Turn strategies recover accuracy, they double inference latency and token costs. Practitioners must weigh the accuracy recovery against these operational costs.
• Future Research: The findings suggest that future work on structured generation should focus less on refining token-masking algorithms (decoder) and more on training models to separate internal deliberation from external presentation (prompt/reasoning).

In summary, "The Format Tax" reveals that the cost of structured output is primarily a cognitive load imposed on the model by the prompt, not a technical limitation of the decoder. By decoupling reasoning from formatting, developers can recover the performance of open-weight models, bringing them closer to the capabilities of their closed-weight counterparts.