💬 NLP

Decoupling Task-Solving and Output Formatting in LLM Generation

The paper introduces Deco-G, a decoding framework that improves large language model performance on complex tasks by decoupling problem-solving from rigid formatting requirements through a dedicated Format Estimation Module that ensures compliance without hindering reasoning.

Original authors: Haikang Deng, Po-Nien Kung, Nanyun Peng

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haikang Deng, Po-Nien Kung, Nanyun Peng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're asking a super-smart robot to solve a tricky math problem, but you also demand it write the answer in a very specific, rigid format, like "The final answer is 42." Usually, when you mix the "think hard" part with the "write perfectly" part in one big instruction, the robot gets confused. It's like trying to juggle while walking a tightrope; it often drops the ball (the math) or trips on the rope (the format).

The paper by Haikang Deng and colleagues introduces a new system called DECO-G to fix this. Think of DECO-G as a brilliant team-up between two specialists: a Problem Solver and a Format Police Officer.

The Problem: The "Entangled" Mess

In the old way, you'd give the robot one giant prompt: "Solve this math problem AND make sure you write the answer in a JSON box." The paper argues that this "entanglement" hurts the robot's brain. It suggests that when the robot tries to do both at once, it starts making mistakes in the math just to satisfy the formatting rules.

The authors explicitly argue against the idea that simply telling the robot to "be careful" or using rigid, pre-made templates (like forcing it to follow a strict JSON schema step-by-step) is the best solution. They found that these rigid methods often cut the robot off mid-thought, leading to incoherent answers or wrong math, even if the format looks perfect.

The Solution: The Two-Team Strategy

DECO-G splits the job.

The Problem Solver (The LLM): This is the main robot. It gets only the math problem. It doesn't worry about the format at all. It just thinks, reasons, and solves the problem freely.
The Format Police (The FEM): This is a separate, smaller helper module. It doesn't solve the math. Its only job is to watch the robot's output and whisper, "Hey, you're getting close to the end, but you need to say 'The final answer is' before you give the number."

The magic happens in how they talk. The Format Police uses a special "probabilistic lookahead" (a fancy way of peeking into the future) to guess: "If the robot says 'The' right now, what are the odds it will finish the sentence correctly?" If the odds are low, the Police gently nudges the robot's next word choice to steer it back on track without forcing it to stop thinking.

The Secret Weapons

To make this team-up work fast and without slowing down the computer, the authors introduced three cool tricks:

Instruction-Aware Distillation: Instead of training the Format Police on random, boring chat, they trained it specifically on how the robot behaves when it's actually trying to solve problems. This makes the Police much smarter at knowing what the robot is likely to do next.
Flexible Trie Building: Imagine a map of all the possible ways the answer could look. Usually, these maps get huge and messy if the answer can be different lengths. The authors built a "flexible" map that merges paths, so the computer doesn't get overwhelmed trying to check every single possibility.
HMM State Pruning: The Format Police has a massive brain with thousands of hidden states. The authors realized that most of the time, the robot only really cares about a tiny handful of those states. So, they told the Police to ignore the rest, cutting the computing work by about 13 times (from roughly 1.08 GFLOPs down to 0.08 GFLOPs per step) while keeping the accuracy almost the same.

What the Numbers Say

The team tested this on three different challenges:

Math Problems (GSM8k): When solving grade-school math, DECO-G consistently beat the other methods. For example, on the Llama-3.1-8B model, it got 85.2% of the answers right while keeping 100% format compliance. In contrast, a rigid method called "Outlines" only got 81.3% right, and a standard "Prompt-Only" approach got 82.3% right but failed the format most of the time.
Event Extraction: When asked to pull specific details from news stories, DECO-G improved the scores for identifying who did what (the "Argument Id" metric went up to 39.4 for Llama).
LLM-as-a-Judge: When the robot had to grade summaries, DECO-G helped it align better with human judges, achieving the highest average correlation scores (like 0.418 for Coherence on Llama).

The Verdict

The paper suggests that by separating the "thinking" from the "formatting," we can get the best of both worlds: answers that are mathematically correct and perfectly formatted. The authors measured this across real datasets and found that DECO-G consistently outperforms trying to do everything in one go or using rigid, forced constraints.

However, they note a few caveats. This system isn't a magic wand that works instantly on any robot; you have to train a new "Format Police" for each specific robot model you use. Also, for some models (like the Qwen series), the robot's brain is so focused that you have to push the Format Police harder (using a control factor of γ=2) to get it to listen.

In short, DECO-G suggests that if you want a robot to be a genius at math and a perfectionist at formatting, you shouldn't ask it to do both at the same time. Give it a partner to handle the rules, and watch it shine.

Technical Summary: Decoupling Task-Solving and Output Formatting in LLM Generation

Problem Statement

Large Language Models (LLMs) have demonstrated increasing proficiency in complex reasoning tasks such as mathematical problem-solving and automatic evaluation. However, performance often degrades when prompts combine task instructions with rigid output-format requirements. This "entanglement" forces the model to simultaneously optimize for reasoning accuracy and strict format adherence, creating competing goals that hinder the model's reasoning capabilities.

Existing approaches attempt to mitigate this through two main avenues, both of which have limitations:

Relaxing Constraints: Removing or softening format requirements often leads to low compliance rates, rendering outputs unusable for structured applications.
Constrained Decoding: Frameworks like Outlines or Guidance enforce strict token transitions to guarantee format compliance. However, these methods often intervene invasively, disrupting the model's internal reasoning flow and leading to abrupt cut-offs or incoherent outputs.

Recent studies (e.g., Tam et al., 2024; Long et al., 2025) indicate that stricter constraints correlate with greater reasoning degradation, suggesting a critical need for a framework that decouples format constraints from task solving without sacrificing the LLM's reasoning potential.

Methodology: DECO-G

The authors propose DECO-G, a decoding framework that explicitly separates task reasoning from output formatting. The core philosophy is to delegate format adherence to an auxiliary module, allowing the primary LLM to focus solely on problem-solving.

Architecture

The framework operates through a dual-module system:

Task-Solving LLM: Receives only the task-specific prompt (e.g., a math problem) without explicit formatting instructions.
Format Estimation Module (FEM): Receives the format constraints (e.g., "The final answer is..."). The FEM performs a probabilistic lookahead to estimate the likelihood that future token sequences will satisfy the format constraints.

At each decoding step, the FEM's estimated compliance probability is used to reweigh the LLM's original token distribution. The final probability distribution is constructed as:
$P(x_t|x_{<t}, \alpha) \propto P_{LM}(x_t|x_{<t}) [P_{FEM}(\alpha|x_{<t}, x_t)]^\gamma$
where $\alpha$ represents the format constraints and $\gamma$ is a steering strength hyperparameter.

Key Technical Innovations

To make this approach practical and efficient, the authors introduce three specific innovations:

Instruction-Aware Distillation:
Prior controllable generation methods (e.g., GeLaTo, Ctrl-G) distill Hidden Markov Models (HMMs) from unconditional random sampling. This fails to capture the instruction-conditional behavior of fine-tuned LLMs. DECO-G addresses this by distilling the HMM exclusively on instruction-response pairs from the Natural-Instructions-v2 dataset. This ensures the FEM accurately models the LLM's behavior when following specific task instructions.
Flexible Trie Construction:
Real-world format constraints often involve complex templates with fixed text (Pivots) and variable-length fields (Wildcards). Standard Deterministic Finite Automata (DFA) construction leads to state-space explosion ( $O(K^N)$ ) when handling chained variable-length segments. DECO-G introduces a flexible trie algorithm that enforces State Convergence. This strategy merges all valid paths of a wildcard segment back to a single canonical entry state for the next segment, reducing complexity to linear growth ( $O(N \times K)$ ).
HMM State Pruning:
The probabilistic lookahead in the FEM involves calculating emission probabilities over the entire vocabulary, which is computationally expensive ( $O(h|V|)$ ). The authors observe that the hidden state distribution is highly sparse. By pruning the HMM to consider only the top- $k$ most probable states (e.g., top 5%), they reduce the complexity to $O(k|V| + h \log h)$ . Empirical results show that keeping just 5% of states retains over 98% of the model's performance while reducing FLOPs by approximately 13 $\times$ .

Experimental Results

The authors evaluated DECO-G across three distinct tasks: Mathematical Reasoning (GSM8k), Event Argument Extraction (ACE05), and LLM-as-a-Judge (SummEval), using models like Llama-3.1-8B, Qwen2.5-7B, and Qwen3-8B.

Mathematical Reasoning (GSM8k): DECO-G consistently outperformed baselines. While standard prompting (Prompt-Only) achieved high accuracy but low format compliance, and structured generation (Outlines) guaranteed compliance but degraded accuracy, DECO-G achieved 100% format compliance while maintaining or improving reasoning accuracy. For instance, on Llama-3.1-8B, DECO-G achieved 85.2% accuracy compared to 81.3% for Outlines and 82.3% for Prompt-Only (NL).
Event Argument Extraction (ACE05): DECO-G improved overall extraction quality, particularly in identifying Argument IDs (AI) and Argument-attached IDs (AI+), suggesting better integration of roles and extracted arguments.
LLM-as-a-Judge (SummEval): DECO-G demonstrated the strongest average correlation with human annotators across coherence, consistency, fluency, and relevance metrics. Qualitative analysis revealed that DECO-G allows for more natural integration of scoring phrases at arbitrary positions, whereas baselines often forced rigid appending of ratings.

Significance and Claims

The paper claims that DECO-G represents a novel paradigm in controllable text generation by achieving a "responsibility separation" between reasoning and formatting.

Preservation of Reasoning: By removing the cognitive burden of formatting from the LLM's immediate generation process, the framework unlocks the model's full reasoning potential.
Guaranteed Compliance: Unlike unconstrained prompting, DECO-G provides a mathematical guarantee of format adherence through the FEM's reweighing mechanism.
Practical Efficiency: Through instruction-aware distillation, flexible trie construction, and state pruning, the authors demonstrate that the computational overhead is negligible (constituting ~0.53% of the LLM's main forward pass), making the framework viable for real-world deployment.

The authors conclude that this decoupling approach leads to overall performance gains, driven by the ability to enforce strict constraints without the reasoning degradation typically associated with rigid decoding frameworks. They acknowledge limitations, such as the need to distill a new HMM for different LLM families and the sensitivity of the steering hyperparameter $\gamma$ to the model's token distribution skewness.