💬 NLP

Repair, Not Improvement: Decomposing Constrained Decoding in Tool-Call Abstention

This paper demonstrates that constrained decoding in tool-call scenarios often fails to repair abstention errors because the negative impact of enforcing stop tokens frequently outweighs the benefits of restricting token choices, ultimately revealing that prior claims about language-specific performance differences are unsupported.

Original authors: Janghoon Lee (Redrob)

Published 2026-08-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Janghoon Lee (Redrob)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Setup: AI, Rules, and the Art of Saying "No"

Imagine you are teaching a very eager, super-fast robot to be a helpful assistant. This robot is great at writing stories, solving math problems, and even pretending to be a wizard. But there's a catch: sometimes, the best thing this robot can do is admit it doesn't know the answer or that it shouldn't do anything at all. In the world of AI, this is called "abstention." If you ask a robot to check the weather, but you actually want to know a legal secret, the robot should say, "I can't help with that," rather than confidently making up a fake legal opinion.

To keep these robots from making up wild stories or breaking rules, engineers use a trick called "constrained decoding." Think of this like giving the robot a strict coloring book. Instead of letting the robot draw anywhere on the page, the coloring book only has specific outlines (like a JSON format or a list of allowed tools) that the robot is allowed to color inside. If the robot tries to draw outside the lines, the computer stops it immediately. For a long time, scientists thought this trick was almost free: it made the robot's answers look perfect and neat without hurting its actual brainpower. But there was a big question mark hanging over one specific situation: what happens when the robot needs to say "I can't do this"? Does the strict coloring book force the robot to pick a tool even when it shouldn't? This paper dives into that exact question, testing whether the rules that make answers look good actually break the robot's ability to know when to stop.

The Story: When Rules Break the "No" Button

The researchers in this paper decided to put this idea to the test using a clever experiment. They took several AI models (ranging from tiny ones with 0.6 billion "brain cells" to slightly larger ones with 4 billion) and asked them a simple question: "Should I use a tool, or should I just say 'None'?" They did this in two languages, English and Korean, to see if the language changed the result.

To understand exactly what was going wrong, they didn't just compare "free" answers to "restricted" answers. They added a middle step, like a three-act play:

The Free Act: The robot answers however it wants, with plenty of space to explain itself.
The Short Act: The robot answers in just one line, but it's still free to say whatever it wants. This tests if just cutting off the robot's explanation hurts its accuracy.
The Restricted Act: The robot answers in one line, but it's forced to pick from a strict list of tools (or "None"). This is the standard "constrained decoding" method.

By comparing these three acts, the scientists could separate two different problems: the cost of cutting the robot's speech short, and the cost of forcing it to pick from a list.

The Big Discovery: Repair, Not Improvement

The results were surprising and a bit counterintuitive. The paper found that when you force a robot to pick from a strict list of tools, it doesn't actually get smarter at deciding when to say "no." In fact, for the smallest models, the strict rules made them significantly worse at abstaining. In one specific case (the tiny 0.6B model in Korean), the strict rules caused a drop of 29.5 percentage points in accuracy compared to letting the robot speak freely.

However, the story gets more interesting when you look at why the rules helped in some cases. The researchers discovered that the strict rules acted like a "repair kit" for messy answers, not a "brain upgrade." When the robot was forced to pick from the list, it stopped producing gibberish or unreadable text. It turned "nonsense" into "correct-looking answers." But it didn't turn "wrong decisions" into "right decisions."

Here is the crucial part: The researchers looked at 698 instances where the strict rules changed a wrong answer into a right one. They found that 545 of those were cases where the robot had originally produced no readable answer at all. Zero of them were cases where the robot had already made the correct decision but was blocked by the format. This means the rules didn't fix the robot's judgment; they just fixed its handwriting.

The Hidden Trap: The "Stop" Token vs. The "Enum"

The paper also broke down the strict rules into two parts to see which one was the villain:

The Stop Token: This is the rule that says, "Stop writing after one line."
The Enum: This is the rule that says, "You can only pick from this specific list of tools."

They found these two parts were fighting each other. The "Stop Token" (cutting the answer short) was actually the main culprit, hurting the robot's ability to say "None" by about 20.0 points in some cases. The "Enum" (the list of tools) tried to fix this by adding back about 19.5 points of accuracy. When you put them together, they almost canceled each other out, leaving a tiny net loss of 0.5 points.

This explains why previous studies might have missed the problem. If you just look at the final number, it looks like the rules didn't do much harm. But if you look under the hood, you see a massive loss being hidden by a massive repair. The rules didn't improve the robot's decision-making; they just patched up the damage caused by cutting the robot's speech short.

The Verdict: Don't Expect a Brain Upgrade

The paper concludes with a clear warning for anyone building AI systems. If you use strict formatting rules to make your AI look professional, don't expect it to become better at knowing when not to act. The rules will make the output look clean and readable, but they won't fix a robot that is confused about whether it should help or not.

In fact, the study suggests that these rules might actually push the robot to act more often than it should. When the robot is forced to pick from a list, it tends to move its "decision line" toward picking a tool, even when the right answer is to do nothing. This is dangerous for safety: a robot that is forced to pick a tool might confidently call a weather API when you ask it a legal question, simply because the rules didn't give it a clear way to say "I can't."

The researchers tested this across different models and languages (English and Korean) and found the pattern held true. The strict rules are great for fixing messy text, but they are not a magic wand for improving judgment. As the paper puts it, the rules "make unreadable output readable, which is worth having, and it leaves the judgement where it found it." So, if you want your AI to know when to say "no," you can't just rely on a strict checklist; you have to teach the robot itself to know when to stop.

Technical Summary: Repair, Not Improvement: Decomposing Constrained Decoding in Tool-Call Abstention

Problem Statement

Recent work on constrained decoding (e.g., Lee et al., 2026a) has established that format constraints (like JSON schemas) incur a performance cost, but that the majority of this cost stems from the prompt instructions rather than the decoder enforcement. However, that prior work explicitly excluded function calling from its conclusions, noting that constraints in this domain encode correctness requirements (ruling out specific answers) rather than mere presentation preferences (ruling out specific ways of writing an answer).

This paper investigates the specific case of tool-call abstention: the scenario where a model must correctly decline to call any tool when none applies. In this setting, a constrained decoder (restricted to an enum of tool names plus an abstention token) faces a unique challenge. The constraint does not just format the output; it restricts the set of possible answers, potentially deleting the correct "no tool" decision. The central question is whether the decoder's contribution to performance degradation is negligible in this setting, as suggested by prior work for other formats, or if it is significant because the constraint directly interferes with the model's judgment.

Methodology

The study employs a rigorous experimental design to isolate the effects of the decoder from the prompt and to separate the effects of length constraints from lexical constraints.

Experimental Setup

Models: Open-weight Qwen3 models (0.6B, 1.7B, and 4B parameters).
Languages: English and Korean.
Benchmark: MetaTool Task2, utilizing matched pairs of "tool-needed" and "tool-abstention" items. The dataset includes 200 abstention items and 200 tool-needed items where queries are identical across conditions.
Scoring: Strict deterministic rules adopted from Lee et al. (2026b). An abstention is correct only if the output contains the abstention token and no tool name (or a tool name appearing after it). Multi-candidate outputs are scored as wrong.

Three-Condition Design

To decompose the "constraint tax," the author runs three conditions on byte-identical prompts:

contract-free: Unconstrained generation with a 256-token budget (allows for explanations).
contract-short: Unconstrained generation but stopped at the first newline (removes the ability to explain, isolating the cost of length restriction).
constrained: Generation restricted to an enum of the 10 candidate tools plus the abstention token, stopped at the first newline.

Metrics and Decomposition

The author defines three differences to isolate specific effects:

$\Delta_{length}$ : acc(short) - acc(free). Measures the cost of restricting the output length (stopping at the first line).
$\Delta_{mask}$ : acc(constrained) - acc(short). Measures the marginal cost of the grammar (the enum) given that the length is already controlled.
$\Delta_{total}$ : acc(constrained) - acc(free). The total cost of constrained decoding, which is the sum of the above two.

This design allows the author to distinguish between the cost of "formatting" (stopping early) and the cost of "masking" (restricting the token set).

Mechanism Variable

The study measures masked_mass, defined as the probability mass the model assigns to tokens forbidden by the enum grammar in the unconstrained condition. This is the complement of the "feasible mass" proposed by Reddy et al. (2026).

Key Contributions

Decomposition of the Constraint Tax: The paper demonstrates that the total cost of constrained decoding in tool-abstention settings is not a single negative value but a sum of two components with opposite signs.
- The stop token (length constraint) often causes a significant drop in accuracy (e.g., -20.0 points).
- The enum constraint (lexical masking) often recovers a portion of that loss (e.g., +19.5 points).
- Consequently, the total effect ( $\Delta_{total}$ ) can be near zero or negative, masking the fact that the grammar is actively "repairing" the damage caused by the length constraint rather than improving the model's inherent judgment.
Repair vs. Improvement: The author shows that the constraint does not improve the model's decision-making capability. Of the 698 abstentions "repaired" by the constraint (turned from wrong to right), 545 had no readable answer in the unconstrained output, and 0 were cases where the model had made the correct judgment but failed due to formatting. The constraint simply forces the model to output a valid token, recovering readability but not enhancing the underlying logic.
Shift in Decision Boundary: While the constraint "repairs" readability, it also shifts the decision boundary. Against a free-generation baseline, the constraint increases tool-selection accuracy but decreases abstention accuracy. This indicates the grammar pushes the model toward naming a tool, even when abstention is correct.
Language Independence: Contrary to hypotheses that language-specific probability distributions might alter the cost, the study finds that the within-item statistic (comparing failures between languages on items both got right without constraints) shows no significant difference between Korean and English. However, the masked probability mass does not fully explain the language difference: in the logistic regression, the language coefficient excluded zero, indicating a residual language effect that the mass variable does not account for.
Non-Linear Mechanism: The relationship between masked_mass and the abstention penalty is not linear. There is a breakpoint at approximately 0.480. Below this threshold, the mass predicts little about the penalty; above it, the penalty increases steeply.

Results

Abstention Performance: Against the unconstrained, free-generation baseline, $\Delta_{total}$ on abstention is negative (indicating a performance drop) with intervals excluding zero in four of six model/language cells, with a worst-case drop of -29.5 points. It is never positive with an interval excluding zero.
Decomposition: In the smallest model (0.6B) for Korean, the stop token alone costs -20.0 points, but adding the enum recovers +19.5 points, leaving a net cost of -0.5. This confirms the "repair" hypothesis: the grammar fixes the damage caused by the stop token.
Tool Selection: On tool-needed items, the constraint consistently improves accuracy (positive $\Delta_{total}$ in all cells), confirming that the constraint helps when a tool is required.
Language Claims: Both preregistered claims regarding language differences failed to support the initial hypothesis of a language-dependent cost. The within-item statistic showed no significant difference between Korean and English, and the masked probability mass did not absorb the variance separating the languages (the language coefficient remained significant in the regression).

Significance and Claims

The paper concludes that the "nearly free" enforcement of constraints found in prior work does not hold for function calling abstention. The primary significance lies in the decomposition of the constraint effect:

Separation of Concerns: The stop token (length control) and the enum (lexical control) are separable components with opposing effects. A two-condition design (comparing free vs. constrained) sums these into a single number, potentially hiding a large loss and a large recovery that cancel out.
Nature of the Gain: The benefit of constrained decoding in this setting is form, not judgement. It makes unreadable outputs readable but does not improve the model's ability to decide when to abstain. In fact, it may degrade that decision by biasing the model toward tool invocation.
Practical Implication: Practitioners should report the stop token and the grammar separately. Relying on a single aggregate metric can lead to the false conclusion that constrained decoding is neutral or beneficial for abstention, when in reality, it may be actively harming the model's ability to correctly decline tool calls.

The paper maintains a modest scope, noting that its findings are specific to the abstention setting and the specific benchmark used (where the gold tool is always the first candidate, a property that limits generalizability but does not affect the differences measured). It does not propose new training methods or architectures, but rather provides a measurement framework to better understand the costs of current inference-time constraints.