💻 computer science

Brief Is Better: Non-Monotonic Chain-of-Thought Budget Effects in Function-Calling Language Agents

This paper reveals that in function-calling language agents, brief Chain-of-Thought reasoning (around 32 tokens) significantly boosts accuracy by acting as an effective function-routing mechanism, whereas extended reasoning degrades performance through increased hallucinations, leading to the proposal of a structured "Function-Routing CoT" method that guarantees reliability without extensive budget tuning.

Original authors: Xuan Qi

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xuan Qi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a very smart, but slightly anxious, personal assistant to book a flight for you. You give them a list of 3 possible airlines (the "candidate functions") and ask them to pick the right one and fill out the form.

This paper asks a simple question: How much time should your assistant spend "thinking" before they actually book the ticket?

Most people assume: "The more they think, the better the result." We usually tell our assistants, "Think deeply, consider all angles, and don't rush!"

The authors of this paper discovered that this assumption is completely wrong. In fact, for this specific type of task, thinking too much makes the assistant terrible at their job.

Here is the breakdown of their discovery using simple analogies:

1. The "Goldilocks" Zone of Thinking

The researchers tested their AI assistants with different "thinking budgets" (how many words they were allowed to write before giving the answer).

Zero Thinking (Direct Answer): The assistant guesses immediately. They get it right about 44% of the time.
Brief Thinking (32 words): The assistant takes a quick moment to say, "Okay, I need to pick the airline." They get it right 64% of the time. This is the sweet spot.
Long Thinking (256+ words): The assistant starts overthinking. They wander down rabbit holes, doubt their first instinct, and start making things up. They get it right only 25% of the time. This is worse than not thinking at all!

The Analogy: Imagine you are trying to find a specific key in a dark room.

No thinking: You grab the first thing that feels like a key. (Sometimes it works).
Brief thinking: You shine a flashlight for one second, see the key, and grab it. (Best result).
Long thinking: You start searching the whole room, knocking over furniture, convincing yourself the key is actually a spoon, and eventually grabbing a random object that looks like a key but isn't. (Worst result).

2. Why Does Overthinking Fail? (The "Confusion Spiral")

The paper found two main reasons why long thinking destroys performance:

The "Wrong Turn" (Misdirection): When the assistant thinks for too long, they start arguing with themselves. "Maybe it's Airline A? No, wait, Airline B is better? But what about Airline C?" By the time they finish their long thought process, they have forgotten the original goal and pick the wrong airline entirely.
The "Hallucination" (Making things up): In their long, rambling thoughts, the assistant gets so confused that they invent a brand new airline that doesn't exist in your list. They confidently say, "I will book with SkyHigh Airlines," even though that airline wasn't on the menu.

The Metaphor: It's like a GPS that starts giving you directions.

Brief: "Turn left." (You arrive).
Long: "Turn left... wait, maybe right? No, let's go straight... actually, let's drive to the moon first..." (You crash).

3. The "Super-Shortcut" Solution: FR-CoT

Since the researchers realized the problem was that the assistant got lost during the thinking process, they invented a new rule called Function-Routing CoT (FR-CoT).

Instead of letting the assistant ramble, they force them to say the name of the airline first, before they are allowed to think anything else.

The Rule: "You must write: 'I choose [Airline Name]' immediately. Then, you can think about the details."
The Result: This acts like a seatbelt. It locks the assistant into the correct choice before they have a chance to overthink and change their mind.
The Outcome: This method gets the same high accuracy as the "brief thinking" method, but it completely eliminates the risk of the assistant inventing fake airlines. It's a structural guarantee that they won't hallucinate.

4. Does this apply to all AI?

The researchers tested this on different sizes of AI models (small and medium).

The Small/Medium Models: They followed the rule perfectly. Brief thinking = Good. Long thinking = Disaster.
The "Self-Stopping" Model: One specific model (Phi-3) was a bit different. When asked to think for a long time, it naturally stopped talking after a while because it got bored or satisfied. Because it stopped itself, it didn't crash as hard as the others. But even this model did best when it was told to think briefly.

The Big Takeaway

We often think "more compute" and "more reasoning" equals "smarter AI." This paper proves that for picking tools or APIs, less is more.

Don't let your AI overthink.
Keep the thinking window short (about 8 to 32 words).
Force the AI to commit to the right tool immediately.

If you let an AI "think" for too long before making a structured decision, it doesn't get smarter; it just gets confused and starts making things up. Brief is better.

1. Problem Statement

The paper addresses a critical gap in the design of language agents: How much reasoning (Chain-of-Thought or CoT) should an agent perform before selecting a tool or API function?

Context: In function-calling agents, the model must select a function from a predefined set and generate JSON arguments. Practitioners often assume "more reasoning is better," allocating large token budgets (e.g., 512 tokens) for CoT before action.
Hypothesis: The authors challenge this assumption, proposing that the relationship between reasoning length and accuracy in structured tool-use settings is non-monotonic. They investigate the optimal CoT budget ( $d$ ) across a range of token counts to determine if extended reasoning degrades performance due to "overthinking" or format erosion.

2. Methodology

The study employs a systematic experimental design using the Berkeley Function Calling Leaderboard (BFCL) v3 Multiple-function benchmark.

Models Evaluated:
- Primary: Qwen2.5-1.5B-Instruct (resource-constrained representative).
- Validation: Qwen2.5-7B-Instruct (cross-scale) and Phi-3-mini-4k-instruct (cross-architecture).
Experimental Protocol:
- Budget Sweep: Tested six token budgets ( $d \in \{0, 32, 64, 128, 256, 512\}$ $d \in {0, 32, 64, 128, 256, 512}$ ).
  - $d=0$ : Direct answer (no CoT).
  - $d>0$ : "Think step by step" prompt with a hard stop at $d$ tokens, followed by a forced JSON generation phase.
- Dataset: 200 tasks sampled from BFCL v3 Multiple.
- Evaluation: Accuracy measured by matching the generated JSON function name and arguments against ground truth.
Analysis Techniques:
- Oracle Analysis: Determining the minimum budget required for a correct answer per task ( $d^*$ ).
- Error Decomposition: Categorizing failures into (1) Hallucinated functions (names not in the set), (2) Wrong valid function selection, and (3) Wrong arguments.
- Entropy Gating: Analyzing pre-reasoning action entropy ( $H_0$ ) to predict if CoT would help.
- Proposed Intervention: Function-Routing CoT (FR-CoT), a structured prompt template forcing the model to commit to a valid function name immediately.

3. Key Findings & Results

A. Non-Monotonic CoT Effect

The central finding is a striking non-monotonic relationship between reasoning length and accuracy:

Brief CoT is Optimal: A budget of 32 tokens yields the highest accuracy.
- Qwen2.5-1.5B: Accuracy jumps from 44.0% (no CoT) to 64.0% (+45% relative improvement).
- Qwen2.5-7B: Accuracy jumps from 40.5% to 82.5%.
- Fine-grained sweep: The true optimum is even shorter, at 8–16 tokens.
Extended CoT is Detrimental: Increasing the budget beyond the optimum causes a severe collapse in performance.
- At $d=256$ , Qwen2.5-1.5B accuracy drops to 25.0% (worse than no CoT).
- At $d=512$ , it drops further to 22.5%.
- This collapse is statistically significant ( $p < 0.001$ ).

B. Mechanism: Error Decomposition

The paper identifies why performance collapses:

At $d=0$ : The dominant error is Wrong Valid Function Selection (30.5%). The model picks a valid candidate but the wrong one.
At $d=32$ (Brief): This error is nearly eliminated (dropping to 1.5%). The short reasoning acts as a "Function Routing" step, anchoring the model to the correct candidate.
At $d=256$ (Extended): The model suffers Dual Failure:
1. Misdirection: Wrong valid function selection resurges to 28.0%.
2. Hallucination: The model invents functions not in the candidate set (18.0%).
- Conclusion: Extended reasoning causes the model to "second-guess" itself, drift away from the correct path, or hallucinate, rather than simply failing to format the output.

C. Oracle Analysis

88.6% of solvable tasks require 32 tokens or less to solve correctly.
The average oracle cost is 27.6 tokens.
This confirms that the vast majority of function-calling tasks do not require deep reasoning chains.

D. FR-CoT (Function-Routing CoT)

To address the routing failure mode without budget tuning, the authors propose FR-CoT:

Mechanism: A structured prompt template that forces the model to output Function: [name] as the very first tokens of the reasoning phase.
Results:
- Achieves accuracy statistically equivalent to free-form $d=32$ (64.0% on 1.5B, 83.0% on 7B).
- Eliminates Hallucination: Reduces function hallucination to 0.0% (compared to 2.5–3.0% in baselines).
- Comparison to Constrained Decoding: FR-CoT outperforms constrained decoding (log-prob forced selection) on the 7B model by +19.5 percentage points (83.0% vs. 63.5%). This proves that structured reasoning (committing early) is more effective than just output constraints (forcing the final token).

E. Cross-Architecture Validation

Qwen2.5 Family: Both 1.5B and 7B models show the non-monotonic collapse. The 7B model suffers a more severe collapse (dropping to 18.0% at $d=256$ ).
Phi-3-mini: Confirms the peak at $d=32$ $d = 32$ (86.0%) but does not collapse below the no-CoT baseline.
- Mechanism: Phi-3 has a high natural EOS (End-of-Sequence) rate (68% at $d=256$ ), meaning it naturally stops reasoning early. Qwen2.5 fills the entire budget (0% EOS rate), leading to forced over-generation and reasoning drift. This confirms the collapse is a reasoning quality effect, not an artifact of the two-phase generation protocol.

4. Significance and Contributions

Redefining Reasoning Budgets: The paper overturns the "more is better" heuristic for function-calling agents. It establishes that brevity (8–32 tokens) is the optimal strategy, while extended reasoning actively harms performance by inducing hallucinations and misrouting.
Structural Reliability (FR-CoT): The proposal of FR-CoT offers a practical, parameter-free solution that guarantees zero hallucination and high accuracy without the need for complex constrained decoding or fine-tuning.
Mechanistic Insight: It distinguishes between "format erosion" (a common assumption) and "reasoning misdirection" (the actual cause of failure). The study shows that long CoT causes models to lose their initial correct intuition.
System Design Implications: For developers building agent systems, the recommendation is to use a fixed, brief CoT budget (or FR-CoT) rather than large, open-ended reasoning windows. This saves compute costs (approx. 10% overhead vs. direct answer) while significantly boosting reliability.
Generalizability: The findings suggest that test-time compute scaling paradigms (successful in math reasoning) do not transfer directly to structured tool-use settings, where the action space is finite and the cost of hallucination is high.

Conclusion

The paper concludes that for function-calling agents, brief reasoning is superior to extended reasoning. The optimal strategy is a fixed budget of 8–32 tokens or the use of FR-CoT to structurally enforce function routing. This approach maximizes accuracy, minimizes hallucination, and avoids the severe performance collapse associated with "overthinking" in structured environments.