🤖 machine learning

Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Use

The paper introduces CARL, a competence-aware reinforcement learning framework that assigns segment-level credit to tool-use decisions by decomposing model rollouts at natural boundaries, enabling LLMs to autonomously learn when to rely on parametric knowledge versus external tools and significantly improving accuracy while reducing unnecessary tool calls, particularly in smaller models.

Original authors: Abhijit Kumar, Zoey Wu, Mohit Suley

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abhijit Kumar, Zoey Wu, Mohit Suley

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Over-Confident" Student

Imagine a brilliant student (the AI) who knows a lot of facts from memory. Sometimes, they need a calculator or a search engine to solve a hard math problem or find a specific fact.

The problem is that this student doesn't know when to use these tools.

If you ask them "What is 2 + 2?", they might pull out a calculator anyway, wasting time.
If you ask them "What is the capital of a country I've never heard of?", they might try to guess from memory and get it wrong, instead of looking it up.

Current AI training methods are like a teacher who only gives a grade at the very end of the exam.

Scenario A: The student guesses the answer, gets it right, and gets an "A." The teacher doesn't know if the student guessed or actually knew the answer.
Scenario B: The student uses a calculator for a simple math problem, gets it right, and gets an "A." The teacher doesn't realize the student wasted time on an easy question.
Scenario C: The student uses a calculator for a hard problem, gets it wrong, and gets an "F." The teacher doesn't know if the student used the calculator poorly, or if the calculator gave bad info, or if the student just didn't know the math.

Because the teacher only sees the final grade, the student learns to use tools all the time (just to be safe) or never (because they might mess it up). They never learn the boundary of what they know versus what they need help with.

The Solution: CARL (The "Smart Coach")

The paper introduces CARL (Competence-Aware Reinforcement Learning). Instead of waiting for the final grade, CARL acts like a smart coach who watches the student's steps in real-time.

1. Breaking the Exam into "Chunks"

The coach breaks the student's thinking process into three distinct "chunks" (segments) where the state of the conversation clearly changes:

The Ask (Invoke): The student decides to use a tool and types a search query.
The Read (Assimilate): The student reads the tool's answer and summarizes it.
The Answer (Commit): The student writes the final answer.

2. The "Credit" System

Instead of giving one grade for the whole exam, the coach gives a tiny "credit score" to each chunk based on how much it helped the final result.

Good Ask: If the student asks a great question that leads to the right answer, that chunk gets a positive credit.
Bad Ask: If the student asks a confusing question that leads to a dead end, that chunk gets a negative credit (even if they eventually fixed it later).
Unnecessary Ask: If the student asks for help on a question they already knew the answer to, the coach says, "You didn't need that!" and gives zero or negative credit for wasting time.

This is the paper's main innovation: Segment-Level Credit Assignment. It isolates exactly which part of the process helped and which part hurt, even if the final answer was correct.

How It Works (The "Critic" Coach)

To do this, CARL trains a special "Critic" model. Think of the Critic as a coach who has watched thousands of practice runs.

The Warm-Up: Before the real training starts, the Critic watches the student answer questions without tools and with tools forced. It learns to spot: "Oh, this student knows the answer to this one, but they are clueless on that one."
The Real Training: As the student practices, the Critic predicts: "If the student continues down this path, what are the odds they get it right?"
- If the student asks a good question, the Critic's prediction of success goes up. The student gets credit.
- If the student asks a bad question, the prediction goes down. The student gets penalized.
- If the student asks a question they didn't need to, the prediction stays flat. The student learns: "Don't bother asking."

The Results: Smarter, Faster, and More Honest

The paper tested this on 7 billion and 3 billion parameter models (think of these as "smart" and "very smart" students).

They Learned to Stop Guessing: The models became much better at knowing when they didn't know the answer. They stopped guessing confidently and started asking for help when it was actually needed.
They Stopped Wasting Time: On easy questions, the models stopped using the calculator or search engine. They saved a lot of time (tokens) and money (API costs).
Better Accuracy: Because they stopped wasting time on easy questions and focused their tools on hard ones, they got more questions right overall.
The "Small Model" Surprise: The paper found that smaller models benefited the most. A smaller model (3B) improved its score by 1.4 times more than the larger model (7B).
- Analogy: A smaller student has a smaller memory bank. Knowing exactly when to open the library book (tool) is a superpower for them. A larger student already has so much in their head that they don't need the library as often, so the improvement is less dramatic.

Summary

The paper teaches AI models to be humble. Instead of blindly using tools or guessing, they learn to recognize the edge of their own knowledge.

Old Way: "I'll use the search engine for everything just in case."
New Way (CARL): "I know this answer, so I'll just say it. But for that other question, I definitely need to look it up."

This makes AI faster, cheaper to run, and more accurate, especially for smaller, more efficient models.

Technical Summary: CARL (Competence-Aware Reinforcement Learning)

1. Problem Statement

Large Language Models (LLMs) equipped with tools (e.g., search engines, calculators) often fail to distinguish between tasks that require external assistance and those solvable via parametric knowledge. Humans naturally recognize the boundaries of their own knowledge (e.g., knowing $2+2$ does not require a calculator, while $347 \times 28$ does), but LLMs typically do not.

Existing approaches face two primary limitations:

Prompt-based methods (e.g., ReAct, RAG): These instruct models to use tools at inference time but do not train the model to recognize when tools are necessary. Consequently, models often invoke tools unnecessarily, increasing latency and cost without improving accuracy.
Trajectory-level Reinforcement Learning (RL): Current RL methods (e.g., Search-R1) assign a single outcome reward (e.g., Exact Match) to an entire trajectory. This creates a credit assignment problem:
- Indiscriminate Reinforcement: In a successful trajectory, both helpful tool calls and unnecessary or erroneous calls receive positive reinforcement.
- Lack of Selectivity: The model cannot learn to skip tools on easy questions because the final success signal is shared across all segments.
- External Dependency: Step-level credit methods often rely on external judges (e.g., GPT-4) whose knowledge may diverge from the student model's, or require expensive step-level annotations.

2. Methodology: CARL

The authors propose CARL (Competence-Aware Reinforcement Learning), a framework that trains a critic to learn the model's own competence boundary, enabling fine-grained credit assignment without external judges.

2.1 Structural Decomposition

CARL leverages the fact that tool-use trajectories possess structurally observable boundaries that pure reasoning trajectories lack. A trajectory is decomposed into segments based on these boundaries:

Invoke: The model generates reasoning and a code block (e.g., search(query)). The environment executes the code, producing raw output (stdout).
Assimilate: The model reads the raw output and generates a <context> block, distilling relevant information. The raw output is replaced by this extraction in the context.
Commit: The model generates the final answer.

This decomposition creates a Semi-Markov Decision Process (SMDP) where each segment is an "option."

2.2 Segment-Level Advantage

Instead of assigning a single scalar reward to the whole trajectory, CARL derives per-segment advantages from a single binary terminal reward ( $R \in \{0, 1\}$ ) using the SMDP Bellman equation.

For a trajectory with segments $k=0 \dots N$ , the advantage for segment $k$ is:

Intermediate Segments: $A(seg_k) = V_\phi(s_{k+1}) - V_\phi(s_k)$
Final Segment: $A(seg_N) = R - V_\phi(s_N)$

Here, $V_\phi(s)$ is the value function (critic) estimating the probability of success given the current context.

Mechanism: A "good" search increases the estimated probability of success ( $V$ rises), yielding a positive advantage. A "bad" search or a failed extraction decreases $V$ , yielding a negative advantage. An unnecessary search on a question the model already knows leaves $V$ unchanged, yielding near-zero advantage.
Key Distinction: This allows opposing signs within the same trajectory (e.g., a positive advantage for a correct query followed by a negative advantage for a poor extraction), a capability absent in trajectory-level methods.

2.3 Critic Training and Warm-up

The critic $V_\phi$ is a two-layer MLP attached to the shared LLM backbone.

Training Target: The critic is trained via Monte Carlo (MC) targets to minimize the Mean Squared Error (MSE) between $V_\phi(s)$ and the terminal reward $R$ .
Warm-up Phase: To prevent early random gradients from corrupting the policy, a curated warm-up phase is essential. The critic is trained on base-model rollouts labeled by final correctness. Questions are classified into:
- Tier 1: Beyond parametric competence (model fails without tools).
- Tier 2: Within parametric competence (model succeeds without tools).
- Trajectories are generated under both "forced-tool" and "no-tool" conditions to teach the critic to distinguish between necessary and unnecessary tool use.
No External Judges: The critic learns solely from the model's own rollouts and binary outcomes, ensuring the competence boundary reflects the specific model's capabilities.

2.4 Policy Optimization

The policy is updated using PPO (Proximal Policy Optimization) at the segment level. Each segment receives its own advantage scalar, applied uniformly to all tokens within that segment. The total loss combines the PPO objective and the critic's MSE loss.

3. Key Contributions

Competence-Aware Value Function: A value function trained on binary outcomes that learns to separate parametrically solvable questions from tool-dependent ones. At 7B scale, it achieves an AUC of 0.93 in distinguishing these tiers.
Per-Segment Credit from Single Outcome: By utilizing the SMDP Bellman equation at structurally observable boundaries, CARL derives independent advantages for each segment (invoke, assimilate, commit) from a single binary reward. This enables the model to receive positive credit for helpful calls and negative credit for errors or unnecessary calls within the same successful trajectory.
Emergent Selectivity: The method enables models to learn when to ask for help, reducing unnecessary tool calls without explicit instruction or external supervision.

4. Experimental Results

The authors evaluated CARL on five benchmarks (GSM8K, HotpotQA, 2WikiMQA, FinQA, Musique) using Qwen2.5 models at 3B and 7B scales, comparing against strong RL baselines (Search-R1 PPO and GRPO).

Accuracy Gains: CARL outperforms the best RL baseline (Search-R1 PPO) by 6.7 Exact Match (EM) points at 7B and 9.7 EM at 3B on average across search-based datasets.
- The largest gains occur on Musique, a compositional multi-hop benchmark, with improvements of +8.3 EM (7B) and +9.0 EM (3B).
- Gains are disproportionately larger for smaller models (3B improvement is 1.4× the 7B improvement), suggesting that knowing when to ask is critical for models with limited parametric memory.
Tool-Use Efficiency:
- On questions answerable from parametric knowledge (Tier 2), CARL issues 53% fewer tool calls than Search-R1 PPO.
- Despite calling tools less often, CARL remains ~10 EM points more accurate on these same questions.
- Total token generation is reduced by 23% on average at 7B, driven by skipping retrieval on easy questions.
Critic Calibration: The trained critic accurately estimates the probability of success ( $V(s_0)$ $V (s_{0})$ ).
- AUC 0.93 (7B) and 0.85 (3B) for separating Tier 1 vs. Tier 2 questions.
- Calibration: With curated warm-up, the Expected Calibration Error (ECE) is 0.043, meaning the predicted success probability closely matches actual success rates.
Sign Divergence: The critic correctly assigns opposing signs to segments within a single trajectory (e.g., $+0.15$ for a good search, $-0.22$ for a bad extraction), a behavior impossible for trajectory-level methods.

5. Significance and Claims

The paper claims that CARL addresses the fundamental limitation of trajectory-level RL: the inability to isolate which specific tool call contributed to success or failure. By exploiting the structurally observable boundaries of tool-use trajectories, CARL transforms a single binary outcome into a sequence of meaningful, segment-level advantages.

The authors emphasize that this approach allows the model to learn its own competence boundary rather than being instructed when to use tools. This results in:

Reduced Hallucination: The model is less likely to fabricate answers on difficult questions (Tier 1) because it learns to route them to tools rather than guessing.
Efficiency: Unnecessary tool calls are suppressed, reducing latency and API costs.
Scalability: The benefits are most pronounced for smaller models, suggesting that segment-level credit assignment is a practical alternative to scaling parametric memory for handling tail-distribution knowledge.

The paper concludes that while retrieval quality remains a ceiling for performance (as shown by ablation studies where better search APIs further improved results), the credit-assignment mechanism of CARL is the primary driver for improved selectivity and accuracy in tool-augmented LLMs.