Technical Summary: TACO – Tool-Augmented Credit Optimization for Agentic Tool Use

1. Problem Statement

Agentic multimodal models that "think with images" by writing and executing code (e.g., cropping, zooming, or transforming an image) represent a promising paradigm for fine-grained visual question answering. However, existing reinforcement learning from verifiable rewards (RLVR) approaches face a critical limitation: outcome-only rewards (based solely on the final answer) cannot distinguish between useful, redundant, or misleading tool calls.

• The Credit Assignment Gap: Standard RLVR assigns the final reward to the entire trajectory. Consequently, a helpful tool call may be under-rewarded if the final answer is wrong due to later errors, or a harmful call may be rewarded if the model happens to guess the correct answer.
• The Confounding Factor: Recent analyses indicate that accuracy gains in tool-use RL are often driven by the model's intrinsic improvement rather than the tool itself.
• Limitations of Existing Process Rewards: Step-wise rewards for text reasoning cannot isolate external tool observations, while process rewards defined on tool outputs often rely on expensive external judge models (e.g., GPT-4o) and fail to determine if a call actually changed the correctness of the answer.

The core challenge is to develop a self-supervised, judge-free signal that scores a tool call by its specific effect on final correctness and delivers credit only to the tokens responsible for that call.

2. Methodology: TACO

The authors propose Tool-Augmented Credit Optimization (TACO), a variant of Group Relative Policy Optimization (GRPO) designed for code-tool visual agents. TACO couples two distinct advantage channels to address the credit assignment problem:

2.1 Differential Answer-Probe Reward (DAPR)

DAPR is a self-supervised, judge-free reward that measures the marginal value of a tool call.

• Mechanism: The method inserts two lightweight "probes" into the reasoning trajectory:
  1. Pre-tool probe (Tool-off): After the initial reasoning ($T_1$) but before code execution, the model is prompted to generate an answer ($a_1$) based on the context $(q, I, T_1)$.
  2. Post-tool probe (Tool-on): After the tool executes and the model reasons over the new observation ($T_2$), the model generates a second answer ($a_2$) based on the full context including the tool output.
• Scoring: Both answers are scored using a rule-based answer checker ($r_{out} \in \{-1, 0, +1\}$). The tool-value reward is the difference: $\Delta = r_{out}(a_2) - r_{out}(a_1)$.
  • $\Delta > 0$: Useful call (turns wrong to right).
  • $\Delta < 0$: Misleading call (turns right to wrong).
  • $\Delta = 0$: Redundant or ineffective call.
• Robustness: By differencing, DAPR cancels out the model's pre-existing knowledge and baseline performance. This design inherently defends against "probe-hacking," where a model might prematurely write an answer into the reasoning trace to inflate a single probe score; since both probes read from the same pre-tool state, such leakage affects both equally, leaving $\Delta$ unchanged.

2.2 Outcome-Gated Advantage Routing (OGAR)

A scalar tool value is insufficient; the reward signal must be routed to the specific tokens responsible for the outcome. OGAR is a parameter-free, token-level rule that gates the final-answer advantage ($A_1$) based on the tool call's outcome ($\Delta$).

• Routing Logic:
  • Useful ($\Delta > 0$): The entire trajectory (pre-tool reasoning, code, and post-tool reasoning) receives the positive credit.
  • Misleading ($\Delta < 0$): The blame is assigned only to the tool branch (code and post-tool reasoning). The pre-tool reasoning ($T_1$) is protected from penalty, preventing the model from learning to distrust correct initial reasoning.
  • Right-but-Redundant ($\Delta = 0, a_1$ correct): The tool branch is denied credit (suppressed), teaching the model to abstain from unnecessary calls.
  • Necessary-but-Failed ($\Delta = 0, a_1$ wrong): The blame is withheld from the tool branch to encourage exploration on difficult items.
• Implementation: The final advantage $A_1$ is masked by a gate $m[t]$ that activates only for responsible segments. The process channel ($A_2$, derived from $\Delta$) is active only when $\Delta \ge 0$.

2.3 Training Pipeline

TACO is trained via a two-stage pipeline:

1. SFT Cold-Start: Supervised fine-tuning on trajectories interleaving reasoning, code, and tool outputs to establish the Think-Code-Answer format.
2. RL with Gated Advantages: GRPO optimization using the coupled objective $L_{TACO} = \alpha_1 L_{GRPO}(m \odot A_1) + \alpha_2 L_{GRPO}(g A_2)$, where $g$ is a trajectory-level gate disabling the process channel for misleading calls.

3. Key Contributions

• TACO Framework: A GRPO variant that integrates DAPR and OGAR to provide a tool-call-level learning signal without external judges.
• DAPR: A novel reward mechanism that scores tool calls via a before/after probe difference. It is judge-free, incurs near-zero API cost, and is robust to probe-hacking.
• OGAR: A parameter-free routing rule that assigns credit or blame strictly based on the outcome of the tool call, suppressing wasted calls without explicit cost terms.
• Probe-Hacking Defense: The paper diagnoses a failure mode in generative probes where models "leak" answers early. It demonstrates that the differential approach (DAPR) neutralizes this attack, whereas additive probe rewards are vulnerable.

4. Experimental Results

The authors evaluated TACO on 12 benchmarks across perception, reasoning, and general multimodal tasks, using a 7B parameter model (Qwen2.5-VL-7B).

• Performance: TACO achieved a macro-average accuracy of 68.1%, outperforming the best existing code-tool agent (PyVision-RL-7B at 63.7%) by 4.4 points. It surpassed other strong baselines like Thyme-7B, DeepEyes-7B, and CodeV-RL-7B by 5.6 to 8.1 points.
• Comparison to Proprietary Models: As a 7B model, TACO significantly outperformed the closed-source GPT-4o (68.1% vs. 58.5%).
• Efficiency: TACO was the most accurate and fastest agent. By learning to invoke tools only when necessary, it reduced end-to-end latency (e.g., 2.3s on V* vs. 3.6s for PyVision) while maintaining higher accuracy.
• Generalization: The method generalized across base models, improving Qwen3-VL-8B from 72.9% to 78.8%, indicating the gains stem from the credit-assignment mechanism rather than the base model.
• Ablation Studies: Removing DAPR dropped performance by 4.5 points (reopening probe-hacking vulnerabilities), and removing OGAR dropped performance by 2.0 points (blurring credit assignment).

5. Significance and Claims

The paper claims that TACO addresses the fundamental bottleneck in agentic tool use: learning which tool calls are worth making.

• Mechanism over Data: The authors assert that the performance gains are driven by the credit-assignment mechanism (DAPR + OGAR) rather than the base model or data, as evidenced by consistent improvements across different backbones.
• Self-Supervision: The work demonstrates that a high-quality, fine-grained reward signal for tool use can be derived entirely from the agent's own outcome rewards via differencing, eliminating the need for expensive external judges or auxiliary cost terms.
• Behavioral Shift: TACO successfully trains agents to "crop precisely where it helps rather than over-call," achieving a state where the agent abstains from tools when they do not contribute to correctness, thereby improving both accuracy and inference efficiency.

The authors remain modest regarding the scope, noting that the method relies on rule-based checkers (verifiable answers) and currently operates most cleanly in single-call regimes, leaving multi-call attribution and open-ended generation as future work.