R2IF: Aligning Reasoning with Decisions via Composite Rewards for Interpretable LLM Function Calling

1. Problem Statement

Large Language Models (LLMs) are increasingly used for function calling (invoking external tools via APIs) to access real-time data and solve complex tasks. However, current Reinforcement Learning (RL) approaches for function calling suffer from a critical misalignment:

• Outcome-Driven vs. Process-Aware: Existing RL methods primarily reward the final correctness of the tool call (e.g., whether the API arguments match the ground truth) but ignore the quality of the reasoning process (Chain-of-Thought, or CoT) that leads to that decision.
• Post-hoc Rationalization: Models often generate reasoning that merely justifies a correct answer after the fact, rather than reasoning that actively guides the selection of tools and the construction of parameters.
• Parameter Sensitivity: Function calling requires strict adherence to tool specifications (e.g., specific formats like "City, State" or default values). Standard RL often fails to penalize reasoning that ignores these constraints, leading to "correct" tool names but malformed arguments.
• Lack of Interpretability: Without grounded reasoning, error diagnosis is difficult, and the model's reliability in unseen scenarios is compromised.

2. Methodology: R2IF Framework

The authors propose R2IF (Reasoning-aware RL for Interpretable Function calling), a framework that aligns the reasoning process with executable tool-call decisions. The core innovation is a Composite Reward Function optimized via Grouped Proximal Policy Optimization (GRPO).

The total reward $R$ is defined as:
$$R = R_{binary} + R_{cer} + R_{smv}$$

A. Binary Reward ($R_{binary}$)

This acts as a hard gate to ensure structural validity and correctness.

• Format Validity ($R_{format}$): Enforces strict output structure (e.g., exactly one <reason> block followed by one <tool> block).
• Correctness ($R_{correctness}$): Checks if the predicted tool call (name and arguments) exactly matches the ground truth.
• Weighting: This component is weighted heavily (factor of 3) to prevent the model from drifting into "soft" rewards that ignore basic syntax.

B. Chain-of-Thought Effectiveness Reward ($R_{cer}$)

This component measures whether the generated reasoning actually helps the model make the correct decision, rather than just accompanying it.

• Mechanism: A "faithful student model" (pre-trained on the same data) samples multiple continuations based on the generated reasoning prefix.
• Calculation: It calculates the success rate of the student model conditioned on the reasoning prefix ($v(R|Q,T)$) and subtracts the baseline success rate without that reasoning ($v_{base}$).
• Goal: $R_{cer} = v(R|Q,T) - v_{base}$. A positive reward implies the reasoning provided actionable information that improved the tool-call success rate.

C. Specification-Modification-Value Reward ($R_{smv}$)

This is a fine-grained reward designed to enforce parameter-level alignment with tool specifications. It evaluates three pillars for each argument:

1. Specification ($r_{spec}$): Does the reasoning correctly identify the tool's constraints (e.g., "needs City, State")?
2. Modification ($r_{mod}$): Does the reasoning describe necessary transformations from the user query to the valid argument (e.g., inferring "CA" for "San Francisco")?
3. Value ($r_{val}$): Is the final argument value correctly instantiated in the tool call?

• Implementation: It uses semantic similarity between the reasoning snippet and ground-truth specification/modification documents, combined with a hard check on whether the parameter exists in the final output.

3. Key Contributions

1. Hybrid Reward Design: The first framework to jointly optimize reasoning quality and final function-call correctness, explicitly addressing the misalignment between CoT and tool decisions in RL.
2. Distribution-Based CER: Introduces a reward that quantifies reasoning utility without relying on subjective human scoring, using a student model to verify if reasoning improves decision-making.
3. Parameter-Level SMV: A novel reward that supervises the process of argument construction (Specification, Modification, Value), ensuring the model understands tool constraints rather than just guessing values.
4. Interpretability Definition: Formalizes interpretability as the alignment between reasoning and executable decisions, grounded in parameter specification and value instantiation.

4. Experimental Results

The method was evaluated on BFCL (Berkeley Function Calling Leaderboard) and ACEBench using various backbones (Qwen2.5 series and Llama3.2-3B).

• Performance Gains:
  • BFCL: R2IF outperformed baselines by up to 34.62% on Llama3.2-3B (Overall accuracy).
  • ACEBench: Achieved top overall accuracy across all models, with a 15.4% gain on Qwen2.5-7B.
  • Robustness: Maintained high accuracy (>96%) on "Irrelevance" tasks (rejecting tools when appropriate), proving it does not over-optimize for tool invocation at the cost of safety.
• Reasoning Quality (ACE Metric):
  • Baselines often showed negative Average CoT Effectiveness (ACE), meaning their reasoning was detrimental or irrelevant.
  • R2IF achieved positive ACE (e.g., +0.05 for Llama3.2-3B on BFCL), proving the reasoning actively supports the tool call.
• Ablation Studies:
  • Removing SFT warm-start significantly hurt smaller models (3B), highlighting the need for a stable output template before RL.
  • Removing SMV degraded performance on "Live" (real-world) scenarios, confirming its role in handling parameter constraints.
  • Removing CER reduced performance on complex, single-turn tasks, showing its necessity for generalization.
• Scalability: Performance improved monotonically with model size (1.5B to 7B) and generalized well across different model families (Qwen and Llama).

5. Significance

• Reliability: R2IF moves beyond "black box" tool calling by ensuring that the model's internal logic (reasoning) is consistent with its external actions (API calls).
• Debuggability: By enforcing that reasoning must explicitly address parameter specifications and modifications, the framework makes it easier to diagnose why a model failed (e.g., did it miss a format constraint?).
• Real-World Applicability: The method's success on "Live" benchmarks (real user queries) and its ability to handle distribution shifts make it a strong candidate for deploying reliable, tool-augmented LLMs in production.
• Paradigm Shift: It challenges the notion that outcome-based rewards are sufficient for complex tasks, demonstrating that process supervision is critical for structured, high-stakes interactions like function calling.

Limitations

The authors note that the current evaluation is limited to single-turn tasks. The framework's ability to maintain coherent reasoning across multi-turn conversations and complex interactive workflows remains untested. Additionally, the reward design is highly tailored to specific function-calling constraints, which may require adaptation for other types of reasoning tasks.