💬 NLP

Learning How to Use Tools, Not Just When: Pattern-Aware Tool-Integrated Reasoning

This paper proposes a pattern-aware tool-integrated reasoning framework that improves large reasoning models by explicitly learning both code execution and algorithmic encoding patterns before aligning pattern selection with teacher preferences, achieving significant accuracy gains on challenging math benchmarks like MATH500 and AIME24.

Original authors: Ningning Xu, Yuxuan Jiang, Shubhashis Roy Dipta, Hengyuan Zhang

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ningning Xu, Yuxuan Jiang, Shubhashis Roy Dipta, Hengyuan Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: It's Not Just About Calling the Doctor; It's About Which Doctor to Call

Imagine you have a very smart student (a Large Reasoning Model) who is trying to solve difficult math problems. This student has a superpower: they can ask a computer program (a tool) to do the heavy lifting for them, like a calculator or a coding assistant.

For a long time, researchers focused on teaching the student when to ask for help. They asked, "Should I solve this in my head, or should I call the computer?"

However, this paper points out a missing piece of the puzzle: How the student asks for help matters just as much as when they ask.

The authors discovered that the student often picks the wrong "style" of help, even if they know they need help. They identified two main styles (or "patterns") of using the tool:

The "Calculator" Pattern: The student treats the computer like a simple calculator. They write down the math problem and ask the computer to just crunch the numbers.
- Analogy: Asking a chef to just chop an onion for you. You give them the onion; they do the one specific task.
The "Algorithmic" Pattern: The student treats the computer like a programmer. They write a full script or a set of instructions to solve the whole problem from start to finish.
- Analogy: Asking a chef to build a whole restaurant kitchen, write a recipe book, and then cook a meal. You give them the goal; they build the system to achieve it.

The Problem: The Wrong Tool for the Job

The paper shows that students often mix these up, leading to failures even when their logic is perfect.

Example 1 (The Overflow): Imagine a problem asking for a massive calculation (like $1000!$ $1000!$ ). If the student uses the Calculator Pattern, they might try to feed the whole giant number into the computer at once. The computer's "brain" gets too full, and it crashes (an "overflow error").
- The Fix: The student should have used the Algorithmic Pattern, writing a loop to multiply the numbers step-by-step, which keeps the computer's memory safe.
Example 2 (The Infinite Loop): Imagine a problem asking to find a specific prime number hidden inside the digits of Pi. If the student uses the Algorithmic Pattern, they might try to write a program that checks every single digit of Pi at once. The computer runs out of memory or time because the list is too long.
- The Fix: The student should have used the Calculator Pattern, asking the computer to check just the first 10 digits, then the next 10, one by one.

The Takeaway: The failure wasn't that the student couldn't do the math; it was that they chose the wrong method of using the tool.

The Solution: A Two-Step Training Camp

To fix this, the authors created a two-stage training method to teach the student how to be a "Pattern-Aware" reasoner.

Stage 1: Learning Both Styles (Code Competence)
First, they teach the student how to be good at both styles. They show the student thousands of examples where they solve problems using the Calculator style, and thousands where they use the Algorithmic style.

Analogy: Before teaching a driver which route to take, you first make sure they know how to drive a manual transmission and an automatic transmission. They need to be fluent in both.

Stage 2: Learning to Choose (Pattern Preference)
Once the student knows how to drive both cars, the second stage teaches them which car to pick for which road. They use a "Teacher" (a very smart AI) to show the student: "For this specific problem, the Algorithmic style is the winner. For that one, the Calculator style is the winner."

Analogy: A driving instructor telling the student, "For this steep mountain road, use the manual gear. For this flat highway, use the automatic."

The Results: Smarter Choices, Better Scores

The paper tested this new method on tough math competitions (like MATH500 and AIME24).

Before: The student could use the tool, but they often crashed or got stuck because they picked the wrong style. Their success rate was around 26.7%.
After: By learning to match the problem to the right pattern, the success rate jumped to 50.0%.
Code Reliability: The ability to actually generate working code improved from 64% to 70.5%.

Summary

This paper argues that for AI to get really good at math, it's not enough to just know when to use a computer. The AI must also learn how to use it—knowing whether to treat the computer like a simple calculator or a complex programming engine. By teaching the AI to recognize the "shape" of the problem and pick the right tool-use style, they can turn failures into successes.

Technical Summary: Pattern-Aware Tool-Integrated Reasoning

Problem Definition

Tool-integrated reasoning (TIR) has emerged as a critical paradigm for enhancing Large Reasoning Models (LRMs) by enabling interaction with external verifiers, such as code interpreters, to produce executable and verifiable reasoning steps. While prior research has largely focused on determining when to invoke tools (e.g., timing, multi-round calls), this work identifies a significant gap in understanding how tools are applied once invoked.

The authors observe that models frequently suffer from pattern mismatch, where the strategy used to apply the tool is misaligned with the specific demands of the problem. The paper formalizes two distinct usage patterns:

Calculator-pattern: Using code strictly for direct computation or verification of intermediate arithmetic steps.
Algorithmic-pattern: Encoding the entire problem logic into a full program to solve it.

The paper demonstrates that failures often stem not from a lack of reasoning capability, but from selecting the wrong pattern. For instance, attempting to compute $1000!$ directly via a calculator-pattern leads to overflow errors, whereas an algorithmic approach involving algebraic simplification succeeds. Conversely, a symbolic algorithmic approach to finding a prime in $\pi$ may fail due to context limits, while a scanning (calculator-style) approach succeeds.

Methodology

To address pattern mismatch, the authors propose a two-stage learning framework:

1. Training Set Construction

The authors construct a dataset where each problem $x$ is paired with two distinct solutions:

Chosen Solution ( $y^+$ ): The solution using the pattern deemed most suitable by a teacher model.
Counter Solution ( $y^-$ ): The solution using the alternative pattern.
Both solutions follow a standardized output schema (reasoning $\to$ Python block $\to$ outputs $\to$ final answer) to ensure consistency.

2. Stage One: Code Competence Acquisition

The model undergoes Supervised Fine-Tuning (SFT) on the combined dataset of both patterns ( $D_{SFT} = \{(x_i, y_{alg}), (x_i, y_{calc})\}$ ).

Objective: Minimize negative log-likelihood over response tokens.
Goal: To ensure the model learns to generate reliable code for both patterns without bias toward a specific strategy, establishing a stable foundation for tool use.

3. Stage Two: Pattern Preference Alignment

Following SFT, the model undergoes Direct Preference Optimization (DPO) to learn the correct pattern selection strategy.

Mechanism: Using a frozen reference policy ( $\pi_{ref}$ , the SFT model), the model is trained to increase the likelihood of the "winner" (pattern-appropriate) solution relative to the "loser" (mismatched) solution.
Objective: Maximize the preference score $\sigma(\beta [\log \pi_\theta(y^+) - \log \pi_\theta(y^-) - (\log \pi_{ref}(y^+) - \log \pi_{ref}(y^-))])$ .
Goal: To align the model's pattern selection with teacher preferences, enabling it to adapt its strategy based on problem context.

Key Contributions

Formalization of Pattern Mismatch: The work identifies and formalizes the phenomenon where misaligned tool-use strategies derail otherwise sound reasoning, distinguishing between calculator and algorithmic patterns.
Two-Stage Framework: The proposal of a novel framework that first builds dual-pattern code competence via SFT and then refines strategy selection via DPO, avoiding the need for a separate reward model.
Empirical Validation: Demonstration that modeling how tools are used yields significant improvements in both code reliability and end-task accuracy on challenging mathematical benchmarks.

Experimental Results

The framework was evaluated on the OpenR1-Math-220k dataset for training and tested on out-of-domain benchmarks: MATH500, AIME24, and AMC23. The base model used was R1-Distill-Qwen-1.5B.

Key Performance Metrics:

Code@1 (Executable Code Generation):
- On MATH500, Code@1 improved from 64.0% (SFT only) to 70.5% (SFT + DPO).
- On AIME24, Code@1 improved from 26.7% to 50.0%.
- On GSM8K, Code@1 improved from 72.0% to 80.1%.
Pass@1 (Overall Accuracy):
- On AIME24, overall accuracy rose from 26.7% to 50.0%.
- On MATH500, accuracy increased from 58.2% to 63.2%.

The results indicate that while SFT is crucial for enabling the model to generate executable code, the addition of DPO significantly enhances the model's ability to select the correct strategy, thereby reducing failures caused by pattern mismatch.

Significance and Claims

The paper claims that the effectiveness of tool-integrated reasoning depends not merely on the ability to invoke tools, but on the strategic alignment of how those tools are applied. By explicitly modeling the choice between calculator and algorithmic patterns, the proposed approach offers a lightweight yet effective method to substantially improve reasoning accuracy. The authors assert that their pattern-aware approach transforms failure cases (caused by overflow or context limits) into successes by switching to the complementary pattern, validating that strategic adaptation is as critical as computational capability in LRMs.