💬 NLP

VisTIRA: Closing the Image-Text Modality Gap in Visual Math Reasoning via Structured Tool Integration

The paper introduces VisTIRA, a tool-integrated reasoning framework that addresses the performance gap between text and image-based mathematical reasoning in vision-language models by decomposing visual problems into natural language rationales and executable Python steps, supported by a new synthetic dataset and training pipeline.

Original authors: Saeed Khaki, Ashudeep Singh, Nima Safaei, Kamal Ginotra

Published 2026-03-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Saeed Khaki, Ashudeep Singh, Nima Safaei, Kamal Ginotra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a brilliant student who is amazing at solving math problems when they are written out in plain text on a computer screen. You can read the numbers, understand the words, and crunch the equations perfectly.

But now, imagine someone hands you a photo of a handwritten math homework sheet. Suddenly, you start making mistakes. You misread a squiggly "3" as an "8," you confuse a plus sign for a multiplication sign, or you get lost in the messy layout of the page. Even though the math is the same, the way it's presented (the photo) trips you up.

This is the problem the paper VisTIRA is trying to fix. It's about closing the gap between how AI handles text versus how it handles images of math.

Here is the breakdown of their solution, using some everyday analogies:

1. The Problem: The "Photo vs. Text" Gap

Current AI models (Vision-Language Models) are like students who are great at reading a textbook but terrible at reading a messy whiteboard.

Text Mode: The AI reads "x squared plus 5 equals 10." Easy.
Image Mode: The AI sees a photo of a handwritten "x² + 5 = 10." It might think the "2" is a "z," or it might miss that the "5" is actually a "S" from a bad scan.
The Result: The AI gets the wrong answer, not because it can't do math, but because it can't read the picture clearly.

2. The Solution: VisTIRA (The "Calculator-Checking" Student)

The authors created a new framework called VisTIRA. Think of VisTIRA not as a student trying to memorize the answer, but as a student who has a strict rule: "I will never trust my own brain for the heavy lifting; I will use a calculator."

Here is how VisTIRA works, step-by-step:

Look and Think: The AI looks at the math problem image. It says, "Okay, I think this is a geometry problem involving a triangle."
Write the Recipe: Instead of trying to solve it in its head, it writes a set of instructions (Python code) to solve it. "First, calculate the area of the base. Then, multiply by the height."
Run the Calculator: It sends those instructions to an external computer program (a calculator) to actually do the math.
Check the Result: The computer gives back the answer. The AI checks if that answer makes sense.
Repeat: If the answer looks weird, it goes back to step 2, fixes its instructions, and tries again.

The Analogy: Imagine you are trying to navigate a city.

Old AI: Tries to memorize the map and drive without GPS. If the map is blurry (the image), it gets lost.
VisTIRA: Looks at the blurry map, writes down the turn-by-turn directions, and then uses a GPS (the code) to drive the car. Even if the map is messy, the GPS ensures the car arrives at the right destination.

3. The Training: Teaching the AI to "Do Homework"

To teach VisTIRA how to do this, the researchers didn't just show it answers. They created a massive library of "Homework Solutions" (called SnapAsk).

They took real-world photos of math homework.
They used a super-smart AI (a "Teacher") to solve them by writing out the thought process and the code to solve them.
They filtered out any solutions where the "thought process" didn't match the "code result."
They used this high-quality data to train the smaller AI models, teaching them to always double-check their work with code.

4. The "Magic Glasses" (OCR)

The paper also tested a trick called OCR (Optical Character Recognition).

The Idea: Before the AI tries to solve the image, it uses a special tool to "read" the text in the picture and turn it into plain text first.
The Result: This acts like giving the AI a pair of magic glasses that make the blurry handwriting crystal clear.
The Catch: This works wonders for smaller, less powerful AI models (they need the glasses). But for the giant, super-smart AI models, the glasses sometimes just add extra noise or confusion, because those giants are already pretty good at reading blurry text on their own.

5. The Big Discovery

The researchers found two main things:

The Gap is Real: AI is significantly worse at math when it's an image compared to text.
The Fix: The best way to fix this isn't just making the AI "smarter." It's teaching the AI to slow down, write down its plan, and use a tool (code) to verify the math.

Summary

VisTIRA is like teaching a student to stop guessing and start using a calculator. By forcing the AI to break a messy image problem into clear steps and then using code to solve the numbers, it stops making silly reading errors and starts getting the right answers, even when the math looks like a messy photo on a piece of paper.

They also released a huge new dataset of math problems converted into images so other researchers can test their own "magic glasses" and "calculator-checking" systems.

1. Problem Statement: The Image–Text Modality Gap

The paper identifies a critical performance discrepancy in Vision-Language Models (VLMs): they significantly underperform on mathematical reasoning tasks when problems are presented as images compared to when they are presented as plain text.

The Phenomenon: Even state-of-the-art models show a "modality gap," where accuracy drops when a math problem is visually typeset (containing dense formulas, multi-line equations, diagrams, and mixed symbolic-textual context) versus when it is provided as text.
Root Causes: The gap arises from compounded failures in:
1. Visual Parsing: Misinterpreting layout-level structures (e.g., fraction baselines, integral bounds, superscripts, piecewise braces).
2. Symbolic Reasoning: Errors in reading the parsed text cascade into incorrect algebraic manipulation.
Limitations of Current Approaches: Existing VLMs often rely on Chain-of-Thought (CoT) prompting but lack structured tool integration. Furthermore, current datasets rarely provide step-by-step executable trajectories paired with images, limiting the ability to train models to verify their own reasoning.

2. Methodology

The authors propose VisTIRA (Vision and Tool-Integrated Reasoning Agent), a framework designed to bridge this gap through structured reasoning and tool use.

A. The VisTIRA Framework

VisTIRA transforms a visual math problem into an iterative loop of Natural Language Rationales and Executable Python Code.

Process:
1. The model receives an image $I$ .
2. It generates a reasoning step ( $\rho_i$ ) and a corresponding Python code snippet ( $A_i$ ).
3. The code is executed externally (using tools like SymPy/Python), producing an output ( $O_i$ ).
4. The output is fed back into the model to inform the next reasoning step.
5. The loop continues until a final answer is boxed ( $\boxed{\text{answer}}$ ).
Goal: This "closed-loop" design defers exact arithmetic to a symbolic engine, reducing hallucinations and compounding errors common in pure text-based reasoning.

B. Data Construction & Supervision

To train VisTIRA, the authors constructed two key datasets:

SnapAsk (Real-World Data): A proprietary dataset of ~303k real-world homework-style images.
- Generation: A "Teacher" VLM (e.g., GPT-4o/GPT-5) generates tool-integrated trajectories (Rationale $\to$ Code $\to$ Execution $\to$ Output).
- Filtering: Trajectories are retained only if the model's CoT reasoning matches the final answer produced by the code execution (consistency filter). This yielded ~148k high-quality training samples.
NuminaMath (Controlled Benchmark): A pipeline to convert existing text-only CoT math corpora (NuminaMath) into visually typeset images using LaTeX rendering.
- This creates paired text-image datasets, allowing for controlled evaluation of the modality gap.
- The authors released 360k clean and 360k noisy (stochastic degradation applied) rendered images.

C. OCR Grounding Analysis

The paper investigates OCR grounding as a complementary strategy.

Method: Extracting text from math images using a state-of-the-art OCR model (DeepSeek-OCR) and feeding it alongside the image.
Hypothesis: Explicit text extraction can compensate for weaker visual encoders in smaller models.

3. Key Contributions

VisTIRA Framework: An iterative, tool-integrated reasoning agent that decomposes visual math problems into rationale-code-execution loops.
Large-Scale Trajectory Supervision: The creation of a verified corpus of ~148k tool-integrated trajectories from real-world images (SnapAsk) for Supervised Fine-Tuning (SFT).
Modality Gap Benchmarking:
- A LaTeX-based pipeline to convert text math problems into images.
- Release of 360k clean and 360k noisy rendered NuminaMath images.
- A standardized 5k test set with DeepSeek-OCR transcriptions.
Diagnostic Analysis of OCR: A detailed study showing that OCR grounding significantly helps smaller models but offers diminishing returns or redundancy for larger models.

4. Experimental Results

Experiments were conducted on Qwen2.5-VL-7B (smaller model) and GPT-5 (large model) using the SnapAsk and rendered NuminaMath datasets.

A. Tool-Integrated Reasoning Effectiveness

Performance Gain: Fine-tuning on VisTIRA trajectories significantly improved performance over instruction-only baselines.
- SnapAsk (Real-world): Qwen2.5-VL-7B improved from 32.53% (Base) to 37.96% (VisTIRA).
- NuminaMath (Rendered): Improved from 58.77% to 60.97%.
Ablation: A "CoT-only" SFT (without code execution) provided minimal gains (+1.65% on SnapAsk), confirming that the tool-integrated supervision (interleaving code and execution) is the primary driver of improvement, not just domain adaptation.

B. Quantifying the Modality Gap

Text vs. Image: Both models performed better on text than images.
- Qwen2.5-VL-7B: Text (64.19%) vs. Image (58.77%) $\to$ 5.42% gap.
- GPT-5: Text (79.94%) vs. Image (73.94%) $\to$ 6.00% gap.
Scale Correlation: The gap is inversely correlated with model size; smaller models suffer more from visual parsing errors.

C. Impact of OCR Grounding

Small Models (Qwen2.5): OCR grounding provided substantial gains.
- Image + OCR: 63.89% (vs. 58.77% for Image-only).
- Insight: OCR helps smaller models overcome weak visual perception of dense notation.
Large Models (GPT-5): OCR grounding showed marginal gains or slight redundancy.
- Image + OCR: 76.37% (vs. 73.94%).
- Insight: Stronger models already parse visual text well; adding raw OCR text can introduce noise or redundancy.
Length Analysis: Problems where OCR helped significantly contained longer, more complex text, confirming that smaller models struggle with dense notation in images alone.

D. Robustness (Noisy Images)

When tested on noisy rendered images (simulating real-world photos with blur, noise, and artifacts), the modality gap persisted or widened for large models, while smaller models remained stable (hitting a "visual parsing ceiling"). This confirms the gap is structural, not just an artifact of image quality.

5. Significance and Conclusion

Closing the Gap: The paper demonstrates that structured reasoning (VisTIRA) and complementary grounding (OCR) are effective strategies to mitigate the image-text modality gap.
Complementary Strategies:
- Tool Integration is essential for scaling performance across all model sizes by offloading computation.
- OCR Grounding acts as a compensatory signal for smaller models with weaker visual encoders but is less critical for larger models.
Community Impact: By releasing the SnapAsk trajectories and the NuminaMath rendered dataset (clean and noisy), the authors provide the community with the necessary resources to further research visual mathematical reasoning and modality alignment.
Future Work: The authors suggest that future research should focus on adaptive fusion mechanisms to dynamically decide when to use OCR based on model size and problem complexity, rather than static concatenation.

In summary, VisTIRA proves that equipping VLMs with the ability to "think and compute" via external tools, supported by high-quality trajectory supervision, is the most effective path forward for solving complex visual math problems.