Deconstructing Multimodal Mathematical Reasoning: Towards a Unified Perception-Alignment-Reasoning Paradigm

This paper systematically reviews recent advancements in Multimodal Mathematical Reasoning by proposing a unified Perception-Alignment-Reasoning paradigm, categorizing existing approaches around four fundamental questions regarding information extraction, representation, reasoning, and evaluation, while outlining future research challenges.

The Gist

Imagine you are trying to solve a complex math problem, but instead of just reading numbers on a page, you are looking at a messy construction site. You have a blueprint (the text), a pile of bricks and beams (the image), and a list of instructions (the question).

For a long time, AI models were like students who could only read the instructions but were terrible at looking at the construction site. They would guess the answer based on the words alone, often ignoring the fact that a beam was actually broken or a measurement was wrong.

This paper introduces a new way to teach AI how to solve these "multimodal" math problems (problems that mix pictures and words). The authors call their new system the PAR Framework, which stands for Perception, Alignment, and Reasoning.

Think of solving a math problem with a picture as a three-step relay race:

1. Perception: The "Eagle Eye"

The Problem: Current AI models often have "bad eyesight." They might look at a graph and think a line is going up when it's actually going down, or they might miss a tiny "x" in a diagram.
The Solution: This step is about training the AI to be a super-observant detective. It needs to look at the image and say, "Okay, I see a red circle here, a blue line there, and the numbers on the side go from 0 to 10."

• Analogy: Imagine you are a chef. Before you can cook, you must look at your ingredients. If you think you have salt but it's actually sugar, your dish will fail. "Perception" is the chef carefully checking every single ingredient to make sure they are real and correctly identified.

2. Alignment: The "Translator"

The Problem: Even if the AI sees the picture correctly, it doesn't know how to talk to its "math brain." The picture says "a long line," but the math brain needs to hear "length = 5." If they don't speak the same language, the AI gets confused.
The Solution: This step is about translation. The AI takes what it saw (the visual facts) and turns them into a language the math engine understands (like code, formulas, or logical steps).

• Analogy: Think of this like a diplomat at a peace treaty. One side speaks "Image," and the other speaks "Math." The diplomat (Alignment) has to take the Image side's request ("The bridge is too short") and translate it into the Math side's language ("We need to add 3 meters to the span"). If the translation is wrong, the treaty fails.

3. Reasoning: The "Engineer"

The Problem: Once the AI has the facts and the translation, it has to actually solve the problem. Old models often just guessed the answer or took a shortcut.
The Solution: This step forces the AI to build the solution step-by-step, checking its work as it goes. It's not just about getting the right number; it's about proving how it got there.

• Analogy: Imagine an architect building a skyscraper. They don't just wave a magic wand and hope the building stands. They lay one brick, check if it's level, lay the next, and check again. "Reasoning" is the architect double-checking every brick to ensure the tower doesn't fall over.

---

The New Rulebook: APE

The authors also realized that we were grading these AI students unfairly. We used to only check if their final answer was right (like a teacher just looking at the "A" or "F" on a test). But if the student got the right answer by guessing, they didn't really learn.

The paper proposes a new grading system called APE:

• Answer: Did you get the right number? (The traditional test).
• Process: Did you show your work? Did you make sense in the middle steps? (Did you use the right bricks?).
• Executable: Can we run your solution like a computer program to prove it works? (Can we actually build the tower?).

Why Does This Matter?

Right now, AI is getting really good at math, but it's still "hallucinating" (making things up) when pictures are involved. This paper provides a roadmap to fix that.

By breaking the problem down into Seeing (Perception), Translating (Alignment), and Building (Reasoning), and by grading the AI on its Process and Proof, we can build AI that doesn't just guess the answer, but actually understands the world around it.

In short: This paper is the instruction manual for teaching AI to stop guessing and start truly "seeing" and "thinking" about math problems that involve pictures.

Technical Summary

1. Problem Statement

Multimodal Mathematical Reasoning (MMR) involves solving mathematical problems that require the integration of textual information and visual modalities (e.g., diagrams, charts, tables, geometric figures). While Large Language Models (LLMs) have achieved state-of-the-art results in text-based mathematical reasoning, they face significant limitations in real-world multimodal tasks:

• Perception Failures: Models often misinterpret visual structures, failing to extract critical constraints like incidence, parallelism, or numeric scales.
• Alignment Errors: There is a frequent disconnect between visual evidence and symbolic mathematical expressions, leading to incorrect variable binding or logic.
• Inconsistent Reasoning: Models produce reasoning steps that are hallucinated or inconsistent with the visual input.
• Evaluation Limitations: Existing benchmarks primarily evaluate the final answer, failing to verify the correctness, faithfulness, or executability of intermediate reasoning steps. This masks whether a correct answer was derived via sound reasoning or lucky guessing.

2. Methodology: The PAR and APE Frameworks

The authors propose a systematic, process-centric framework to deconstruct MMR, moving beyond simple benchmark cataloging to analyze the internal mechanisms of Multimodal Large Language Models (MLLMs).

A. The Perception–Alignment–Reasoning (PAR) Framework

The paper decomposes the MMR workflow into three interdependent stages:

1. Perception (What to Extract?):

   • Goal: Extract structured, computation-relevant mathematical facts from multimodal inputs (Text $T$, Diagrams $D$, Charts/Tables $C$, Images $I$).
   • Output: A set of facts $F$ spanning three levels:
     • Low-level primitives: Points, lines, axes, objects.
     • Structural relations: Incidence, parallelism, row/column layouts.
     • Quantitative attributes: Lengths, angles, values, units.
   • Task Families: The framework categorizes tasks into Geometry, Chart/Table problems, and Visual Math Word Problems, noting the evolution from symbolic parsers to neural encoders and LMM-based pipelines.

2. Alignment (How to Represent & Align?):

   • Goal: Map perceived visual facts to symbolic or executable representations to ensure downstream reasoning is interpretable and verifiable.
   • Techniques:
     • Executable Intermediates: Converting visual content into formal languages (e.g., geometry description languages, SQL, code) for checkable execution.
     • Symbolic–Neural Hybrids: Combining neural perception with symbolic reasoning engines (e.g., theorem provers).
     • Cross-modal Frameworks: Using architectures (like BLIP-2, LLaVA) to stabilize vision-language coupling, often via "describe-then-reason" curricula.
     • Pre-training & Fine-tuning: Leveraging large-scale datasets (e.g., Geo170K, MultiMath-300K) to learn alignment priors.

3. Reasoning (How to Perform Reasoning?):

   • Goal: Execute stable and verifiable inference over aligned representations.
   • Paradigms:
     • Deliberate Chains: Chain-of-Thought (CoT), Tree of Thoughts (ToT), and Graph of Thoughts (GoT) to externalize intermediate steps.
     • RL-based Methods: Using Reinforcement Learning (e.g., GRPO, MCTS) with process reward models (PRMs) to optimize long-horizon decision sequences.
     • Tool-Augmented Reasoning: Delegating computation to external tools (calculators, symbolic solvers, code interpreters) to enforce formal correctness.
     • Process Feedback: Using verifiers to check intermediate steps (e.g., code execution, proof checking) rather than just the final output.

B. The Answer–Process–Executable (APE) Evaluation Hierarchy

To address the limitations of answer-only evaluation, the authors introduce a three-level evaluation hierarchy:

1. Answer Level: Standard accuracy metrics (Exact Match). Limitation: Cannot distinguish between correct reasoning and lucky guesses; conflates perception, alignment, and reasoning errors.
2. Process Level: Evaluates the faithfulness of intermediate steps. Checks if reasoning is grounded in visual evidence and if steps are logically valid (e.g., step-type scoring, error detection).
3. Executable Level: The highest standard of verification. Requires the reasoning to be expressed in a form that can be run or formally verified (e.g., executing code, running a theorem prover, checking SQL queries). This directly tests the alignment and reasoning correctness.

3. Key Contributions

• Unified Taxonomy: The paper provides the first unified framework (PAR) that organizes diverse MMR methods (from geometry solvers to chart QA) based on their functional role in the reasoning pipeline.
• Evaluation Standard: The introduction of the APE hierarchy shifts the focus from "getting the right answer" to "reasoning correctly," emphasizing process-level and executable-level verification.
• Comprehensive Survey: The authors systematically map over 50 benchmarks and datasets (summarized in Table 1 and Table A1) to specific PAR stages and APE levels, identifying gaps in current evaluation (e.g., a lack of executable benchmarks for chart reasoning).
• Diagnostic Insights: By analyzing methods through the PAR lens, the paper identifies specific failure modes, such as the brittleness of symbolic parsers under domain shifts and the hallucination risks in pure neural CoT approaches.

4. Results and Findings

While this is a survey/framework paper rather than an experimental study proposing a new model, the analysis yields several critical findings:

• Current Limitations: Most existing MLLMs rely heavily on "Answer-level" evaluation, which hides significant failures in perception and alignment. Models often achieve high accuracy on simple tasks but fail on complex, multi-step reasoning requiring strict visual grounding.
• Methodological Trends:
  • Perception: Moving from handcrafted rules to LMM-based pipelines, but fine-grained structural understanding remains a bottleneck.
  • Alignment: "Executable intermediates" (code/SQL) are the most robust method for ensuring correctness but are brittle to domain shifts. Hybrid approaches (Neural + Symbolic) show promise for balancing flexibility and rigor.
  • Reasoning: Reinforcement Learning (RL) and Tool-Augmented methods are emerging as the most effective ways to stabilize long reasoning chains, though they come with high computational costs.
• Benchmark Gaps: There is a scarcity of benchmarks that support Executable-level evaluation for non-geometry tasks (e.g., complex charts or scientific documents). Most current datasets focus on Answer or Process levels.

5. Significance and Future Directions

• Theoretical Impact: The PAR/APE framework provides a rigorous "lens" for researchers to diagnose where a model fails (perception vs. alignment vs. reasoning) rather than just that it fails. This is crucial for developing more interpretable and reliable AI.
• Practical Applications: The framework guides the development of systems for:
  • Education: Intelligent tutoring systems that provide process-aware feedback.
  • Accessibility: Translating visual math into speech/braille with executable checks for accuracy.
  • Professional Systems: AR/VR and engineering tools requiring verifiable design and calculation.
• Future Challenges:
  • Scalable Verification: Developing lightweight, generalizable executable interfaces for diverse modalities (beyond geometry).
  • Robustness: Improving model stability against visual perturbations and domain shifts.
  • Data Quality: Reducing data contamination and ensuring high-fidelity, process-annotated training data.

In conclusion, the paper argues that the future of Multimodal Mathematical Reasoning lies not just in larger models, but in structured perception, explicit alignment to executable forms, and rigorous process-level evaluation. The proposed frameworks serve as a roadmap for achieving reliable, verifiable, and interpretable multimodal intelligence.