Technical Summary: Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Tasks

Problem Statement

Multimodal Large Language Models (MLLMs) have demonstrated promising reasoning capabilities, yet their performance in specialized domains like STEM (Science, Technology, Engineering, and Mathematics) remains difficult to evaluate rigorously. Current benchmarks suffer from three primary limitations:

1. Modality Redundancy and Shortcuts: Many existing datasets allow models to bypass visual evidence by relying solely on textual descriptions, leading to "modality collapse" where visual grounding is weak.
2. Lack of Structured Visual Reasoning: Scientific figures, diagrams, and plots are underrepresented in training data, and models struggle to generate structured visuals with precise symbols and logical constraints.
3. Outcome-Oriented Evaluation: Existing protocols focus almost exclusively on final-answer accuracy. This overlooks the quality, faithfulness, and coherence of intermediate reasoning steps, making it difficult to distinguish between genuine cross-modal reasoning and superficial shortcuts.

Methodology

The authors introduce STEPSTEM, a benchmark and evaluation framework designed to enforce strict text-vision complementarity and evaluate reasoning at the step level.

1. Data Curation (STEPSTEM Benchmark)

• Scope: The dataset comprises 283 graduate-level problems across Engineering, Physics, Chemistry, Mathematics, Biology, and other domains.
• Complementarity Enforcement: Problems are curated such that solutions cannot be derived from text alone or vision alone. This includes "strongly dependent" problems (requiring geometric/spatial reasoning) and "reasoning-simplifying" problems (where visuals reduce semantic ambiguity).
• Multi-Solution Trajectories: Each problem is annotated with one or more ground-truth solution trajectories. These trajectories are interleaved sequences of textual reasoning and visual evidence (images/diagrams).
• Visual Annotation: For visual reasoning steps, experts annotate relevant local evidence using bounding boxes (BBoxes) to enable fine-grained supervision for image alignment.
• Task Diversity: The benchmark includes 14 drawing tasks where the final output is an image, extending evaluation from "image-text to text" to "image-text to image."

2. Evaluation Framework

The authors propose a step-level evaluation protocol that moves beyond final-answer accuracy:

• Local Alignment Score:
  • Textual Steps: Uses attention-rollout-based semantic similarity (via Qwen3.5-9B) followed by monotonic dynamic programming to align predicted steps with reference steps, allowing for skipped trivial steps but preserving temporal order.
  • Visual Steps: Compares BBox-anchored reference features with sliding-window features from predicted images. A diversity penalty is applied to prevent a single generic image from satisfying multiple distinct visual steps.
• Global Logic Coverage: An LLM-based judge assesses whether the full predicted trajectory covers all critical textual propositions and visual evidence required by the reference solution, rewarding semantic entailment over verbatim matching.
• Metric Fusion: The local and global scores are fused using a weighted Root Mean Square (RMS). When multiple reference solutions exist, the framework reports the maximum fused score to avoid penalizing valid alternative reasoning paths.
• Final Answer Evaluation: Textual and visual final answers are evaluated separately for correctness.

Key Contributions

1. STEPSTEM Benchmark: A carefully curated dataset of 283 graduate-level STEM problems that enforce strict text-vision complementarity to eliminate unimodal shortcuts. It includes multi-solution trajectories with interleaved text and visual steps.
2. Fine-Grained Evaluation Framework: A novel step-level evaluation protocol that assesses intermediate multimodal reasoning steps (local alignment and global coverage) rather than just final outcomes. It utilizes dynamic programming and diversity penalties to handle interleaved image-text reasoning.
3. Comprehensive Empirical Analysis: An evaluation of representative MLLMs (including unified, interleaved generation, and understanding-only models) revealing significant gaps in current capabilities. The study demonstrates that step-level scores correlate strongly with final-answer accuracy but provide a more diagnostic view of model behavior.

Experimental Results

Experiments were conducted on 269 text-answer problems and 14 image-answer problems across various model categories:

• Overall Difficulty: Current MLLMs struggle significantly with graduate-level multimodal STEM reasoning. The best-performing understanding-only models (Gemini 3.1 Pro, Claude Opus 4.6) achieved only 38.29% accuracy on text-answer problems. The best unified model (Gemini 2.5 Flash Image) achieved 50.80% on image-answer problems.
• Modality Limitations:
  • Understanding-only models (e.g., Claude Opus 4.6) achieved high scores on text-related process metrics (Local-T, Global-T) but scored 0% on all image-related metrics (Img. Acc., Local-I, Global-I), confirming they cannot generate visual intermediate steps.
  • Unified models (e.g., GPT-5.4 + GPT-Image) achieved the highest fused process scores (71.52) and strong image generation capabilities but still exhibited limited final-answer accuracy (33.21% for text answers), suggesting they can generate plausible visual structures but struggle to integrate them effectively for solving complex problems.
  • Interleaved generation models (e.g., Janus-Pro, OmniGen2) showed non-trivial ability to produce intermediate visual content but failed to produce correct image-based final answers (0% accuracy).
• Process vs. Outcome: Step-level scores were strongly associated with final-answer correctness. Trajectories leading to correct answers consistently received higher fused process scores. However, the step-level evaluation revealed that some models could reach correct answers with poor reasoning trajectories, while others followed coherent paths but failed on the final step.
• Multi-Reference Utility: The use of multiple ground-truth solutions was shown to be essential. Single-reference evaluation could unfairly penalize valid but structurally different reasoning paths. The multi-reference approach improved evaluation stability and fairness.

Significance and Claims

The paper claims that STEPSTEM positions itself as a benchmark for the fine-grained evaluation of multimodal reasoning. Its significance lies in:

• Exposing Modality Gaps: It highlights that even state-of-the-art models rely heavily on textual reasoning and lack genuine cross-modal STEM reasoning capabilities.
• Shifting Evaluation Paradigm: It argues that final-answer accuracy is insufficient for complex multimodal tasks. The proposed step-level framework provides a more informative diagnostic signal, distinguishing between models that occasionally guess correctly and those that follow a plausible, grounded reasoning process.
• Future Research Direction: The authors posit that STEPSTEM will support future research in multimodal process supervision, reasoning-oriented data construction, and the development of more faithful cross-modal reasoning systems.

The paper concludes that while current models show promise, there is "substantial headroom" for improvement in genuine cross-modal STEM reasoning, and that evaluating the reasoning process itself is critical for advancing the field.