Technical Summary: SciVQR

Problem Statement

Scientific reasoning is a cornerstone of human intelligence, requiring the integration of multimodal inputs, domain-specific expertise, and multi-step inference across diverse subjects. While Multimodal Large Language Models (MLLMs) have advanced significantly in natural language understanding and basic visual perception, they struggle with rigorous scientific reasoning tasks that demand deep, structured analysis of multimodal content. Existing benchmarks fail to adequately evaluate these capabilities due to three primary limitations:

1. Limited Disciplinary Coverage: Many benchmarks focus on a narrow set of subjects (e.g., only mathematics or physics) or restrict content to K-12 levels, failing to reflect the breadth of real-world scientific knowledge.
2. Lack of Difficulty Diversity: Current datasets often lack a spectrum of difficulty, ranging from basic factual recall to complex, graduate-level multi-hop reasoning, leading to performance saturation or insufficient challenge.
3. Absence of Solution Traceability: Most benchmarks evaluate only the final answer, ignoring the reasoning process. This prevents fine-grained analysis of reasoning fidelity, structure, and the identification of specific failure modes (e.g., hallucinations or missing steps).

Methodology

To address these gaps, the authors introduce SciVQR, a comprehensive multimodal benchmark designed for advanced scientific reasoning evaluation.

Data Curation

• Sources: The dataset comprises 3,254 questions sourced from high school, college, and graduate-level exams, textbooks, and academic competitions across multiple countries.
• Domains: It covers 6 core scientific subjects (Mathematics, Physics, Chemistry, Geography, Astronomy, and Biology) and 54 specialized subfields (e.g., Astrophysics, Organic Chemistry, Geomorphology).
• Multimodality: Questions include diverse scientific visual formats such as equations, charts, molecular structures, geological diagrams, and biological illustrations. Visual information is indispensable to solving the problems.
• Annotation & Quality Control:
  • Over 20 postgraduate annotators collected and processed the data, adhering to copyright regulations.
  • A rigorous multi-stage quality assurance process involved human experts and LLM assistance to filter low-resolution images, verify text via OCR (Mathpix), remove duplicates, and standardize formats.
  • Difficulty Levels: Questions are categorized into Easy, Medium, and Hard based on their original sources.
  • Solution Traces: A critical feature is that 46% of the questions include detailed, expert-authored solution traces. This enables the evaluation of reasoning processes, not just final answers.
  • Language: The dataset supports bilingual evaluation with 81.8% English and 18.2% Chinese questions.

Evaluation Framework

The authors evaluated a range of state-of-the-art MLLMs, including open-source models (e.g., Qwen2.5-VL, InternVL3, MiMo-VL) and proprietary models (e.g., GPT-4o, o1, o4-mini, Gemini-2.5-Flash).

• Metrics: Performance was measured by accuracy on multiple-choice and free-form questions.
• Reasoning Quality: For questions with solution traces, the authors employed a structured rubric to evaluate Chain-of-Thought (CoT) generation across five dimensions: Faithfulness (alignment with ground truth), Informativeness, Redundancy, Hallucination, and Missing Steps.
• Settings: Experiments were conducted in a zero-shot setting, comparing direct prompting against CoT prompting strategies.

Key Contributions

1. SciVQR Benchmark: The first multimodal benchmark to simultaneously offer broad disciplinary coverage (6 subjects, 54 subfields), a wide difficulty spectrum (high school to graduate level), and solution traceability for reasoning analysis.
2. Reasoning-Centric Evaluation: By including expert-authored solution traces for nearly half the dataset, SciVQR shifts the evaluation focus from answer correctness to the fidelity and structure of the reasoning process.
3. Comprehensive Empirical Analysis: The paper provides a detailed breakdown of model performance across subjects, difficulty levels, and languages, highlighting the specific strengths and weaknesses of current MLLMs in scientific domains.

Results

The evaluation of leading MLLMs on SciVQR reveals several critical findings:

• Performance Gap: Proprietary models generally outperform open-source models. The best proprietary models (o4-mini, Gemini-2.5-Flash) achieved an overall accuracy of 79.3%, while the top open-source model (Qwen2.5-VL-72B) reached 61.8%. However, reasoning-optimized open-source models like MiMo-VL-7B (69.2%) are beginning to close this gap.
• Importance of Reasoning Optimization: Models specifically optimized for reasoning (e.g., o1, o4-mini, MiMo-VL) significantly outperformed non-reasoning counterparts (e.g., GPT-4o). Reasoning-optimized models showed marked improvements across all subjects, particularly in Mathematics and Physics, where deep inference is required.
• Subject-Specific Challenges: Models performed significantly worse in Mathematics and Physics compared to Biology and Geography. This suggests that tasks requiring quantitative analysis and complex multi-step inference remain a bottleneck, whereas tasks relying more on textual knowledge and conceptual understanding are better handled.
• Impact of CoT: Chain-of-Thought prompting generally improved performance, especially for reasoning-optimized models. For instance, Qwen2.5-VL-72B saw a +7.9% overall gain with CoT. However, for some models, CoT did not yield benefits or even degraded performance, indicating limitations in instruction following or the generation of verbose, unstructured reasoning.
• Reasoning Quality: Reasoning-optimized models (e.g., o4-mini) generated more faithful, complete, and structured reasoning traces compared to instruction-tuned models. Scaling model size (e.g., Qwen2.5-VL from 7B to 72B) yielded only marginal improvements in reasoning quality, whereas reasoning optimization strategies provided substantial gains.
• Error Analysis: Quantitative and qualitative error analysis identified that reasoning errors are the primary failure mode, even when visual perception is accurate. Specifically, standard models frequently suffer from "Reasoning Premise Errors" (incorrect initial assumptions), while reasoning-optimized models tend to make "Reasoning Inference Errors" (logical inconsistencies within extended chains).

Significance

The authors posit that SciVQR represents a significant advancement in the evaluation of MLLMs by moving beyond simple answer verification to a deeper scrutiny of the reasoning process. The benchmark's unique design—combining broad disciplinary coverage with expert-verified solution traces—provides a clearer understanding of how models arrive at conclusions and where they fail.

The paper claims that achieving true multimodal scientific intelligence requires not just larger models, but deeper multi-step inference capabilities and better integration of interdisciplinary knowledge. The results underscore that optimizing for step-by-step reasoning is crucial for advancing MLLMs toward true scientific reasoning, and that current models, even the most advanced ones, still face substantial limitations in handling complex, multi-modal scientific tasks. SciVQR is intended to serve as a pivotal tool for driving the development of more transparent, structured, and sophisticated scientific reasoning in future AI systems.