RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation

1. Problem Statement

Current Large Vision-Language Models (LVLMs) excel in multimodal reasoning but are evaluated primarily through multiple-choice or short-answer formats. These traditional benchmarks suffer from two critical flaws:

• "Right Answers for Wrong Reasons": They fail to distinguish between logically valid reasoning and coincidentally correct conclusions derived from flawed logic.
• Oversimplified Evaluation: Existing methods that do assess reasoning processes often use linear frameworks that ignore the non-linear, hierarchical nature of real-world multimodal reasoning. They also fail to account for complex intermodal relationships (how visual and textual clues interact, conflict, or support each other).

2. Methodology

The authors propose a comprehensive framework consisting of a new evaluation metric (RPTS) and a dedicated benchmark (RPTS-Eval).

A. Reasoning Process Tree Score (RPTS)

RPTS is a tree-structured metric designed to evaluate the faithfulness of the reasoning process, not just the final answer.

• Structure: The model's reasoning is parsed into a tree where leaf nodes represent atomic evidence (visual clues, textual clues) and non-leaf nodes represent intermediate conclusions.
• Scoring Mechanism:
  • Step-wise Evaluation: An LLM (specifically GPT-4) scores individual reasoning steps based on logical coherence and semantic alignment with ground truth, rather than scoring the entire chain at once.
  • Weighted Aggregation: The final score is a weighted average of step scores. The weight $w_i$ of a node $n_i$ is determined by its height $h$ in the tree relative to a focused step $h_f$ and a decay factor $\lambda$:
    $$w_i = \lambda^{|h_f - h|}$$
  • Hyperparameters:
    • $h_f$: The step height to focus on (e.g., focusing on the first inference step).
    • $\lambda$: Controls the decay speed of weights.
  • Error Localization: By adjusting these parameters, RPTS can pinpoint exactly where in the reasoning chain the model fails (e.g., early visual extraction vs. later logical deduction).

B. RPTS-Eval Benchmark

To validate RPTS, the authors constructed a new dataset comprising 390 reasoning instances across 374 images.

• Data Construction: A hybrid human-AI approach where GPT-4 generates story inspirations, and human annotators design rigorous reasoning paths, visual/textual clues, and ground truths.
• Multimodal Interaction Types: The benchmark categorizes instances into three distinct relationship types to test different reasoning dynamics:
  1. Guided: One modality explicitly or implicitly directs the retrieval of information from the other.
  2. Adversarial: One modality interferes with or contradicts the other (requiring the model to resolve conflicts).
  3. Independent: Modalities are unrelated and must be processed separately.
• Capabilities: Covers six skills: Recognition, OCR, Commonsense, Math, Image Comparison, and Spatial Awareness.

3. Key Contributions

1. RPTS Metric: A novel, tree-structured scoring system that detects "right answers for wrong reasons" and enables granular error localization within the reasoning chain.
2. RPTS-Eval Benchmark: A high-quality dataset with reliable atomic evidence annotations and diverse intermodal relationship types, addressing the lack of rigorous reasoning evaluation in current datasets.
3. Intermodal Classification: The formal definition of three relationship types (Guided, Adversarial, Independent) to better classify and understand multimodal reasoning challenges.
4. Comprehensive Analysis: Extensive experiments revealing significant gaps in current LVLMs, particularly regarding image-based inference and cross-lingual transfer.

4. Experimental Results

The authors evaluated 15 models (including GPT-4o, Llava-Next, InternVL2, Qwen-VL) on RPTS-Eval.

• Accuracy vs. Faithfulness: When filtering out cases where models arrived at correct answers via flawed reasoning (RPTS < 0.5), accuracy dropped significantly for most models.
  • GPT-4o showed the highest robustness (smallest drop), indicating superior logical consistency.
  • Open-source models (e.g., Llava-Next, InternVL2) suffered larger accuracy drops, revealing a lack of logical robustness.
• Step Analysis:
  • Most models performed poorly at the first inference step ($h_f=1$), struggling to derive necessary information directly from visual clues.
  • Visual vs. Textual: Models consistently scored lower when drawing conclusions from visual clues compared to textual clues, highlighting a weakness in image processing for inference.
• Language Disparity: Significant performance gaps were observed between English and Chinese contexts. Models generally performed better in English, suggesting training methodologies struggle to transfer multimodal reasoning capabilities across languages.
• Model Size: Larger models (e.g., InternVL2-76B) showed better performance but still exhibited limitations in complex adversarial scenarios compared to GPT-4o.

5. Significance

• Faithful Evaluation: RPTS shifts the focus from "answer correctness" to "reasoning faithfulness," providing a more rigorous standard for evaluating AI safety and reliability in high-stakes domains (e.g., legal analysis, medical diagnosis).
• Diagnostic Tool: The ability to localize errors to specific steps in a reasoning tree allows researchers to pinpoint specific failure modes (e.g., poor visual grounding vs. logical deduction errors).
• Future Research Direction: The benchmark highlights that current open-source LVLMs need significant improvement in visual grounding and cross-lingual reasoning capabilities. It sets a new standard for developing models that can handle non-linear, conflicting multimodal evidence.