Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs

1. Problem Statement

Multimodal Large Language Models (MLLMs) are designed to integrate vision and language into a shared representation space. However, a critical question remains: Does this joint representation enable consistent reasoning across modalities?

The authors identify a phenomenon termed Cross-Modal Inconsistency, where an MLLM produces different answers to semantically identical prompts depending on the input modality (text-only, image-only, or mixed).

• The Gap: While models can often read text from images (OCR), they may not "reason" about that information as effectively as when receiving native text.
• The Risk: This suggests that model performance might be driven by the format of the input rather than the content, leading to unreliable behavior in real-world applications where information is presented in various formats.
• Limitations of Prior Work: Existing benchmarks either evaluate single models, fail to control for Optical Character Recognition (OCR) errors (conflating recognition failure with reasoning failure), or rely on data that models may have seen during pre-training.

2. Methodology: REST and REST+ Benchmarks

To address these gaps, the authors introduce two systematic benchmarks: REST (Render-Equivalence Stress Tests) and REST+.

A. Benchmark Design

The core design principle is presenting semantically identical information in three formats:

1. Text: Native text input.
2. Image: The same text rendered as an image.
3. Mixed: A combination (e.g., question in text, context in image, or vice versa).

Datasets Used:

• SOEBENCH: A newly generated dataset of systems of linear equations with symbolic variables (A–E) and digits (0–9). It ensures zero prior exposure (no data contamination) and minimal OCR complexity (simple symbols).
• Established Benchmarks: MMLU, ARC, and GSM-Symbolic, where text is rendered as images.

Experimental Controls:

• OCR Verification: Before evaluating reasoning, models must transcribe the image text. Only questions with perfect OCR are included in the consistency analysis to isolate reasoning capabilities from recognition failures.
• REST+ (Visual Variations): To test robustness, REST+ creates 10 permutations of each image, varying:
  • Resolution: 50, 100, and 200 DPI.
  • Font: DejaVu Sans, Courier New, and Cursive.
  • Color: Black vs. various colors (Red, Green, Blue, etc.).

B. Evaluation Metrics

1. Render-Equivalence Rate (RER): The percentage of questions where the model outputs the same answer across all three modalities. (Higher is better).
2. Cross-Modality Failure Rate (CFR): The percentage of questions solvable in at least one modality but not all. (Lower is better).
3. Max Modal Coverage (MMC): The percentage of questions solvable in at least one modality, indicating the model's total potential if the optimal format is chosen.

C. Internal Representation Analysis (RQ4)

The authors analyze the cosine similarity of internal embeddings between matching text and image inputs across model layers. They hypothesize that higher similarity in the joint embedding space correlates with higher cross-modal consistency.

3. Key Contributions

1. New Benchmarks (REST/REST+): The first systematic evaluation of cross-modal consistency that strictly controls for OCR difficulty and data contamination.
2. Comprehensive Evaluation: Evaluation of 15 state-of-the-art MLLMs (including GPT-5-mini, Claude Haiku 4.5, Qwen, Gemma, InternVL, etc.).
3. Mechanistic Insight: A correlation analysis showing that cross-modal inconsistency is linked to the modality gap (the distance between text and image embeddings in the joint space).
4. Visual Characteristic Analysis: Detailed study on how resolution, font, and color impact reasoning, revealing that models are sensitive to visual rendering even when text is legible.

4. Key Results

A. Universal Inconsistency

• No model is consistent: None of the 15 evaluated MLLMs achieve perfect cross-modal consistency.
• Wide Variance: RER scores range from 6.6% (DeepSeek-Tiny) to 90.7% (GPT-5-mini).
• Text Preference: Almost all models perform significantly better on the text modality than on image or mixed modalities. For example, GPT-4o-mini achieves >7% higher accuracy on text than image for MMLU.
• Impact of Modality: For low-consistency models (e.g., Phi-4), nearly 50% of solvable questions yield incorrect answers if the input is not in the "optimal" modality.

B. OCR is Not the Sole Culprit (RQ2)

• Even when controlling for perfect OCR (using SOEBENCH), inconsistency persists.
• OCR-First Strategy: Instructing models to transcribe text before solving (OCR-first) does not consistently improve performance and sometimes degrades it, proving that the issue is reasoning, not just recognition.

C. Visual Characteristics (RQ3)

• Resolution: Lower resolutions (50 DPI) significantly hurt performance for some models (e.g., Claude, Gemma), but high-performing models (InternVL3-14B) remain robust.
• Font: Font type (Serif vs. Sans vs. Cursive) has little impact on reasoning consistency.
• Color: Surprisingly, colored text (especially red and yellow) often improves image accuracy compared to black text.
• Token Efficiency: Models generally require more visual tokens than text tokens to achieve the same accuracy. Only Qwen2.5-VL-32B showed higher efficiency with vision tokens.

D. Representation Correlation (RQ4)

• There is a positive correlation between the cosine similarity of text and image embeddings and the RER score.
• Models with higher consistency have internal representations for the same concept that are closer together in the embedding space. This suggests the "modality gap" is a mechanistic cause of inconsistency.

5. Significance and Implications

• Reliability Concern: The findings challenge the assumption that MLLMs have seamlessly integrated vision and language. The "modality gap" means models may fail critical tasks simply because the information is presented as an image rather than text.
• Benchmarking Shift: Future evaluations must control for OCR and test consistency across modalities, not just raw accuracy on a single format.
• Architecture Insights: The correlation between embedding similarity and consistency suggests that improving the alignment of text and image representations (reducing the modality gap) is a viable path to more robust MLLMs.
• Efficiency vs. Performance: While compressing text into images (as proposed by DeepSeek-OCR) saves tokens, this paper shows it may come at the cost of reasoning quality, as models reason less effectively on visual tokens.

In conclusion, the paper establishes that current frontier MLLMs are fundamentally inconsistent when reasoning across modalities, a flaw driven by the structural separation of text and image representations rather than mere recognition errors.