Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs

This paper systematically diagnoses the performance gap between text and image inputs in multimodal LLMs, revealing that visual text primarily amplifies reading errors rather than reasoning failures, and proposes a self-distillation method that effectively bridges this gap by training models on their own text-based reasoning traces paired with image inputs.

The Gist

Here is an explanation of the paper "Reading, Not Thinking," using simple language and creative analogies.

The Big Idea: The "Pixel vs. Letter" Problem

Imagine you have a brilliant student (the AI) who is amazing at reading books. If you hand them a page of text, they can solve complex math problems, write code, and answer science questions instantly.

But, if you take that exact same page of text, print it out, take a photo of it, and hand the photo to the student, they suddenly act confused. They might make silly math mistakes, forget how to format their answers, or stop thinking step-by-step.

This paper investigates why this happens. The authors call this the "Modality Gap." It's the gap between how well an AI understands text when it sees it as letters (digital code) versus when it sees it as pixels (a picture).

---

The Investigation: What's Going Wrong?

The researchers tested seven different AI models on seven different types of tasks (like math, science, and coding). They found that the problem isn't that the AI is "dumb" when looking at pictures. The problem is actually two specific things:

1. The "Bad Font" Confusion (Reading Errors)

When AI models are trained, they mostly see text that looks like standard computer fonts (like Arial or Times New Roman).

• The Analogy: Imagine you are used to reading clean, printed newspapers. If someone hands you a note written in messy handwriting or a weird, distorted font, you might misread a "3" as an "8" or miss a minus sign.
• The Finding: The AI gets tripped up by the visual details. If the font is weird, the resolution is low, or the colors are inverted, the AI misreads the numbers and symbols. This causes calculation errors.
• The Twist: When the researchers tested the AI on real-world photos (like a screenshot of a Wikipedia page or a PDF from a real journal), the AI did surprisingly well! It was only the fake, synthetic images (made by computers for testing) that caused the AI to fail. The AI is actually quite good at reading real documents; it just hates bad test conditions.

2. The "Brain Freeze" (Reasoning Collapse)

This was the most surprising discovery.

• The Analogy: When you ask a human to solve a hard math problem, they usually say, "Let me think... first I do this, then I do that..." (this is called Chain-of-Thought).
• The Finding: When the AI sees text as an image, it often skips the "thinking" part. It tries to guess the answer immediately without showing its work. It's like a student who, when given a photo of a test, just blurts out a guess instead of writing down the steps.
• The Result: Because they skip the steps, they make more mistakes. The AI didn't lose its ability to think; it just lost the habit of showing its work when looking at pictures.

---

The Solution: Teaching the AI to "Read" Again

The researchers didn't want to rebuild the AI from scratch. Instead, they used a clever trick called Self-Distillation.

• The Analogy: Imagine the AI is a teacher and a student.
  1. First, the AI reads the problem as text and writes out a perfect, step-by-step solution (the "Teacher" part).
  2. Then, they show the photo of that same problem to the AI (the "Student" part).
  3. They tell the Student: "Look at this photo, but you must write the exact same step-by-step solution that the Teacher wrote for the text version."

By training the AI to copy its own smart reasoning from the text version onto the image version, they bridged the gap.

• The Result: On a tough math test (GSM8K), the AI's score jumped from 30% (when looking at images) to 92% (after the training). It learned to stop guessing and start thinking, even when looking at pixels.

---

Key Takeaways for Everyone

1. It's not the AI's fault: The AI isn't bad at "seeing." It's just that the way we test it (using weird fonts and low-res images) confuses it.
2. Real life is easier: AI models are actually quite good at reading real-world documents (like PDFs and screenshots). The "gap" is mostly an artifact of how researchers build their tests.
3. Don't judge a book by its cover (or pixels): When text becomes a picture, the AI stops "thinking" step-by-step. We need to train it to keep its reasoning habits alive, even when the text is just a picture.
4. The Fix is Simple: You don't need a super-complex new AI. You just need to teach the AI to trust its own reasoning, even when the input changes from letters to pixels.

In short: The AI isn't blind; it just needs to be reminded to slow down and think, even when it's looking at a picture instead of a text file.

Technical Summary

Here is a detailed technical summary of the paper "Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs."

1. Problem Statement

Multimodal Large Language Models (MLLMs) often exhibit a significant performance drop, termed the "modality gap," when processing textual content presented as images (pixels) compared to when the same content is provided as native text tokens. While humans read text through vision, MLLMs process text and images through separate pathways, compressing visual input into tokens before fusing them with textual representations.

Existing research has documented this gap but lacks a systematic diagnosis of its causes. Furthermore, prior evaluations rely heavily on synthetically rendered text images, which may conflate rendering artifacts (e.g., specific fonts, low resolution) with genuine perceptual deficits. The paper aims to:

1. Diagnose the root causes of the modality gap.
2. Determine if the gap is universal or dependent on task/data types.
3. Propose a method to bridge this gap with minimal architectural changes.

2. Methodology

Evaluation Setup

The authors evaluated 7 MLLMs (including Qwen2.5/3-VL, InternVL3/3.5, Pixtral, and GPT-5.2) across 7 benchmarks covering math, coding, knowledge, and reasoning.

• Datasets:
  • Synthetic: MMLU, ARC, GPQA, GSM8K, HumanEval (rendered as images).
  • Natural: QASPER (arXiv PDFs) and SQuAD (Wikipedia screenshots).
• Input Modalities: To isolate the source of errors, five input modes were tested:
  1. Pure Text: Native tokens.
  2. Pure Image: Content rendered as an image.
  3. Instr.+Image: Text instructions + Image content.
  4. OCR-1P (One Pass): Model extracts text and solves the task in a single pass.
  5. OCR-2P (Two Pass): Model extracts text (OCR), then the extracted text is fed back as a Pure Text query to solve the task. This separates "reading" from "reasoning."

Error Analysis

A grounded-theory error analysis was conducted on over 4,000 error cases. Using a human-in-the-loop approach with LLM agents for coding, errors were classified into categories (e.g., calculation, reasoning, format, factual recall) to determine which capabilities degrade under visual input.

Proposed Solution: Self-Distillation

Motivated by the findings, the authors propose a self-distillation method:

• Concept: Train the model to reproduce its own high-quality text-mode reasoning traces (Chain-of-Thought) when given image inputs.
• Implementation: The model is fine-tuned using LoRA on pairs of (Image Input, Text-Mode CoT Output).
• Variants: Tested strategies including adapting the Vision Encoder (ViT), the Language Model (LM), or both.

3. Key Findings & Results

A. The Gap is Task- and Data-Dependent

• Synthetic vs. Natural: The gap is massive on synthetic renderings (e.g., math tasks on GSM8K dropped by >60 points for some models) but disappears or reverses on natural document images (arXiv/Wikipedia). In natural settings, MLLMs often match or exceed text-mode performance, suggesting the gap is partly a distributional mismatch between synthetic evaluation data and pretraining data.
• Rendering Artifacts: Rendering choices are critical confounds.
  • Font: Switching fonts alone caused accuracy swings of up to 47 percentage points. Handwriting fonts caused the largest degradation.
  • Resolution: Performance drops sharply below certain resolution thresholds, though some models (e.g., InternVL3.5 with Visual Resolution Router) are more robust.

B. Diagnosis: "Reading, Not Thinking"

The error taxonomy revealed that image mode selectively amplifies perceptual errors while leaving cognitive capabilities largely intact:

• Reading Errors: Calculation/Math errors increased by 1.5× (from 11.1% to 16.7%), and format errors rose significantly. These stem from misreading digits or operators from pixels.
• Reasoning Errors: Conceptual, factual, and logical reasoning errors remained largely unchanged between text and image modes.
• Chain-of-Thought (CoT) Collapse: A critical finding is that models produce significantly shorter outputs in image mode, often bypassing step-by-step reasoning entirely. For example, Qwen3-VL-8B reduced output length from 618 chars (text) to 32 chars (image), jumping directly to incorrect answers.

C. Effectiveness of Self-Distillation

The proposed self-distillation method successfully bridged the gap:

• Performance: On GSM8K, image-mode accuracy for Qwen3-VL-8B rose from 30.71% to 92.72% (nearly matching text-mode performance of 93.56%).
• Mechanism: Adapting the Language Model (LM) was the primary driver of improvement, though ViT adaptation also contributed.
• Generalization: The model trained on GSM8K generalized to unseen benchmarks (ARC, MMLU, HumanEval) without catastrophic forgetting, narrowing the text-image gap across all tasks.
• Teacher Signal: Using unfiltered traces (including those with wrong answers) worked nearly as well as filtered traces, indicating that the structure of the reasoning is more important than the correctness of the final answer for the distillation signal.

4. Significance and Contributions

1. Systematic Diagnosis: The paper moves beyond simply reporting the gap to explaining why it exists. It identifies that the gap is driven by reading failures (OCR/perception) and reasoning suppression (CoT collapse), not a lack of world knowledge or logical ability.
2. Benchmark Critique: It highlights that synthetic renderings are poor proxies for real-world visual text understanding. The choice of font, resolution, and rendering style drastically alters evaluation outcomes, urging the community to report these specifications.
3. Practical Remedy: The self-distillation approach offers a lightweight, effective solution to close the modality gap without retraining from scratch or redesigning architectures. It demonstrates that MLLMs can learn to "read" pixels as effectively as tokens if guided by their own reasoning traces.
4. Implications for Future Work: The findings suggest that future MLLMs should focus on improving visual feature alignment and encouraging robust reasoning chains under visual input, rather than assuming a fundamental inability to process text as images.

In summary, the paper concludes that MLLMs are capable of "thinking" well with visual text, but they struggle to "read" the pixels accurately and often fail to engage their reasoning mechanisms when the input is visual. These issues can be effectively mitigated through targeted self-distillation.