🤖 AI

SeePhys Pro: Diagnosing Modality Transfer and Blind-Training Effects in Multimodal RLVR for Physics Reasoning

The paper introduces SeePhys Pro, a benchmark revealing that current multimodal models struggle with modality transfer from text to images, and demonstrates that apparent gains from multimodal RLVR training often stem from residual textual cues rather than genuine visual reasoning.

Original authors: Kun Xiang, Terry Jingchen Zhang, Zirong Liu, Bokai Zhou, Yueling Tang, Junjie Yu, Jiacong Lu, Shangrui Huang, Heng Li, Likui Zhang, Kunkun Liu, Changzheng Zhang, Yangle Fang, Boqiang Guo, Hui-Ling Zhe

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kun Xiang, Terry Jingchen Zhang, Zirong Liu, Bokai Zhou, Yueling Tang, Junjie Yu, Jiacong Lu, Shangrui Huang, Heng Li, Likui Zhang, Kunkun Liu, Changzheng Zhang, Yangle Fang, Boqiang Guo, Hui-Ling Zhen, Dandan Tu, Yinya Huang, Xiaodan Liang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a student how to solve a physics problem. You have two ways to give them the information:

The Textbook Method: You write out the problem in words, describing the setup, the numbers, and the rules.
The Diagram Method: You draw a picture of the setup, label the parts, and write the numbers directly on the drawing.

Intuitively, you'd expect a smart student to solve the problem equally well whether you give them the words or the picture, as long as the information is the same. But this paper, SEEPHYS PRO, discovered that for current AI models, this isn't true. In fact, they get significantly worse when you switch from words to pictures.

Here is a breakdown of what the researchers found, using simple analogies.

1. The "Same Story, Different Format" Test

The researchers created a special test called SEEPHYS PRO. Think of it like a "translation challenge" for AI.

For every single physics problem, they created four versions that tell the exact same story but change how the information is presented:

Level 1 (Text Only): "A block weighs 5kg and is on a ramp." (All info is in words).
Level 2 (Structure in Image): They draw the ramp and the block, but the text still says "5kg." (The shape is a picture, the numbers are words).
Level 3 (Variables in Image): They draw the ramp, the block, and write "5kg" right on the block in the picture. (The shape and numbers are now in the image).
Level 4 (Full Picture): The entire problem, including the instructions and numbers, is handwritten on a piece of paper and scanned as one image.

The Discovery:
When the AI models tried these tests, their performance dropped like a stone as the information moved from text to images.

The "Reading the Label" Bottleneck: The biggest drop in performance happened at Level 3. This is when the AI had to look at the picture and "ground" the variables (e.g., "Okay, that squiggly line is a wire, and the number '12' next to it means 12 Volts").
The Analogy: Imagine a chef who can perfectly follow a written recipe (Level 1). If you hand them a photo of the ingredients with labels (Level 3), they suddenly forget how to cook. They can see the picture, but they can't connect the label "2 cups of flour" to the actual bag of flour in the photo. They get confused by the visual binding.

2. The "Blindfolded Training" Experiment

The researchers then asked a second question: If we train these AIs using Reinforcement Learning (RL) on thousands of physics problems with pictures, will they get better at reading the pictures?

To test this, they set up a "Blind Training" experiment.

Normal Training: The AI sees the problem text and the picture.
Blind Training: The AI sees the problem text, but the picture is completely blacked out (masked). It's like training a student to solve physics problems while wearing a blindfold, but the teacher keeps giving them the same "correct answer" rewards.

The Shocking Result:
Even though the AI was trained with no pictures at all (just black screens), it still got better at solving the unmasked tests where pictures were present.

The Analogy: Imagine a student who practices for a driving test while sitting in a car with the windows painted black. They never see the road. Yet, when they finally take the real test with windows open, they drive better.
Why? The researchers found the AI wasn't learning to "see" the road. Instead, it was learning shortcuts. It memorized patterns in the text, the style of the questions, or the typical answers. It learned to guess the right answer based on the "vibe" of the text, not by actually analyzing the visual diagram.

3. The "Magic Trick" of Accuracy

The paper concludes that when we see an AI's score go up, we shouldn't just cheer. We need to ask: Did they actually learn to see, or did they just get better at guessing based on text clues?

The "Blind Gain": The AI improved its score even when the training images were useless (black). This proves that the improvement came from the text and the patterns of the dataset, not from visual understanding.
The Gap Remains: Even after training, the AI still struggled more with the picture-heavy versions (Level 3 and 4) than the text-only versions. The "modality gap" (the difference between text performance and image performance) didn't close.

Summary

The Problem: Current AI models are "text-heavy." They are great at reading physics problems but terrible at "seeing" them, especially when they have to read numbers and labels directly off a diagram.
The Trap: Standard training methods (Reinforcement Learning) can make the AI look smarter by improving its text-based guessing skills, even if it never actually learns to interpret the visual evidence.
The Lesson: To truly make AI "see," we can't just measure if it gets the right answer. We have to test if it gets the right answer specifically because it looked at the picture, and not just because it recognized the text pattern.

The paper essentially says: Don't be fooled by high scores. If an AI can solve a problem without looking at the picture, it hasn't truly learned to see.

Technical Summary: SEEPHYS PRO

Problem Statement

Multimodal AI systems face a critical challenge in modality consistency: the ability to preserve identical reasoning behaviors when equivalent information is presented in different forms (e.g., text versus diagrams). Current benchmarks often evaluate models on a single input format, obscuring whether improvements in final-answer accuracy stem from genuine, representation-invariant reasoning or from shortcuts specific to a modality. This gap is particularly acute in physics, where diagrams often define the physical system's structure and variable bindings rather than merely illustrating text. Existing evaluations struggle to distinguish between failures in diagram perception, variable reading, physical abstraction, and downstream symbolic reasoning.

Methodology

1. SEEPHYS PRO Benchmark

The authors introduce SEEPHYS PRO, a fine-grained modality-transfer benchmark grounded in the principle of "same physics, different representation."

Design: Each seed physics problem is transformed into four semantically aligned variants that progressively shift task-critical information from language to vision:
- Level 1 (L1): Text-only (all structure, variables, and quantities in language).
- Level 2 (L2): Structure-in-image (physical topology/relations in the image; variables remain in text).
- Level 3 (L3): Structure + Variables-in-image (diagram includes labels and numerical values).
- Level 4 (L4): Fully rendered image (problem statement, formulas, and diagram converted into a single visual input, including handwritten text).
Data Construction: Derived from over 5,000 PDF pages across public datasets, textbooks, Olympiads, and exams. The dataset contains 1,000 seed questions (4,000 total variants) covering six physics disciplines (Mechanics, Electromagnetism, Optics, Thermodynamics, Waves, Modern Physics).
Metrics: The benchmark defines specific metrics to decompose performance:
- Transfer Gaps: $\Delta S$ (L1→L2 structural transfer), $\Delta V$ (L2→L3 variable grounding), $\Delta R$ (L3→L4 rendering), and $\Delta T$ (Total gap).
- Consistency: The percentage of problems solved correctly across all four levels simultaneously.

2. Training-Time Diagnostics (Blind Training)

To investigate whether Reinforcement Learning with Verifiable Rewards (RLVR) genuinely improves visual grounding or merely exploits statistical shortcuts, the authors developed two large-scale physics RL corpora (PhysRL-40K and PhysRL-8K).

Blind Training Control: A negative control setting where all training images are masked (replaced with black images), rendering the training instances visually unsolvable.
Comparison: Models are trained via standard RL (with images) and Blind RL (without images) and evaluated on unmasked validation sets.
Mechanism Analysis: The authors employ text-deletion, targeted text deletion, image-mask-rate ablation, and format-saturation controls to isolate the sources of performance gains in blind training.

Key Results

Inference-Time Fragility

Evaluation across 15 closed- and open-weight Multimodal Large Language Models (MLLMs) reveals significant fragility under modality transfer:

Performance Degradation: Average accuracy drops from 49.2% (L1) to 35.8% (L4), resulting in an average total transfer gap of 13.4 percentage points.
Variable Grounding Bottleneck: The largest performance drop occurs during Level 2→Level 3 ( $\Delta V = 7.4$ ), where models must ground variables and labels from the image. This indicates that recognizing the diagram structure is less problematic than reading and binding specific numerical quantities to physical entities.
Inconsistency: Even frontier models (e.g., GPT-5.4, Claude-4.7-Opus) exhibit low four-way consistency (e.g., 32.6% for GPT-5.4), meaning a model that solves a text-based problem often fails when the same problem is presented visually.
Human Baseline: Human high-school students (54.0% L1) show greater robustness, with performance actually improving slightly when moving from text to structure (L1→L2), whereas models degrade.

Training-Time Diagnostics

Blind Training Gains: Surprisingly, Blind RL (training with masked images) improves accuracy on unmasked validation sets. For Qwen3-VL-4B, blind RL increased Level-1 accuracy from 9.9% to 20.9% and Level-4 accuracy from 6.4% to 13.0%.
Lack of Gap Closure: Despite accuracy gains, Blind RL fails to close the modality-transfer gaps. The total gap ( $\Delta T$ ) and variable-grounding gap ( $\Delta V$ ) remain large or even widen after training.
Mechanism of Gains: Mechanism analysis reveals that blind gains are driven by residual textual cues (e.g., problem templates, answer options, numerical ranges) and dataset-level statistical regularities rather than valid visual evidence.
- Deleting text from training data eliminates blind gains.
- Blind gains persist even when format rewards are saturated, ruling out simple format compliance as the sole cause.
Cross-Benchmark Validation: These findings extend to external math and physics benchmarks (e.g., MathVerse, PhyX), suggesting that outcome-only RL often amplifies non-visual shortcuts rather than fostering true visual grounding.

Significance and Claims

The paper argues that current evaluations of multimodal reasoning are insufficient if they rely solely on final-answer accuracy. The authors claim:

Representation Invariance is Missing: Frontier MLLMs are not yet representation-invariant reasoners; they are highly sensitive to how information is encoded, with variable grounding being a primary failure mode.
RLVR Limitations: Reinforcement learning with outcome-only rewards can improve benchmark performance without actually improving the model's ability to use visual evidence. The "gains" observed in standard RL may largely stem from the model learning to exploit textual and distributional shortcuts rather than learning to ground reasoning in images.
Diagnostic Necessity: Future research must evaluate multimodal reasoning through modality-transfer robustness and training-time grounding diagnostics (such as blind training controls) to distinguish between genuine visual learning and shortcut-based performance improvements.

The work highlights the need for benchmarks and training protocols that explicitly test whether improvements rely on task-critical visual evidence, rather than just whether the model can arrive at the correct answer through alternative, non-visual pathways.