🤖 AI

When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning

This paper introduces Visualized Task Semantics (VTS) to reveal a significant performance gap in multimodal models when instructions are embedded as pixels rather than text, and proposes a prompt-region grounding method that effectively bridges this semantic channel gap without requiring OCR or region metadata at inference.

Original authors: Yongxin Wang, Ruizhe Zhou, Yueling Tang, Yingying Zhu, Xuemin Zhao, Xiaojun Chang, Xiaodan Liang

Published 2026-08-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yongxin Wang, Ruizhe Zhou, Yueling Tang, Yingying Zhu, Xuemin Zhao, Xiaojun Chang, Xiaodan Liang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot to solve puzzles. This robot, known as a Multimodal Large Language Model (MLLM), is a bit like a genius student who can read text and look at pictures at the same time. Usually, you give the robot a picture of a math problem and a separate text message saying, "Solve this!" The robot reads the text instruction, looks at the picture, and gives the answer. But what if you didn't give the instruction as a text message? What if you wrote the instruction inside the picture itself, like a sign painted on a wall?

Scientists have been wondering if this robot would still work just as well. Does it matter if the instruction is a "text token" (a digital word) or a "pixel" (a tiny dot of color on a screen)? In the world of artificial intelligence, there's a suspicion that robots might be great at reading text but terrible at "reading" instructions that are part of the image. They might be able to copy the words perfectly (like a photocopier) but fail to actually understand that those words are the boss telling them what to do. This paper dives into that mystery to see if the robot gets confused when the rules are written in the picture instead of the chat box.

The Mystery of the "Pixelated" Question

The researchers, a team from universities and tech companies, decided to test this idea with a clever experiment they call Visualized Task Semantics (VTS). Imagine you have a standard math test. Usually, the question is typed out above the image. For their experiment, they took the exact same question, erased the typed text, and painted the question directly onto the image, right above the problem. They even replaced the original typed instruction with a tiny, boring note that just said, "Help me solve the problem."

They ran this test on six different AI models using four different types of tricky benchmarks (like math charts and complex diagrams). The result was a massive drop in performance. On average, the models got 17.8 points less accurate when the question was painted in the image compared to when it was typed out. In some cases, the drop was as high as 28.0 points.

It wasn't just that the robots couldn't read the letters. The researchers checked, and the models could actually transcribe the painted words correctly most of the time. The problem was deeper: the models were failing to use those words as instructions. It's as if the robot saw the sign on the wall, could spell the words, but didn't realize, "Oh, this sign is telling me what to do!" The researchers call this a "semantic channel gap." The path the information takes (text vs. image) changes how the robot thinks, even if the words are identical.

The Solution: Teaching the Robot to "Ground" the Sign

To fix this, the team developed a new training method called Prompt-Region Grounding. Think of it like teaching a child to point at a sign and say, "This is the rule!"

They used two special tricks during training:

PVRD-SG (The "Meaning Match"): They showed the robot the image with the painted question and a separate, clean text version of that same question. They forced the robot to realize that the specific patch of pixels containing the question means the same thing as the typed text. It's like connecting the physical sign on the wall to the concept of the rule in the robot's brain.
PRMLP (The "Blind Spot" Test): They took the image with the painted question and covered up parts of the text with a black box (masking it). Then, they asked the robot to guess what the hidden text meant based on the rest of the image. This forced the robot to learn the essence of the instruction, not just memorize the exact pixels.

The best part? This training didn't require the robot to use any special tools like OCR (Optical Character Recognition) or to know exactly where the text box was when it was actually taking a test. During the final exam, the robot just sees the image with the painted question and answers directly, just like before.

The Results: Closing the Gap

The results were promising. After this new training, the models' accuracy on the "painted question" tests jumped from 58.0% to 66.3%. That's a significant improvement, narrowing the gap between text instructions and image instructions. The models also got slightly better at the original text-based tests (going from 69.1% to 70.3%), proving they didn't lose their old skills.

The researchers tested this on over 1,000 real-world images they hadn't seen before, like photos of worksheets, forms, and screenshots. The new method improved accuracy by 7.9 points on these real-world pages.

What This Means

The paper concludes that reading text in an image and using that text as an instruction are two different skills. Just because a robot can read the words doesn't mean it knows how to listen to them when they are part of a picture. By teaching the robot to "ground" the visual question to its meaning, they helped it bridge the gap.

The authors are careful to note that while they improved the situation, they didn't completely solve it. There is still a small gap remaining, and the method works best when the text is clearly visible and in a standard format. However, this study proves that the way we talk to AI matters. If we want robots to be truly smart, we need to teach them that a question written in pixels is just as important as a question typed in a chat box.

Technical Summary: When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning

1. Problem Statement

Multimodal Large Language Models (MLLMs) increasingly process screenshots and documents where the task instructions themselves are rendered as text within the image (pixels). While high-resolution models can read such text with high accuracy, it remains unclear whether they utilize these visual instructions with the same efficacy as text-based prompts.

Prior work has observed performance drops when instructions are visualized, but these gaps often conflate multiple factors: canvas expansion, image resizing, renderer artifacts, and the semantic shift from text tokens to pixels. Crucially, existing benchmarks do not distinguish between a model's failure to read the visual text (OCR failure) and its failure to use the visual text as a reasoning instruction (semantic channel gap). This paper posits that reading task-bearing text and grounding it as an instruction are distinct capabilities, and that current MLLMs exhibit a systematic "semantic channel gap" where visual instructions are less effective than typed ones, even when the text is transcribed correctly.

2. Methodology

2.1 Visualized Task Semantics (VTS)

To isolate the semantic channel gap, the authors introduce Visualized Task Semantics (VTS), a controlled intervention.

Original View: The question $q$ is provided as text tokens, and the image $x$ contains visual evidence.
VTS View: The question $q$ is rendered as text pixels inside a white panel above the image $x$ . The native text prompt is replaced with a fixed cue (e.g., "Help me solve the problem").
Controls: The study employs controls for canvas geometry, prompt duplication (question in both channels), and image-only inputs to rule out non-semantic causes for performance drops.

2.2 Prompt-Region Grounding

To bridge the gap, the authors propose Prompt-Region Grounding, a training framework that explicitly aligns the visual representation of the question region with its typed semantics. This method operates during supervised fine-tuning (SFT) and reinforcement learning (GSPO) without altering the inference interface (no OCR or region metadata required at test time).

The framework introduces two region-level objectives applied to VTS and real-world visual-prompt examples:

Prompt-Visual Representation Distillation with Semantic Grounding (PVRD-SG):
- Goal: Align the visual representation of the prompt region with the frozen semantic representation of the typed question.
- Mechanism: The model pools visual tokens from the known prompt region box $b_i$ in the rendered image. This representation $z_i^{vis}$ is optimized to maximize cosine similarity with a cached, frozen text embedding $t_i$ derived from the typed question.
- Constraint: The text target is frozen and stop-gradient, ensuring the model learns to recover the task semantics from the visual region without being forced to transcribe the text.
Prompt-Region Masked Latent Prediction (PRMLP):
- Goal: Ensure the visual representation of the prompt region remains stable and recoverable even when parts of the rendered text are occluded or difficult to observe.
- Mechanism: A fraction of the prompt region in the composite image is masked. The model must predict the representation of the corresponding clean crop (the unmasked prompt region) from the masked input.
- Constraint: This is a latent consistency objective, not pixel reconstruction. It forces the model to maintain a robust internal representation of the instruction region.

2.3 Training Pipeline

Balanced Replay: The model is trained on a paired dataset containing both Original and VTS views of the same problem, sharing the same answer target.
Supervised Fine-Tuning (SFT): The model minimizes a combined loss: standard autoregressive next-token loss ( $L_{sft}$ ) plus the region-level losses ( $L_{sg}$ and $L_{prmlp}$ ).
Reinforcement Learning (GSPO): The supervised checkpoint is further refined using Group Sequence Policy Optimization (GSPO) with rewards for format correctness and answer accuracy.

3. Key Contributions

Quantification of the Semantic Channel Gap: The study demonstrates that moving a question from text tokens to image pixels causes a systematic accuracy drop across six MLLMs and four benchmarks. The average drop is 17.8 points, with some pairs dropping by as much as 28.0 points.
Decoupling Reading from Reasoning: Diagnostic tests reveal that models often transcribe visual questions correctly (87.6% exact match) but fail to use them effectively (48.4% accuracy). Re-inserting the transcribed text into the text channel recovers significant accuracy, proving the gap is not merely an OCR failure but a failure to ground visual text as an instruction.
Prompt-Region Grounding Method: The authors introduce a training recipe that closes this gap. By aligning the visual prompt region with typed semantics and enforcing latent consistency, the method improves VTS accuracy significantly without requiring OCR or region detection at inference.
Generalization: The method improves performance on held-out real-world pages (UIs, worksheets, forms) and independent benchmarks (VISTA-Bench, OCRBench v2), suggesting the learned grounding capability transfers beyond the specific VTS rendering.

4. Experimental Results

The experiments were conducted primarily on Qwen3-VL-4B and Qwen3-VL-8B, with transfer results on InternVL3.5-8B.

Baseline Performance: Across four benchmarks (MATH-Vision, MathVista, ChartQA, MMMU), the average VTS accuracy for the baseline (Balanced Replay) was 61.4%, compared to 69.8% on the Original interface (a gap of 8.4 points).
Method Performance: The proposed method ("Ours") raised the average VTS accuracy to 66.3% while maintaining Original accuracy at 70.3%.
- This reduces the semantic channel gap from 8.4 points (Balanced Replay) to 4.0 points.
- Specific gains on VTS: +7.8 (MATH-Vision), +7.0 (MathVista), +6.2 (ChartQA), +12.4 (MMMU).
Component Analysis:
- PVRD-SG provided the largest single-component gain (+2.6 points over PRMLP alone).
- PRMLP added an additional +1.8 points when combined with PVRD-SG.
- Reading specifically from the prompt panel outperformed full-image or random-region summaries.
Real-World Generalization: On 1,000 held-out real-world pages, the method improved accuracy by 7.9 points (from 58.0% to 65.8%) over standard SFT, demonstrating robustness to diverse layouts and fonts.
Cost Efficiency: The improvements were achieved at matched training costs (FLOPs) compared to standard SFT, proving the gains come from the specific objectives rather than simply more training time.

5. Significance and Claims

The paper claims that the channel carrying a question fundamentally alters reasoning capabilities in MLLMs, even when the wording and answer remain fixed. The primary significance lies in distinguishing two capabilities often treated as identical: reading text in an image and grounding that text as a task instruction.

The authors argue that:

The Gap is Semantic, Not Just OCR: Models can read the text but fail to treat visual pixels as authoritative instructions.
Region-Level Supervision is Effective: Explicitly connecting the visual region of the instruction to its typed semantics allows models to recover the "clean" representation of the task, bridging the channel gap.
Inference Simplicity: The method requires no additional modules (OCR, object detection) at inference time; the model learns to attend to the relevant region implicitly during training.

The paper concludes that while moving questions into pixels is not a new problem, the specific mechanism of prompt-region grounding offers a scalable solution to the semantic channel gap, enabling MLLMs to reason over visual instructions as effectively as typed ones. The authors remain modest, noting that the method assumes known task regions during training and does not claim to be a general task-region detector for arbitrary images.