🤖 AI

Scalable Visual Pretraining for Language Intelligence

This paper challenges the text-only paradigm of foundation model pretraining by demonstrating that unsupervised visual pretraining on raw visual documents, which preserves rich information like figures and layouts, consistently outperforms text-only approaches and offers a scalable pathway to enhanced language intelligence.

Original authors: Yiming Zhang, Zhonghan Zhao, Wenwei Zhang, Haiteng Zhao, Tianyang Lin, Yunhua Zhou, Demin Song, Kuikun Liu, Haochen Ye, Haian Huang, Yuzhe Gu, Haijun Lv, Qipeng Guo, Bin Liu, Gaoang Wang, Kai Chen

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yiming Zhang, Zhonghan Zhao, Wenwei Zhang, Haiteng Zhao, Tianyang Lin, Yunhua Zhou, Demin Song, Kuikun Liu, Haochen Ye, Haian Huang, Yuzhe Gu, Haijun Lv, Qipeng Guo, Bin Liu, Gaoang Wang, Kai Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a super-smart robot how to understand the world. For a long time, the standard recipe has been to feed it mountains of books, articles, and websites, but with a catch: before the robot reads them, humans (or computers) strip away all the pictures, the squiggly math symbols, the charts, and the fancy page layouts, turning everything into plain, boring text. It's like trying to teach someone how to bake a cake by only giving them a list of ingredients written on a napkin, while throwing away the photo of the finished cake, the diagram of the mixing bowl, and the colorful graph showing how the oven temperature changes.

A new paper from 2026 suggests this "text-only" approach is missing a huge chunk of the puzzle. The researchers, led by a team from Shanghai and Zhejiang, argue that by throwing away the visual parts of documents, we are throwing away the very clues the robot needs to become truly smart. They call their new method Visual Pretraining (VP), and it's like teaching the robot to read the actual pages of a textbook, complete with diagrams and equations, rather than just the text description of them.

The Big Discovery: Seeing is Believing (and Learning)
The team tested this idea on some of the smartest AI models available (like Qwen and Llama). They took the exact same pile of scientific documents—think physics papers, math textbooks, and technical reports—and split them into two groups.

Group A (The Old Way): They converted the pages into plain text, stripping out all the visuals.
Group B (The New Way - VP): They fed the raw images of the pages directly to the model, letting it "see" the figures, tables, and layouts without ever turning them into words first.

The result? The robot trained on the raw images (Group B) consistently outsmarted the one trained on plain text (Group A). In fact, on tricky science and math benchmarks, the visual learner scored higher. For example, on a tough math test called AIME, the visual model improved its score significantly more than the text model. The paper suggests that the "lossy" process of converting a complex page with a diagram into a string of words actually deletes the very information needed to solve hard problems.

The Efficiency Hack: Doing More with Less
Here's the really cool part: the visual method wasn't just better; it was also way more efficient. The text-based model had to chew through about 80 billion text tokens to learn from these documents. The visual model, however, only needed about 20 billion visual tokens to get the same (or better) results. That's using only 25% of the data budget!

Think of it like this: If the text model is trying to describe a city by listing every single street name and building address, it takes forever. The visual model, by contrast, just looks at a map. It gets the whole picture instantly. The researchers found that the more "visual stuff" (like complex diagrams and equations) a page had, the bigger the advantage the visual model had. On pages packed with charts and formulas, the visual model's advantage grew huge, while on plain text pages, they were about the same.

No Cheat Codes, Just Pure Vision
You might wonder, "Did they cheat by showing the robot pictures and the answers?" No. They didn't use any special "image-to-text" labels or cheat sheets. The robot was just told to look at the next patch of the image and guess what it would look like, similar to how it guesses the next word in a sentence.

Surprisingly, this "just looking" training made the robot better at everything, not just looking. It got better at understanding text too, and even better at connecting pictures to words when tested later. The researchers measured this by checking how well the robot's "brain" aligned pictures with words, and found that the visual training made the two concepts fit together much more tightly, as if the robot finally understood that a picture of a cat and the word "cat" are actually the same thing.

What This Means (and What It Doesn't)
The authors are careful to say this isn't a magic wand that replaces reading text. Text is still great for learning facts that are already written down clearly. But for scientific documents, where the layout, the equation's shape, and the diagram's position carry crucial meaning, the visual approach suggests a new, scalable path forward.

They admit they haven't solved everything yet. For instance, they aren't sure if this works as well for random photos of nature or videos, since their tests were mostly on high-density scientific PDFs. But for the specific case of learning from complex documents, the paper suggests that letting the AI see the world as it is—visual, messy, and full of diagrams—might be the key to unlocking its true intelligence. It's a shift from "reading the description of the map" to "actually looking at the map."

Technical Summary: Scalable Visual Pretraining for Language Intelligence

Problem Statement

Current large foundation models rely predominantly on pretraining with massive text corpora. This paradigm operates on the implicit assumption that all learnable knowledge can be losslessly encoded as a linear sequence of text tokens. However, this approach inherently discards rich visual information present in scientific documents, such as figures, typeset equations, tables, and page-level spatial layouts. Converting these visual sources into plain text via parsing (e.g., HTML/LaTeX extraction or OCR) is a lossy process that strips away geometric constraints, symbolic topologies, and structural correspondences essential for scientific reasoning. Existing multimodal models often treat visual inputs merely as conditioning context for text prediction rather than integrating them deeply into the predictive learning objective.

Methodology: Visual Pretraining (VP)

The authors propose Visual Pretraining (VP), a framework where a foundation model learns directly from raw visual documents without text extraction or image-text pairing supervision.

Core Architecture and Training

Input Representation: Raw scientific document pages are rendered as images. A frozen vision tower encodes these images into visual features.
Sparse Foreground Extraction: To improve efficiency, blank background regions (margins, whitespace) are filtered out using patch-level statistics (pixel variance, luminance). The remaining foreground patches are ordered in raster-scan order to form a sparse visual sequence.
Projection and Prediction: The sparse visual features are projected into the LLM's hidden space via a learnable linear projection. The model is trained using an autoregressive next-visual-latent prediction objective.
- Unlike standard next-token prediction which targets discrete text tokens, VP targets continuous visual latents.
- The loss function utilizes an in-batch contrastive loss (InfoNCE) to predict the next foreground visual latent $z_{t+1}$ given the current state, distinguishing it from other patches in the batch.
Joint Training: The final objective combines standard text next-token prediction ( $\mathcal{L}_{CE}$ ) with the visual latent prediction loss ( $\mathcal{L}_{VP}$ ), interleaved according to a fixed ratio. The visual encoder remains frozen; only the LLM, the visual projection module, and the prediction head are updated.

Experimental Setup

The study employs a strictly controlled comparison between VP and Text Pretraining (TP):

Corpus: Both methods use the exact same underlying scientific PDF corpus.
Representation Difference:
- TP: PDFs are parsed into plain text (using MinerU2.5), resulting in ~80B text tokens.
- VP: PDFs are rendered as images and converted to sparse visual tokens, resulting in ~20B visual tokens.
Backbones: Experiments are conducted on multiple architectures, including Qwen3.5, Qwen3, Llama 3.2 Vision, and Llama 3.1.
Evaluation: All checkpoints undergo identical Supervised Fine-Tuning (SFT) before evaluation on scientific reasoning and multimodal benchmarks.

Key Contributions

Demonstration of Information Loss in Text-Only Pretraining: The paper provides empirical evidence that reducing visual documents to plain text incurs substantial information loss, and that visual pretraining recovers this discarded information.
Autoregressive Visual Pretraining Framework: The authors introduce a method to train foundation models to predict document patches in latent space, deeply integrating visual latents into the predictive process without requiring paired image-text data.
Scalable Alternative to Text-Only Pretraining: Through unified empirical studies across architectures and benchmarks, the work establishes VP as an effective, efficient, and scalable alternative to traditional text-only pretraining.

Results

1. Effectiveness in Scientific Reasoning

VP consistently outperforms TP on scientific reasoning benchmarks (MMLU-Pro, GPQA, AIME-25, HLE) across both native multimodal models and language-only models.

Performance Gains: On the Qwen 3.5 backbone, VP improved GPQA Diamond scores from 76.24 to 79.29 (+3.05) and MMLU-Pro from 83.91 to 85.09.
Architecture Agnostic: Improvements were observed across all tested backbones, indicating the benefit is not tied to a specific architecture.
Reasoning vs. Knowledge: The gains are most pronounced on benchmarks requiring knowledge of geometric figures and schematics (GPQA, MMLU-Pro), while improvements on HLE (focused on hard multi-step reasoning) were marginal, suggesting VP specifically recovers visual-structural knowledge.

2. Scalability and Efficiency

Token Efficiency: VP achieves superior downstream performance while consuming only 25% of the token budget compared to TP (20B visual tokens vs. 80B text tokens for the same corpus).
Scaling Laws: As training progresses, VP shows steadily increasing normalized gains over TP (e.g., 2.88× on AIME-25).
Structure Density: The advantage of VP correlates strongly with visual-structure density. On pages rich in figures, equations, and tables, VP significantly outperforms TP, whereas on text-dominant pages, performance is comparable.

3. Cross-Modal Alignment and Transfer

Despite being trained on unlabeled raw documents without explicit image-text pairing:

Alignment: VP significantly improves cross-modal alignment. Centroid separation between visual and text embeddings drops from 1.665 to 0.661, and cosine similarity rises from 0.631 to 0.907.
Multimodal Benchmarks: VP models outperform both base and TP models on multimodal benchmarks (MMMU-Pro, ChartQAPro, MathVista). Notably, TP sometimes regresses on visually heavy benchmarks (e.g., ChartQAPro), whereas VP yields substantial gains (e.g., +5.4 on ChartQAPro).
Mechanism: Attention map analysis reveals that visual reasoning in VP attends to the same semantic evidence (constraints, computation steps) as text reasoning, suggesting the model learns to extract reasoning-relevant visual cues.

Significance and Claims

The paper claims that Visual Pretraining is a scalable pathway for learning both language and visual intelligence. It challenges the default assumption that language models must be trained on text-only representations.

Complementary, Not Replacement: The authors position VP not as a replacement for text pretraining, but as a complementary objective that allows models to retain text exposure while learning from the native visual form of scientific documents.
Efficiency: The framework demonstrates that foundation models can be trained on highly compressed visual corpora (25% of the token count) to achieve or exceed the performance of text-pretrained models.
Future Directions: The work suggests that visual-latent prediction could serve as an effective pretraining objective for high-density visual corpora (charts, formulas, layouts) and opens a path toward training foundation models primarily on large-scale visual streams with lightweight text alignment.

The authors remain modest, noting that their approach is grounded in text pretraining and that the gains should be viewed as evidence that visual pretraining complements and extends foundation model capabilities, rather than operating as a fully independent visual pretraining regime. They also acknowledge that the transferability to broader visual corpora (e.g., natural images or video) remains an open question.