🤖 machine learning

How Much Information Can a Vision Token Hold? A Scaling Law for Recognition Limits in VLMs

This paper investigates the information capacity limits of vision tokens in Vision-Language Models by identifying a three-phase transition from stability to collapse under increasing visual density and formulating a universal scaling law to guide the optimization of context compression efficiency and accuracy.

Original authors: Shuxin Zhuang, Zi Liang, Runsheng Yu, Hongzong Li, Rong Feng, Shiqin Tang, Youzhi Zhang

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shuxin Zhuang, Zi Liang, Runsheng Yu, Hongzong Li, Rong Feng, Shiqin Tang, Youzhi Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: How Much Can a Single "Vision Token" Carry?

Imagine you are trying to send a very long letter to a friend, but you are only allowed to use a tiny number of postcards to do it. You have to cram the entire letter onto these few postcards.

In the world of AI, Vision-Language Models (VLMs) work similarly. They look at an image (like a scanned page of text) and turn it into a sequence of digital "tokens" (little data packets) that the AI brain can read.

This paper asks a fundamental question: Is there a limit to how much information we can cram into a single vision token? If we try to put too much text into an image and force the AI to compress it into too few tokens, what happens?

The Experiment: The "Stress Test"

The researchers didn't just guess; they ran a stress test. They created images filled with pure text (like novels, laws, or letters) and slowly increased the amount of text in each image. They kept the number of "postcards" (vision tokens) the AI was allowed to use fixed, then watched what happened as the text got longer and denser.

The Three Stages of Failure

Instead of the AI getting slightly worse as the text got longer, the researchers found that performance didn't fade away slowly. Instead, it went through three distinct "phases," like a light bulb flickering before it burns out:

The Stable Phase (The "Easy Mode"):
- What happens: The AI reads the text perfectly.
- Analogy: Imagine a clear window. You can see everything perfectly because there isn't too much stuff in front of it. The AI has plenty of "room" in its tokens to describe the text.
The Instability Phase (The "Flickering Mode"):
- What happens: The AI starts making mistakes, but it's chaotic. Sometimes it gets the text right; other times, it fails completely, even if the amount of text is almost the same.
- The Cause: This isn't because the AI is "dumb." It's because of grid alignment.
- Analogy: Imagine the AI looks at the image through a grid of square windows (like a chain-link fence). If a letter happens to sit right on the line between two windows, the AI gets confused. It might see half a letter in one window and the other half in the next, and it can't put them together.
- The Fix: The researchers proved this by slightly shifting the image (like nudging a picture on a wall). When they shifted it, the "bad" letters suddenly became "good" again. This means the information was still there; it was just hidden behind the wrong part of the grid.
The Collapse Phase (The "Hard Wall"):
- What happens: The AI suddenly gives up. The error rate skyrockets, and it can no longer read the text at all.
- The Cause: This is a true capacity limit.
- Analogy: Imagine you have a backpack that can only hold 5 pounds. If you try to stuff 50 pounds of books into it, the backpack doesn't just get "a little messy"—it rips open, and everything falls out. No matter how you shift the backpack or rearrange the books, it simply cannot hold that much weight. The AI has run out of "mental space" to encode the information.

The "Magic Formula" (Scaling Law)

The researchers didn't just stop at finding the problem; they created a mathematical rule (a scaling law) to predict exactly when the AI will fail.

They found that two things matter most:

How much text is there? (The total load).
How crowded is the text? (The density).

They combined these into a single "Difficulty Score."

The Discovery: The amount of text matters much more than how crowded the letters look.
The "Instability Zone" is Consistent: They found that the "flickering" phase (where the AI is confused) always lasts for about 2.2 times the length of the text, no matter how big the image is. It's a predictable buffer zone before the total collapse.

Why This Matters (According to the Paper)

The paper concludes that while turning images into text tokens is a great way to save computer power, there is a hard physical limit to how much information can be compressed this way.

For Developers: If you want to build an AI that reads long documents, you can't just guess how many tokens to use. You need to calculate the "Difficulty Score" first. If the text is too dense, you must give the AI more tokens (more "postcards") or the system will hit the "Hard Wall" and fail.
The Takeaway: Vision tokens are powerful, but they are not magic. They have a finite capacity, and once you cross that line, the information is lost forever, not just confused.

Summary in One Sentence

The paper proves that AI vision systems have a "breaking point" where they can no longer read text, caused first by the image getting misaligned with the AI's internal grid, and finally by simply running out of digital space to hold the information.

Technical Summary: How Much Information Can a Vision Token Hold?

Problem Statement

The escalating demand for long-context understanding in Large Language Models (LLMs) is constrained by the quadratic computational complexity of self-attention mechanisms. While context compression techniques exist, relying solely on text-only modalities discards critical spatial and structural semantics (e.g., tabular alignments, hierarchical layouts). Recent vision-centric approaches, such as DeepSeek-OCR, encode rendered text images into continuous vision tokens to preserve these semantics while reducing token counts.

However, viewing the vision encoder as a lossy channel with finite capacity raises a fundamental, unexplored question: What is the upper bound of semantic information that can be compressed into vision tokens? Unlike discrete text tokenization (e.g., BPE) which guarantees lossless reconstruction, vision encoders face a trade-off between computational efficiency (minimizing token count) and representational capacity. If the information density per token exceeds the encoder's limit, recognition fails. This paper investigates the quantitative relationship between the information quantity within an image and the number of vision tokens used to encode it, aiming to identify the capacity limits of visual tokens.

Methodology

Experimental Setup

To isolate the capacity limits of vision tokens from linguistic priors, the authors designed a controlled stress test:

Data Synthesis: Texts from six diverse domains (Novels, Laws, Economics, Medicine, Newspapers, Letters) were collected. A Block-wise Shuffling strategy was employed, where text was segmented into blocks, randomly sampled, and concatenated. This forces the model to rely on visual perception rather than predicting the next token based on semantic context.
Typography Control: Typographical parameters (font size, line/character spacing) were adjusted to simulate varying densities. A dynamic layout algorithm ensured images maintained an aspect ratio between 0.9 and 1.1.
Models: The primary model was DeepSeek-OCR, chosen for its ability to control vision token budgets via input resolution. To verify generalizability, the study also evaluated InternVL3.5-8B (Tile-based High-Resolution strategy) and Qwen2.5-VL-8B (Native Dynamic/NaViT strategy).
Metrics: Recognition quality was measured using Edit Distance (ED) between the predicted sequence and the ground truth.

Investigation of Failure Mechanisms

The authors analyzed performance across increasing text lengths (information quantity) at four resolutions ( $R \in \{512, 640, 1024, 1280\}$ ) to characterize the transition from stability to collapse.

Spatial Alignment Sensitivity (Zone I): To test if errors were due to Vision Transformer (ViT) patch misalignment, a Pixel-Shift Perturbation experiment was conducted. Images were shifted by sub-patch strides to find the optimal alignment.
Capacity vs. Legibility (Zone II): To distinguish between visual legibility limits (characters becoming too small) and information capacity limits, a Visual Density Alignment experiment was performed. Images from stable and collapse phases were resized to a fixed canvas to ensure identical character pixel density, isolating the variable of total information quantity.

Probabilistic Scaling Law

The authors formulated a model unifying Average Vision Token Load ( $G$ ) (characters per token) and Visual Density ( $V$ ) (lines per ViT patch) into a Latent Difficulty ( $Z$ ) metric:
$Z = w_0 + a \log V + \alpha \log G$
A mixture model was used to predict the probability of entering a "Collapse Mechanism" based on $Z$ , treating the transition as a probabilistic process.

Key Results

1. Phase-Transition Phenomenon

The study identified a distinct three-regime behavior in recognition performance as information quantity increases:

Stable Phase: Near-zero ED. The model successfully reconstructs text.
Instability Phase: A region of high variance where ED fluctuates dramatically for identical text lengths.
Collapse Phase: A "Hard Wall" where ED abruptly surges (>0.6), indicating total failure to reconstruct content.

2. Mechanisms of Failure

Instability is Reversible: The Pixel-Shift experiment confirmed that errors in the Instability Phase are caused by spatial alignment sensitivity in ViT patch partitioning. Shifting the image to align characters with patch boundaries often restored performance to near-zero error.
Collapse is Irreversible: The Visual Density Alignment experiment refuted the "Visual Legibility Hypothesis." Even when characters were rendered at the same visual scale, samples in the Collapse Phase failed while Stable samples succeeded. This confirms the collapse is driven by an information capacity limit, where the fixed number of vision tokens cannot encode the semantic payload.

3. Universal Scaling Law

Resolution Independence: The width of the Instability Phase is scale-invariant, consistently spanning an approximate 2.2× range in text length across all resolutions.
Load Dominance: The scaling exponent for token load ( $\alpha \approx 5.53$ ) is nearly double that of visual density ( $a \approx 2.91$ ), indicating that total text length (information quantity) is a stronger predictor of failure than local character density.
Generalizability: The probabilistic scaling law, with fixed scaling exponents derived from DeepSeek-OCR, successfully predicted the phase transitions in InternVL3.5-8B and Qwen2.5-VL-8B, suggesting these limits are intrinsic to current ViT-based vision tokenization.

Significance and Claims

The paper claims to provide the first systematic study on the information capacity limits of vision tokens. Its primary contributions are:

Defining the "Hard Wall": Establishing that vision tokens have a finite information capacity, beyond which recognition collapses irreversibly, distinct from mere visual blurring or alignment issues.
Mechanistic Distinction: Separating reversible errors (caused by ViT grid misalignment) from irreversible errors (caused by capacity exhaustion).
Practical Guidance: The proposed probabilistic scaling law offers a tool to estimate the maximum information quantity a vision token budget can handle. This enables "content-aware" inference, where token budgets or resolutions can be dynamically adjusted based on document density to optimize the efficiency-accuracy trade-off.

The authors conclude that while vision-centric compression is a promising paradigm for long-context modeling, it is bound by intrinsic information limits that must be explicitly modeled and optimized in future VLM architectures. The study notes limitations regarding its focus on English text and ViT-based architectures, suggesting that languages with higher character density or fundamentally different tokenization mechanisms may exhibit shifted capacity limits.