🤖 AI

Visual Text Compression as Measure Transport

This paper introduces a measure transport framework for Visual Text Compression that quantifies task-relevant information loss through precision and coverage costs, enabling a label-free routing mechanism and adaptive foveation strategy that significantly improves downstream performance while reducing token usage.

Original authors: Lv Tang, Tianyi Zheng, Yang Liu, Bo Li, Xingyu Li

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lv Tang, Tianyi Zheng, Yang Liu, Bo Li, Xingyu Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of text documents. You want to read them quickly, but your brain (the AI model) gets tired if you try to read every single word one by one.

Visual Text Compression (VTC) is a clever trick: instead of reading the words, you take a photo of the page and show it to the AI as an image. The AI looks at the picture and "reads" it. This is much faster because the AI sees the whole page as a few big chunks (pixels) rather than thousands of tiny letters. It's like looking at a map of a city instead of reading the address of every single house.

However, the authors of this paper discovered a problem: sometimes this trick works amazingly well, and sometimes it makes the AI stupid.

The Good: On some tasks (like finding a specific fact in a long document), the "photo method" is faster and just as smart as reading the text.
The Bad: On other tasks (like solving a logic puzzle or checking a specific number), the photo method fails miserably. The AI misses tiny details because squishing text into a picture blurs the edges.

The big question was: How do we know when to use the photo and when to read the text?

The Core Idea: "Transporting" Information

The authors came up with a new way to think about this problem using a concept called Measure Transport.

Think of the text as a pile of sand. Each grain of sand is a word.

The Text Path: You carry the sand grain-by-grain in a wheelbarrow. It's slow, but you don't lose any grains.
The Visual Path: You pour the sand into a bucket and carry the bucket. It's much faster, but some sand spills out, and the grains get mixed up.

The paper argues that the "spillage" isn't random. It happens in two specific ways:

The "Smoothing" Spill (Precision Cost): When you pour sand into a bucket, tiny differences between grains get smoothed out. If the task requires distinguishing between "2023" and "2024," the bucket method might blur them together.
The "Scattering" Spill (Coverage Cost): If the important sand is spread across the whole floor, pouring it into a bucket might scatter the clues so far apart that you can't put them back together to solve the puzzle.

The Solution: A "Smart Traffic Light"

The authors built a system that acts like a smart traffic light for the AI. Before the AI tries to read the text or look at the photo, this system calculates a "Transport Efficiency Score."

It asks two simple questions without needing to know the answer to the problem first:

How much detail does this task need? (If it needs tiny details, don't use the photo).
How spread out is the information? (If the clues are scattered everywhere, don't use the photo).

Based on this score, the system decides:

Green Light (Photo): "The task is simple enough or the layout is helpful. Let's use the fast photo method."
Red Light (Text): "This task is too tricky for a photo. Let's read the text carefully."

The "Foveation" Trick: A Magnifying Glass

Sometimes, the system decides to use the photo, but it realizes one small part of the image is too blurry (like a tiny number in a contract).

Instead of giving up, the system uses a digital magnifying glass (called foveation). It zooms in only on that blurry, important part, re-captures it in high definition, and adds it to the picture. It's like looking at a map, seeing a blurry street name, and then zooming in just on that street to read it clearly, without having to zoom in on the whole map again.

What They Found

They tested this on 24 different types of language tasks (like answering questions, summarizing news, or checking facts).

The Result: Their "Smart Traffic Light" was right about 71% of the time. It knew exactly when to switch to the photo and when to stick to the text.
The Benefit: By using this switch, the AI got better at its jobs (higher scores) while using 10% fewer resources (less computing power and time) compared to just reading text all the time.

In Summary

This paper doesn't just say "photos are faster." It says, "Photos are faster, but only if you know when to use them."

They created a mathematical rule that acts like a traffic cop, directing the AI to the fastest path (text or image) based on the specific "shape" of the information, ensuring the AI never loses important details just to save time.

Technical Summary: Visual Text Compression as Measure Transport

Problem Statement

Visual Text Compression (VTC) has emerged as a strategy for long-context processing by rendering text into images and re-encoding them via Vision-Language Models (VLMs). This approach often achieves significant token compression (3–20× fewer tokens than subword tokenization). However, empirical results show that token savings do not predict downstream utility. On some tasks (e.g., multi-hop QA), the visual path outperforms text-only Large Language Models (LLMs), while on others (e.g., logical reasoning, cloze-style QA), performance collapses. Existing benchmarks document these instabilities but lack a principled measure to predict when visual encoding preserves task-relevant information and when it discards it. The central problem is that compression ratio is merely an efficiency statistic, not a predictor of information loss.

Methodology

The authors formulate VTC within the language of measure transport. They treat text tokens and visual tokens as empirical probability measures. The rendering and encoding pipeline is modeled as a push-forward map ( $T\#$ ) induced by the Vision Transformer (ViT) patch encoder. This map transforms localized textual units into coarser visual token supports, altering the geometry in which downstream reasoning occurs.

1. Decomposition of Transport Cost

The paper argues that the push-forward map induces a task-relevant information loss that decomposes into two distinct costs:

Precision Cost ( $\Delta_{intra}$ ): Arising from within-patch aggregation. The ViT merges fine-grained lexical distinctions (e.g., "2023" vs. "2024") into a single patch embedding, harming tasks requiring exact lexical fidelity.
Coverage Cost ( $\Delta_{inter}$ ): Arising from cross-patch fragmentation. Evidence distributed across different patches or pages must be recomposed via 2D spatial attention rather than the native 1D positional structure of text, weakening long-range evidence integration.

2. Downstream-Label-Free Proxy ( $C(x)$ )

To operationalize this theory without requiring downstream labels during inference, the authors derive a proxy cost function $C(x)$ :
$C(x) = \alpha \cdot W(x) + \beta \cdot L(x) \cdot (1 - TRR(x))$

$W(x)$ (Precision Feature): Measures sensitivity to token-level distinctions based on the task's answer format (e.g., extractive vs. classification). Estimated from task specifications, not data labels.
$L(x)$ (Coverage Feature): Measures the dispersion of task-relevant evidence across the input, estimated via BM25 relevance scores between questions and text segments.
$TRR(x)$ (Text Redundancy Rate): Estimated via gzip compression ratio to discount redundant information that buffers fragmentation loss.
$\alpha, \beta$ : Model-specific scalars estimated via synthetic, downstream-label-free probes.

3. Transport Efficiency (TE) and Routing

The authors define a Transport Efficiency (TE) score to balance information survival against compression benefits:
$TE(x) = ISR(x) \cdot VCR(x) = (1 + \gamma - C(x)) \cdot VCR(x)$

$VCR(x)$: Visual Compression Ratio (text tokens / visual tokens).
$\gamma$ : A structured-format bonus for inputs where rendering preserves 2D layout (e.g., tables) that flat token sequences cannot express.
Routing Rule: Select the visual path if $TE(x) \ge \tau$ (where $\tau$ is a fixed threshold), otherwise use the text path. This decision is made without consulting downstream labels.

4. Transport-Informed Foveation

For inputs routed to the visual path, the framework applies foveation to locally repair high-cost regions. It computes a patch-level cost $C_q$ and selectively re-encodes high-cost patches at higher resolution. This acts as a localized inverse of the push-forward map, triggered only when the recovered information outweighs the token overhead.

Key Contributions

Task-Dependent Instability Analysis: Demonstrated that VTC performance is highly non-uniform across NLP benchmarks and that compression ratio alone cannot predict utility.
Measure-Transport Formulation: Recast ViT-based visual encoding as a push-forward map, deriving a decomposed transport cost (precision and coverage) that explains the observed instabilities.
Label-Free Routing and Foveation: Developed a fully downstream-label-free decision criterion ($TE(x)$) and a patch-level refinement mechanism (foveation) derived from the same cost model.
Empirical Validation: Validated the framework on 24 NLP datasets using the Qwen3-4B backbone, showing that the label-free rule matches the per-dataset oracle on 70.8% of datasets and improves the macro-average score over a pure-LLM baseline while reducing token usage.

Results

Evaluated on 24 NLP benchmarks (spanning understanding, reasoning, and generation) using Qwen3-4B:

Routing Performance: The label-free routing rule correctly selected the optimal path (text vs. visual) for 17 out of 24 datasets (70.8%), matching the performance of a supervised oracle that knows the ground truth.
Performance Gains: The routed strategy achieved a +1.5 point increase in macro-average score (+3.3% relative) compared to a pure-LLM baseline.
Efficiency: The routed strategy used 10.3% fewer average input tokens than the pure-LLM baseline.
Foveation: Localized foveation improved the visual path score by +0.4 points on average, with gains concentrated on tasks where specific high-cost regions carried critical evidence (e.g., multi-hop QA).
Cross-Scale and Cross-Architecture: The qualitative patterns held across 4B, 8B, and 32B scales. Cross-architecture validation with InternVL3.5 confirmed that the task-dependent visual-vs-text gap is a structural property of the transport process, not an artifact of a specific model family.

Significance and Claims

The paper claims that the contradictions in VTC performance arise from non-uniform transport cost rather than compression ratio alone. By modeling the visual encoder as a push-forward map, the authors provide a structural account of information loss that is:

Explanatory: It decomposes failure modes into precision (within-patch) and coverage (cross-patch) costs.
Operational: It yields a practical, label-free routing rule and a refinement mechanism (foveation) that share the same underlying cost parameters.
Diagnostic: It identifies when visual compression is trustworthy and when it should be avoided or repaired.

The authors explicitly state they do not claim state-of-the-art results on individual benchmarks or a universal solution for all VTC scenarios. Instead, the contribution is a transport-based diagnostic framework that shifts the focus from "how many tokens are saved" to "what task-relevant information is lost," offering a principled starting point for future research in visual text compression. The framework is presented as a tool to diagnose and manage the trade-offs of VTC, rather than a replacement for all text-based processing.