🤖 AI

Not All Visual Tokens Are Equally Safe to Remove:Consequence-Sensitive Visual Token Compression

This paper introduces a consequence-sensitive visual token compression method for vision-language models that dynamically allocates computational resources based on the potential cost of errors, significantly reducing high-stakes mistakes and overall cost-weighted error rates compared to traditional content-driven or uniform allocation strategies.

Original authors: Jingbo Wen, Liang He, Mingyu Cao, Haoyu Wang, Minxuan Hu, Kangning Cui, Xilu Wang

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jingbo Wen, Liang He, Mingyu Cao, Haoyu Wang, Minxuan Hu, Kangning Cui, Xilu Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the head chef at a busy restaurant, but you have a strict rule: you can only use a fixed amount of ingredients for every single dish you cook, no matter what it is. Usually, chefs try to make the "average" meal taste as good as possible by spreading those ingredients evenly. But what if one dish is a simple side of fries, while another is a life-saving medicine for a patient? If you mess up the fries, it's annoying. If you mess up the medicine, it's a disaster. Most computer programs that look at pictures and answer questions (called Vision-Language Models) act like that old-school chef. They try to save computing power by ignoring "boring" parts of an image, assuming that every mistake they make costs the same amount of energy. But in the real world, a mistake isn't just a mistake; it has a price tag. This paper asks a simple, revolutionary question: What if we stopped trying to make the average meal perfect, and instead focused our limited ingredients on the dishes where a mistake would be the most expensive?

The researchers behind this study, led by Jingbo Wen and Liang He, propose a new way to handle these computer models called "consequence-sensitive visual token compression." To understand their trick, think of an image not as a single picture, but as a mosaic made of thousands of tiny tiles called "tokens." When a computer looks at a photo, it processes all these tiles. To save time and money, current methods try to throw away the tiles that seem least important, like the blue sky in a photo of a car. They do this by looking at the picture itself to decide what to keep. The authors argue this is like judging a book by its cover; sometimes the "boring" background tiles are actually crucial for the specific question being asked.

Instead of just looking at the picture, the authors suggest looking at the question or the task first. They realized that some questions are "high-stakes" (like "What is the total on this invoice?") and some are "low-stakes" (like "What color is the background?"). Their method works in two steps. First, they run a "calibration" phase offline, where they test the model to see exactly how many tiles (tokens) it needs to get high-stakes questions right versus low-stakes ones. Then, when a real user asks a question, the system checks the "consequence" of getting it wrong. If the question is high-stakes, the system allocates a huge budget of tiles to that image, ensuring the computer sees every detail. If the question is low-stakes, it gets a much smaller budget, saving resources for the important stuff.

The team tested this on a tricky setup where high-stakes and low-stakes questions were asked about the exact same images. This was crucial because it proved that the improvement didn't come from the pictures being different, but purely from how the computer decided to spend its energy. They found that by shifting the budget, they could cut the number of expensive, high-stakes errors by more than half (dropping from 0.300 to 0.133) without using any more total computing power than before. In fact, because they spent less time on the easy questions, the whole system actually ran about 21% faster.

The paper also discovered a "tipping point." When the cost of making a mistake is the same for every question, the best strategy is still to treat everyone equally. But as soon as the cost of a mistake starts to vary (for example, if a wrong answer on a medical report is 5 times worse than a wrong answer on a weather report), the system should start shifting its resources heavily toward the dangerous questions. The researchers showed this works across different types of documents, charts, and even different computer models. They concluded that we shouldn't just measure how "accurate" a model is on average; we should measure how well it prevents the specific, costly mistakes that matter most in the real world.

Technical Summary: Consequence-Sensitive Visual Token Compression

1. Problem Statement

Vision-Language Models (VLMs) incur significant inference costs dominated by visual tokens. Existing visual token compression methods (e.g., FastV, SparseVLM) aim to maximize average accuracy under a fixed compute budget by pruning tokens based on input-derived signals such as attention, redundancy, or uncertainty.

However, these methods implicitly assume that all prediction errors carry equal cost. In real-world downstream applications (e.g., document analysis, medical imaging), the consequences of errors are highly asymmetric. Misreading an invoice amount or a transaction identifier is far more costly than misclassifying a background color or a decorative element. Current allocation strategies cannot exploit this asymmetry because they rely solely on visual content properties, ignoring inference-time observable signals regarding the consequence of an error (e.g., question type, task identity).

The paper poses a new optimization problem: Given a fixed total visual token budget and a set of requests with varying error costs ( $c_i$ ), how should the budget be allocated across samples to minimize Cost-Weighted Error (CWE) rather than average accuracy?

2. Methodology

The authors propose a Consequence-Sensitive Visual Token Compression framework that follows a calibrate-then-allocate procedure.

2.1 Framework Overview

The method operates in two stages:

Offline Calibration: Using a validation set, the system estimates tier-specific error–budget curves ( $e_t(b)$ ). Samples are grouped into tiers (e.g., high-consequence vs. low-consequence) based on observable signals (keywords in questions or task metadata). The system evaluates various budget allocations to find the policy that minimizes the estimated cost-weighted objective.
Online Inference: For each incoming request, the system assigns a consequence tier using the observable signal (e.g., a frozen keyword rule or task ID). The pre-calibrated budget for that tier is applied to the input before VLM inference.

2.2 Optimization Formulation

The goal is to minimize the cost-weighted error $J$ :
$\min_{\{b_t\}} \sum_{t} n_t c_t e_t(b_t)$
Subject to:
$\sum_{t} n_t b_t \le B, \quad b_t \ge b_{min}$
Where $n_t$ is the number of samples in tier $t$ , $c_t$ is the consequence weight, $e_t(b)$ is the error rate at budget $b$ , and $B$ is the total budget.

Due to the discrete nature of token budgets (a 13-point grid) and non-convex error curves, the authors employ an exhaustive search over feasible budget pairs rather than relying on continuous Lagrangian derivatives.

2.3 Budget Realization Mechanisms

The paper evaluates two mechanisms to realize the allocated token budgets:

True Token Deletion: Encoding at full resolution, ranking tokens via a fixed selector (e.g., redundancy), and strictly deleting unselected tokens while preserving spatial coordinates and position IDs to maintain multimodal positional encoding integrity.
Resolution Reallocation: Resizing the input image so the encoder naturally emits approximately the target token count. This mechanism requires no model-specific code and is used as the primary deployment method.

2.4 Evaluation Protocol

To isolate the effect of consequence information from content-based signals, the authors introduce a within-task benchmark. High- and low-consequence questions are drawn from the same distribution of dense document images (e.g., DocVQA). This ensures that visual content is identical across tiers, preventing content-driven allocators from exploiting visual sensitivity differences.

3. Key Contributions

New Formulation: Introduction of consequence-sensitive visual token compression that allocates computation to minimize cost-weighted error, utilizing inference-time observable consequence signals rather than just input content.
Controlled Evaluation: A within-task benchmark design where high- and low-consequence questions share identical image distributions. This eliminates content-based shortcuts, proving that gains are derived solely from consequence awareness.
Allocation Frontier Characterization: The authors derive an allocation frontier showing that:
- When error costs are symmetric ( $r=1$ ), uniform allocation is optimal.
- As cost asymmetry grows, the optimal strategy shifts tokens toward high-consequence samples.
- A transition regime exists where the benefit of token transfer increases with the cost gap.
Generalization and Invariance: The principle is shown to be orthogonal to token selection mechanisms (redundancy, attention, saliency) and robust across different VLM architectures (Qwen2.5-VL, LLaVA-OneVision) and budget realization methods.

4. Experimental Results

4.1 Attribution and Within-Task Performance

On the controlled within-task benchmark (DocVQA):

Content-driven allocators (using feature redundancy) performed no better than uniform allocation because visual inputs were identical across tiers.
Consequence-sensitive allocation reduced high-stakes errors from 0.300 to 0.133 (a 2.25× reduction) under the same total token budget.
This came at the cost of increased unweighted average accuracy (0.488 vs. 0.337), confirming the method optimizes for the specific cost-weighted objective.
The anti-allocation strategy (giving more tokens to low-consequence questions) significantly increased CWE, validating the directionality of the approach.

4.2 The Allocation Frontier

Analysis of error–budget curves revealed that high- and low-consequence questions have statistically indistinguishable token sensitivity (slopes of 0.00173 vs. 0.00171). The improvement is purely due to cost weighting.

Break-even Ratios: The optimal allocation shifts from uniform to consequence-biased as the consequence ratio ( $r$ ) increases. For DocVQA, the transition occurs at $r \approx 2.8$ .
Generalization: The method generalizes to ChartQA and InfographicVQA. In InfographicVQA, the method reduced CWE even when the cost ratio was low ( $r=0.7$ ), suggesting that denser information tasks benefit more from consequence-aware allocation.

4.3 Invariance and Mechanism

Mechanism: The gains persist whether using true token deletion or resolution reallocation. Resolution reallocation (which aligns with Qwen2.5-VL's training) achieved even better absolute performance while maintaining the relative gain of consequence-aware allocation.
Model: The principle holds for LLaVA-OneVision, despite architectural differences.
Selectors: The allocation strategy is orthogonal to the token selector. It improves performance regardless of whether tokens are selected via redundancy, random, saliency, FastV, SparseVLM, or VisionZip.

4.4 Mixed Workload and Efficiency

On a realistic mixed workload (DocVQA, TextVQA, VQAv2):

Consequence-sensitive allocation reduced cost-weighted error by 38% compared to uniform allocation.
Compared to full-resolution inference, the deployed two-tier allocation (using resolution reallocation) achieved approximately 21% lower latency while reducing CWE.

5. Significance and Claims

The paper claims that visual token compression should be evaluated not only by how much average accuracy is preserved but by which errors are prevented.

Orthogonality: Consequence-sensitive allocation is an orthogonal axis to existing content-driven pruning. It does not replace token selection methods but complements them by determining how many tokens to allocate to a sample based on its downstream risk.
Practicality: The framework is lightweight and deployment-oriented. It requires only inference-time observable signals (task ID or question keywords) and offline calibration, avoiding complex online optimization or reinforcement learning.
Fundamental Insight: The work establishes that when error costs are asymmetric, the optimal resource allocation strategy fundamentally shifts from preserving average performance to prioritizing high-stakes samples, even when those samples are visually indistinguishable from low-stakes ones.

The authors conclude that in scenarios where downstream decisions carry unequal risks, consequence-sensitive allocation provides a principled and effective method to optimize VLM efficiency without sacrificing critical accuracy.