🤖 machine learning

Clustering and Token Denoising for Faster and More Robust VLMs

The paper introduces ClustRS, a training-free algorithm combining attention-weighted clustering and residual shrinkage denoising, which significantly reduces visual tokens by up to 97% while enhancing robustness to image noise and maintaining or improving performance on VLM benchmarks like ScienceQA-IMG and MM-VET.

Original authors: Baptiste Rossigneux, Inna Kucher, Vincent Lorrain, Emmanuel Casseau

Published 2026-08-21

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Baptiste Rossigneux, Inna Kucher, Vincent Lorrain, Emmanuel Casseau

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a computer that can see a photograph and then answer questions about it, describe a scene, or solve a puzzle based on what it sees. This is the promise of modern visual-language models, a type of artificial intelligence that combines the ability to understand images with the ability to generate human-like text. For these systems to work, they must first translate an image into a long list of digital building blocks, called tokens, which the computer's brain then processes to form an answer. The more detailed the image, the longer this list becomes. While this allows for great accuracy, it creates a massive bottleneck: processing hundreds of these tokens requires enormous computing power, making it difficult to run these smart systems on smaller devices like smartphones or drones. Researchers have long sought ways to cut down this list, removing the unnecessary parts while keeping the important ones, but existing methods often struggle when the images are imperfect or noisy, which is how most real-world photos actually look.

A team of researchers has developed a new, lightweight method to solve this problem without needing to retrain the complex AI models from scratch. Their approach, which they call ClustRS, acts like a highly efficient editor for the list of image tokens. Instead of simply picking the most obvious parts of an image or trying to keep a wide variety of parts, their system first groups similar pieces of information together. It then selects just one representative from each group, ensuring that the final list covers the different ideas in the image without repeating itself. Crucially, this grouping process is designed to ignore the visual "static" or noise that often plagues real-world photos, such as graininess or blur, preventing the system from being misled by corrupted data.

After selecting the best representatives, the method applies a second, refining step. It takes the chosen tokens and gently smooths out the noise within them, using the average characteristics of their group to correct any errors. This process is entirely automatic and requires no additional training, meaning it can be applied immediately to existing models. The researchers tested this technique on standard benchmarks that measure how well these models can reason about images, including tasks that require describing complex scenes or answering scientific questions. They found that their method significantly outperformed previous techniques, especially when the images were heavily distorted by noise or when the number of allowed tokens was drastically reduced. In the most extreme tests, where the system was forced to work with only sixteen tokens instead of the usual hundreds, their method maintained high accuracy while others failed, proving that a smarter way of selecting and cleaning information is more valuable than simply having more data.

The success of this approach highlights a shift in how we might build efficient artificial intelligence. Rather than trying to make the models larger or more complex to handle difficult conditions, the researchers showed that a simple, two-step process of grouping and cleaning can make existing systems much more robust. Their findings suggest that for these models to be truly useful in the real world, where images are rarely perfect and computing power is often limited, the focus should be on the quality and resilience of the information being processed, not just the quantity. By demonstrating that these improvements can be achieved without the heavy cost of retraining, the work opens the door for deploying powerful visual intelligence on a much wider range of everyday devices.

Technical Summary: Clustering and Token Denoising for Faster and More Robust VLMs

Problem Statement

Recent Visual-Language Models (VLMs), such as LLaVA, have achieved impressive results by integrating pre-trained vision encoders with Large Language Models (LLMs). However, a significant bottleneck exists: vision encoders typically generate a large number of visual tokens (e.g., 576 or 729), imposing a quadratic computational burden on the LLM's self-attention mechanisms. This hinders efficient deployment on edge devices.

While token pruning techniques exist to mitigate this, they face two primary limitations:

Training-based methods: Require costly retraining and specialized architectural changes.
Training-free methods: While lightweight, their performance under real-world image noise (e.g., Gaussian noise, salt-and-pepper noise) is under-explored. Existing training-free selectors often rely solely on attention scores (leading to redundancy and brittleness under noise) or solely on diversity (ignoring token salience).

The authors pose two critical questions: How severely does noise degrade current training-free pruning, and can a lightweight, noise-resilient selector be designed without retraining?

Methodology: ClustRS

The paper introduces ClustRS, a two-part, training-free algorithm designed for robust token pruning. It consists of Clustering + Huber distance (C + H) for selection and Robust Residual Shrinkage (RRS) for denoising.

1. Clustering + Huber Distance (C + H)

This component addresses the trade-off between token salience and diversity while ensuring robustness to outliers caused by noise.

Post-Projection Clustering: Unlike methods that cluster raw CLIP features, ClustRS clusters tokens after the vision-language projection layer. This ensures clustering occurs in the semantic embedding space of the LLM, reflecting the model's own understanding of conceptual relationships.
Attention-Weighted Objective: The clustering objective minimizes an attention-weighted within-cluster distortion. Tokens with higher attention scores exert a stronger pull on their cluster centroids, ensuring salient regions are prioritized.
K-means++ Initialization: The algorithm uses an attention-aware K-means++ strategy to seed centroids, spreading them across the feature space based on both distance and attention scores.
Huber Distance: To counteract the sensitivity of standard squared-error distances to noise-induced outliers, the Euclidean distance is replaced with the Huber distance. This loss function behaves quadratically for small residuals (preserving tight cluster formation) but linearly for large residuals (preventing outliers from skewing centroids).

2. Robust Residual Shrinkage (RRS)

Selection alone cannot remove noise from the remaining tokens. RRS acts as a one-pass denoising step on the selected tokens.

Residual Analysis: For each selected token $v_i$ in a cluster with centroid $c_j$ , the token is viewed as the sum of the centroid and a residual ( $r_i = v_i - c_j$ ). The residual is assumed to contain most of the noise variance.
Adaptive Shrinkage: Inspired by the James-Stein estimator, RRS shrinks the token vector toward the cluster centroid. The shrinkage factor $\alpha_i$ is adaptive, determined by the ratio of the cluster's robust scale (median residual norm) to the token's specific residual norm.
Preservation of Semantics: The update rule $\tilde{v}_i = \alpha_i v_i + (1 - \alpha_i) c_j$ is a convex combination that reduces variance without re-normalizing the vector, thereby preserving the directional statistics expected by the LLM.
Adaptation for SigLIP: For SigLIP-based models (like LLaVA-OneVision), the authors introduce a norm-aware salience term. Since SigLIP attention maps are flatter than CLIP's, they augment the attention score with the magnitude of the feature vector (which encodes salience in SigLIP) to restore a dynamic ranking signal.

Key Contributions

Novel Robust Clustering (C + H): A training-free, attention-weighted clustering procedure using Huber distance to select diverse yet salient token representatives that are resilient to noise-induced outliers.
Robust Residual Shrinkage (RRS): A lightweight, one-pass enhancement that selectively dampens the noise component of selected tokens in feature space, preserving semantic directions and avoiding distributional shifts.
Comprehensive Evaluation: Demonstration that ClustRS achieves consistent gains on MM-VET and ScienceQA-IMG benchmarks under a wide range of Gaussian and salt-and-pepper corruptions, outperforming state-of-the-art training-free methods (FasterVLM, DivPrune, VisionZip), particularly under extreme compression ( $k=16$ tokens) and high noise.
Deployment Readiness: The entire pipeline is training-free and compatible with existing inference optimizations (e.g., KV-cache, flash-attention), making it immediately deployable. It also adapts to newer SigLIP-based backbones with minimal modification.

Experimental Results

Experiments were conducted on LLaVA-1.5-7B (CLIP-based) and LLaVA-OneVision (SigLIP-based) using the MM-VET (open-ended reasoning) and ScienceQA-IMG (multiple-choice) benchmarks.

Extreme Compression ( $k=16$ ): ClustRS outperformed baselines by up to 20% under extreme noise conditions. On clean MM-VET images, it scored 0.271, surpassing FasterVLM (0.211) and DivPrune (0.232). Under severe Gaussian noise ( $\sigma=3.0$ ), it maintained a score of 0.140, outperforming all baselines.
Noise Resilience: ClustRS consistently achieved the highest scores across varying noise densities for both Gaussian and salt-and-pepper noise, particularly in low-token regimes where redundancy is minimal.
SigLIP Adaptation: On LLaVA-OneVision, the norm-aware adaptation allowed ClustRS to achieve a clean accuracy of 0.272 with only 16 tokens, dramatically outperforming FasterVLM (0.138) and DivPrune (0.164).
Computational Overhead: The method adds negligible computational cost (approximately 1ms overhead on an A100 GPU for $k=16$ ), which represents less than 1.6% of the total end-to-end generation time for a typical response.

Significance and Claims

The paper claims that ClustRS offers a simple yet powerful alternative to existing pruning rules that rely solely on scores or diversity. By combining robust clustering with residual shrinkage, the method addresses the critical gap between computational efficiency and noise resilience.

The authors emphasize that their approach:

Does not require retraining, making it adaptable to architecture changes.
Improves robustness to diverse corruptions (additive and impulsive noise) without sacrificing performance on clean data.
Enables edge deployment by drastically reducing token counts (up to 97% reduction) while maintaining or improving accuracy.

The study concludes that while token selection methods like C+H provide robust representations, there is potential for future work in refining denoising strategies, particularly for handling spatially distinct but semantically similar tokens which may be discarded by feature-space clustering.