Technical Summary: VisCo – Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression

1. Problem Statement

Vision-Language Models (VLMs) typically process high-resolution images by encoding them into a large number of visual tokens. This results in substantial inference latency and memory overhead due to the quadratic complexity of self-attention mechanisms and the storage requirements for Key-Value (KV) caches. This bottleneck severely limits the deployment of VLMs in resource-constrained and real-time scenarios.

Existing solutions for visual token compression fall into two categories, both with significant limitations:

• Training-free methods: These rely on heuristic metrics (e.g., attention scores or feature similarity) to prune or merge tokens. While efficient, they suffer from significant performance degradation under high compression ratios. They are prone to "attention shift," leading to hallucinations, and often fail to preserve text-guided global semantics or fine-grained details.
• Training-based methods: These introduce external compression modules or require substantial retraining (including re-alignment and instruction tuning). While more robust, they incur heavy retraining costs, compromise the pretrained VLM's priors, and often lack plug-and-play flexibility.

The core challenge is to achieve effective visual token compression that preserves semantic fidelity under aggressive compression (e.g., reducing tokens to a single unit) without requiring massive retraining or external modules.

2. Methodology: VisCo Framework

The authors propose VisCo, a training-efficient, self-compression framework that reuses the pretrained VLM itself as an intrinsic compressor. VisCo is designed as an asymmetric VLM autoencoder with shared parameters, operating in two stages:

A. Asymmetric Architecture

• Encoding Stage: The framework augments the pretrained VLM backbone with a lightweight trainable module. Specifically, it introduces a small set of learnable memory tokens ($X_m$) and utilizes LoRA adapters on the Q/V projections. The encoder processes the joint sequence of visual tokens ($X_v$) and memory tokens ($X_m$).
• Decoding Stage: The LoRA adapters are removed, and the original frozen pretrained VLM backbone is reused directly as the decoder. This design ensures that VisCo requires only lightweight adaptation of the existing capabilities rather than relearning the multimodal generation process from scratch.

B. Sequence Construction and Causal Masking

VisCo strictly adheres to the standard causal masking mechanism of decoder-only VLMs. Memory tokens are appended after the visual tokens in the input sequence. This allows the memory tokens to naturally attend to all preceding visual tokens using the model's intrinsic attention prior, aggregating rich visual information without introducing customized interaction rules.

C. Hierarchical Information Aggregation and Passing

Recognizing that Transformer layers exhibit functional specialization (lower layers encode local patterns, higher layers encode abstract semantics), VisCo employs a hierarchical information-passing mechanism:

1. Memory Bank Construction: During encoding, instead of passing the final hidden states to the decoder, the system retains the Key ($K$) and Value ($V$) matrices corresponding only to the memory tokens at every layer $l$. These are stored in a layer-wise memory bank ($K_{mem}, V_{mem}$).
2. Hierarchical Decoding: During decoding, these layer-wise memory KV caches are directly populated into the decoder's KV cache. This allows the decoder to access multi-granularity semantic information (from low-level textures to high-level discourse) without needing projection layers, leveraging the parameter sharing between the encoder and decoder.

D. Training Objective

VisCo is fine-tuned end-to-end using teacher forcing to maximize the conditional likelihood of the ground-truth answer given the compressed memory and textual inputs. No additional pre-training objectives are introduced.

3. Key Contributions

1. Asymmetric Intrinsic Self-Compression Framework: VisCo utilizes the pretrained VLM itself as the compressor, keeping the backbone intact and requiring only lightweight training (LoRA + learnable memory tokens).
2. Hierarchical Information-Passing Mechanism: The method introduces a mechanism to transfer multi-granularity semantics from encoder layers to the decoder via layer-wise KV caches, preserving the structural hierarchy of visual information under compression.
3. Comprehensive Evaluation: Extensive experiments across 3 VLM backbones (LLaVA-1.5-7B, Qwen2-VL-2B, Qwen2-VL-7B) and 6 benchmarks demonstrate consistent superiority over existing methods, particularly under aggressive compression.

4. Experimental Results

The paper evaluates VisCo on benchmarks including GQA, MMB, MMB-CN, MME, POPE, and MMVet.

• Performance under Aggressive Compression:

  • On LLaVA-1.5-7B (576 tokens $\to$ 32 tokens), VisCo retains 91.8% of the original performance, outperforming the best baseline (VisPruner) by 4.0 points in average score.
  • In the extreme single-token setting (1 token), VisCo retains 85.3% of the original performance, significantly outperforming training-free methods (e.g., SparseVLM at ~72.6%) and even surpassing stronger training-based references like MatryoshkaQuery and VoCo-LLaMA on specific metrics.
  • On Qwen2-VL-2B (which already has built-in compression), VisCo retains 95.2% performance at 25% token retention and 91.4% at 12.5% retention, outperforming fine-tuned baselines like VisionZip.

• Qualitative Analysis:

  • VisCo successfully captures both overall scene context and key regions even with very few tokens, whereas attention-based pruning methods lose global context and details.
  • In a single-token setting, VisCo can generate detailed captions containing object-level descriptions and coherent global semantics.

• Efficiency:

  • VisCo significantly reduces KV cache storage.
  • While the initial encoding step introduces overhead, VisCo achieves faster decoding speeds. In multi-turn dialogue scenarios, the reuse of cached representations allows VisCo to surpass the latency of training-free methods (like FastV) after just three dialogue turns.

• Complementary Representations:

  • An ablation study (VisCo+) shows that combining the learned memory tokens with original visual tokens can actually improve the base model's performance on several benchmarks. This suggests the memory tokens capture complementary information rather than just being a compressed subset.

5. Significance and Claims

The paper claims that VisCo demonstrates that leveraging the intrinsic priors of pretrained VLMs enables robust visual token compression with only lightweight training.

• Overcoming Trade-offs: VisCo avoids the severe performance degradation of training-free methods under high compression and the heavy retraining costs of external-module-based approaches.
• Native Processing Pattern: By following the VLM's native processing pattern (using causal masking and parameter sharing) rather than rewriting it, VisCo achieves performance competitive with heavyweight methods while retaining the flexibility of lightweight adaptation.
• Beyond Compression: The authors suggest that the learned memory tokens act as complementary representations that enrich visual features, potentially improving the model beyond mere compression efficiency.

The work concludes that effective visual token compression hinges on strong information encoding capabilities already present in pretrained VLMs, which can be unlocked through efficient self-compression frameworks like VisCo. Future work is proposed to explore adaptive token allocation and extension to video compression.