Technical Summary: VEN-VL

Problem Statement

Recent advancements in Large Visual-Language Models (LVLMs) have demonstrated significant potential for complex visual understanding, particularly when leveraging Mixture of Experts (MoE) architectures. However, existing efficient methods for accelerating LVLMs face a critical bottleneck: performance degradation caused by limitations in information capacity and information density.

Current approaches generally fall into two categories, both of which suffer from inherent flaws:

1. Input-wise Pruning: Methods that compress single-stream visual features into a compact representation before the LLM. These often overemphasize high compression ratios, failing to establish complete semantic modeling and significantly reducing information capacity and diversity.
2. Layer-wise Pruning: Methods that prune tokens within the LLM based on attention maps. These rely on coarse visual-text alignment in early layers, often mistakenly dropping tokens containing condensed information (e.g., small objects or subtle textures) necessary for fine-grained semantics in later layers.

The core challenge is enabling LVLMs to achieve superior performance using fewer, yet highly information-condensed, visual tokens.

Methodology: VEN-VL

The authors propose VEN-VL (Visual Ensemble MoE), a framework designed to bridge the gap between performance and efficiency by following an "enrich then compact" principle. The architecture consists of three primary components:

1. Multi-Aspect Knowledge Ensemble (MKE)

Goal: Enrich Information Capacity.
Instead of compressing a single visual stream, MKE unifies visual representations from different perspectives to boost semantic diversity while suppressing redundancy.

• Dual-Stream Extraction: It utilizes a primary encoder (e.g., CLIP) for semantic-aligned features ($F_{main}$) and an auxiliary encoder for fine-grained, detail-amplified features ($F_{extra}$).
• Structure-Aware Compression: Rather than simple concatenation, MKE employs a sequential merging strategy:
  • Spatial Merge: Compresses the spatial axis of both streams to transform locality into channel dimensions, preserving 2D grid topology.
  • Self/Cross-Merge: Uses a bipartite merge operator to identify top-$k$ cosine similarities between disjoint feature sets, aggregating matched pairs via weighted averaging. This suppresses redundancy and complements semantics globally.
• Outcome: A unified, compact visual space ($V$) that retains high information capacity and diversity.

2. Hierarchical Token Ensemble (HTE)

Goal: Enhance Information Density.
HTE operates within the deep layers of the LLM, leveraging the inherent specificity of MoE to progressively refine the visual distribution.

• Specialized Visual MoE: The framework introduces a bank of lightweight experts activated exclusively for visual tokens to capture specific visual semantics, alongside a Global Expert ($E_{global}$) that maintains general reasoning capabilities.
• Adaptive Routing: A Top-$k$ gating mechanism dynamically routes visual tokens to the most suitable experts based on their semantic content.
• Reduction-Gated Mechanism: Unlike methods relying on coarse attention maps, HTE uses the routing activation scores as a measure of token saliency. Tokens with high saliency (requiring oriented processing) are retained, while those with low scores (trivial/redundant) are pruned.
• Outcome: A progressive increase in information density across layers, ensuring deep-layer computation focuses only on highly condensed tokens.

3. Structure Information Preservation (SIP)

Goal: Provide Inductive Guidance.
To prevent the loss of structural information during token pruning, SIP endows the model with reconstruction capabilities.

• Semantic Propagation: Pruned token embeddings are re-initialized with structural properties (RoPE). Remaining condensed tokens iteratively propagate holistic semantics to these pruned tokens in a shared space until convergence.
• Explicit Visual Supervision: A reconstruction loss ($\mathcal{L}_{qrec}$) is applied between the reconstructed representation and the original input. Crucially, this uses quantization to map exact values into low-level representations, avoiding the model's over-reliance on regressing exact pixel values (which can lead to focusing on trivial patterns).
• Outcome: Ensures retained tokens restore vanilla condensed information to a great extent, guiding the selection process.

Key Contributions

1. Enrich-Then-Compact Principle: The paper introduces a design philosophy that first enriches the visual space via MKE and then progressively compacts it via HTE, jointly addressing information capacity and density.
2. VEN-VL Framework: A novel Visual Ensemble MoE framework integrating MKE, HTE, and SIP, which effectively bridges the performance-efficiency gap in LVLMs.
3. Structure Information Preservation (SIP): A mechanism that provides inductive guidance for token selection by assigning reconstruction abilities to the model, ensuring structural integrity is maintained during reduction.
4. State-of-the-Art Performance: The framework achieves superior results in complex multi-modal tasks with significantly fewer tokens (retaining only ~7.5% to 10% of visual tokens).

Experimental Results

The authors evaluated VEN-VL on multiple benchmarks, including MMBench, TextVQA, MME, and MMMU, using various LLM backbones (e.g., Qwen2.5, Qwen3).

• Performance Gains: Compared to the strong baseline TinyLLaVA (Qwen2.5-1.5B), VEN-VL achieved substantial improvements: +8.1% on TextVQA, +5.2% on MMB, and +6.8% on MMBCN, despite retaining only ~10% of visual tokens.
• Efficiency: In comparisons with other token pruning methods (e.g., PruMerge, FastV) under fair settings (Qwen3-0.6B backbone), VEN-VL outperformed them significantly while retaining only 7.5% of tokens. For instance, it achieved an average score of 50.1 compared to 40.9 for PruMerge, with a manageable increase in FLOPs relative to the performance gain.
• Scalability: The design demonstrated strong compatibility with existing MoE frameworks (e.g., Kimi-VL), showing potential for scaling to larger models with minimal performance gap and significant inference speed boosts.
• Ablation Studies:
  • Removing MKE reduced average performance, validating the need for multi-aspect enrichment.
  • Applying HTE only in shallow layers or across all layers (Dense) yielded suboptimal results; the "Second-Half" (deep layers) application was most effective.
  • Removing SIP or using standard MSE reconstruction loss ($\mathcal{L}_{rec}$) instead of quantized loss ($\mathcal{L}_{qrec}$) led to performance drops, confirming the necessity of topology-based guidance.

Significance and Claims

The paper claims that VEN-VL effectively addresses the bottleneck of information capacity and density in current efficient LVLMs. By unifying multi-perspective visual representations and utilizing adaptive MoE routers for dynamic token selection, the framework allows models to perform complex visual understanding tasks with a fraction of the visual tokens required by standard approaches.

The authors position VEN-VL as a significant step toward scalable, efficient multi-modal understanding, demonstrating that high performance does not necessitate high token counts if the information density and structural integrity are preserved through specialized ensemble and pruning mechanisms. They acknowledge that while the algorithmic optimization is robust, translating theoretical FLOPs reductions into real-world end-to-end acceleration requires further system-level efforts (e.g., kernel optimizations).