SeGPruner: Semantic-Geometric Visual Token Pruner for 3D Question Answering

1. Problem Statement

Vision-Language Models (VLMs) are increasingly used for 3D Question Answering (3D QA) by processing multi-view images of a scene. However, aggregating observations from multiple viewpoints introduces severe visual token redundancy.

• The Bottleneck: Standard pipelines concatenate tokens from all views, creating an excessively long sequence that exceeds the token budgets of Large Language Models (LLMs), leading to high inference latency and memory consumption.
• Limitations of Existing Solutions:
  • 2D Pruners: Methods like VisPruner or ToMe are designed for 2D inputs. They lack 3D spatial awareness, often discarding critical objects or failing to cover the scene uniformly when applied to multi-view 3D data.
  • Naive 3D Approaches: Existing 3D-aware methods often rely on indirect geometric cues or simple image retrieval. They frequently fail to explicitly preserve semantically salient objects (e.g., the specific chair asked about) while simultaneously ensuring diverse spatial coverage across the 3D scene. This can lead to the loss of critical evidence or insufficient scene understanding.

2. Methodology: SeGPruner

The authors propose SeGPruner, a training-free, plug-and-play token reduction framework that operates between the visual encoder and the LLM. It addresses the problem through a two-stage, cooperative selection process:

A. 3D-Aware Feature Construction

Before pruning, the system constructs a unified 3D representation:

• Back-Projection: Visual tokens (image patches) are projected from 2D image space into a unified 3D world coordinate system using camera extrinsics and depth maps.
• Representation: Each token is assigned a 3D coordinate ($c_i$) and retains its visual feature vector ($f_i$), enabling cross-view spatial comparison.

B. Stage 1: Saliency-aware Token Selector (STS)

• Goal: Preserve tokens corresponding to semantically critical objects.
• Mechanism: It uses an attention-based importance module. Since many modern vision encoders lack a [CLS] token, the method calculates token importance by averaging the attention scores received by each token from all other tokens in the self-attention matrix.
• Action: Tokens with the highest attention scores are selected first to ensure object-centric evidence is retained.

C. Stage 2: Geometry-aware Token Diversifier (GTD)

• Goal: Supplement the salient tokens with spatially diverse tokens to ensure global scene coverage and prevent over-concentration on specific regions.
• Mechanism: Inspired by Farthest Point Sampling (FPS), it iteratively selects tokens from the remaining pool based on a joint semantic-spatial metric.
  • Metric: A fusion distance ($d_{geo}$) is calculated between a candidate token and the currently selected set. This metric combines:
    1. Geometric Distance: Euclidean distance in 3D space ($\|c_r - c_j\|_2$).
    2. Semantic Similarity: Cosine similarity of feature vectors ($1 - \text{cosine}(f_r, f_j)$).
  • Selection: The token with the maximum fusion distance (i.e., the one most dissimilar in both space and semantics to the current set) is selected.
• Action: This ensures the final token set covers the 3D scene uniformly without redundant visual information.

D. Inference

The final token sequence consists of the union of the Salient Tokens (from STS) and Diverse Tokens (from GTD), reordered to match the original visual encoder's sequence before being fed into the LLM.

3. Key Contributions

1. SeGPruner Framework: A novel, training-free token reduction strategy specifically designed for multi-view 3D QA that balances semantic importance with geometric diversity.
2. Saliency-aware Token Selector: An attention-based module that explicitly retains tokens for critical objects, preventing the loss of key reasoning evidence during aggressive pruning.
3. Geometry-aware Token Diversifier: A selector that jointly optimizes for 3D spatial distance and semantic similarity, ensuring broad scene coverage and reducing multi-view redundancy.
4. State-of-the-Art Performance: Demonstrated superior efficiency and accuracy compared to existing 2D and 3D pruning methods.

4. Experimental Results

The method was evaluated on ScanQA and OpenEQA benchmarks using the LLaVA-OneVision-7B base model.

• Efficiency:
  • Reduces the visual token budget by 91% (retaining only ~9-23% of original tokens).
  • Reduces inference latency by 86% (e.g., at 9% retention, latency drops from 1.03s to 0.63s compared to VisPruner).
• Accuracy:
  • ScanQA: Outperforms the full-token base model and other pruning methods (DTC, VisPruner) even at extreme reduction ratios (23% retention). It achieves an EM@1 of 28.0 at 23% retention, surpassing the full-token baseline (27.6).
  • OpenEQA: Maintains robust performance under aggressive reduction. At 8% retention, it outperforms VisPruner by 2.6 points and DTC by 0.6 points.
• Ablation Studies:
  • Using only saliency or only diversity yields suboptimal results.
  • The combination of both modules is essential for robust 3D reasoning, as shown by qualitative visualizations where SeGPruner preserves both object details (e.g., cables, small items) and structural continuity (e.g., tables, shelves) better than 2D-only pruners.

5. Significance

• Bridging the Gap: SeGPruner effectively bridges the gap between 2D VLM capabilities and 3D reasoning requirements by explicitly integrating 3D geometric priors into the token selection process without retraining the model.
• Practical Deployment: By drastically reducing token counts and inference latency while improving or maintaining accuracy, it makes 3D QA feasible for real-time applications in embodied AI, robotics, and autonomous driving where computational resources are constrained.
• Paradigm Shift: It challenges the notion that token pruning must be purely semantic or purely geometric, demonstrating that a semantic-geometric synergy is required for robust 3D understanding.