Technical Summary: CAPruner

Problem Statement

Large Language Models (LLMs) have emerged as powerful tools for 3D Vision-Language (3D-VL) tasks, which require identifying target objects based on their spatial relationships to anchor objects. To facilitate this, scene graphs are often used to represent objects and their pairwise spatial relations. However, in scenes with hundreds of objects (e.g., the InteriorGS dataset averages 554 objects per scene), the number of pairwise relations grows quadratically ($\binom{n}{2}$), leading to token counts that exceed LLM input limits and cause computational inefficiencies.

Existing solutions, such as the proximity-based K-Nearest Neighbors (KNN) strategy used in 3DGraphLLM, attempt to mitigate this by retaining only relations between an object and its nearest neighbors. The authors identify critical flaws in this approach:

1. Task Irrelevance: Spatial proximity does not guarantee task relevance. A necessary relation for a specific query (e.g., a chair relative to a bed) may be distant and thus pruned, while irrelevant local relations are retained.
2. Structural Fragmentation: Proximity-based pruning often disconnects the scene graph. In the ScanNet dataset, over 67% of scenes become disconnected after KNN pruning, preventing the LLM from reasoning across different connected components.
3. Rigidity: Fixed pruning policies fail to adapt to the specific context of a query, forcing models to encode redundant information while missing critical long-range dependencies.

Methodology: CAPruner

The authors propose CAPruner (Conceptual-Adjacent Scene Graph Pruner), a lightweight model designed to select a subset of scene graph edges that are most pertinent to a specific 3D-VL task. The method integrates fuzzy semantic relevance with spatial proximity.

1. Core Components

• Fuzzy Semantic Matching: Instead of relying on strict attribute matching (e.g., color or shape), which can be error-prone in 3D point cloud alignment, CAPruner uses fuzzy matching on object categories. It computes a semantic relevance score ($s_i$) for each object by measuring the maximum similarity between the object's category and the tokens in the natural language task description. This ensures that all potential candidates (e.g., all "chairs" when the query mentions "red chair") are weighted highly, reducing the risk of pruning the correct target.
• Spatial Proximity Weighting: To handle perspective-dependent spatial relations (e.g., "left," "right") which are difficult to predict accurately, the model weights edges based on the Euclidean distance between objects, adhering to Grice's Maxim of Relation (prioritizing proximal objects when semantic conditions are equal).
• Edge Weight Prediction: A Multi-Layer Perceptron (MLP) takes the semantic relevance scores of two endpoint objects and their distance as input to predict an edge weight.

2. Training Strategy (Node-wise Supervision)

A major challenge is the lack of ground-truth annotations for pairwise relation importance in existing 3D-VL datasets (which typically only label the target object). To avoid the prohibitive cost of annotating all relations, CAPruner employs aggregated node-level supervision:

• Aggregation: Edge weights incident to a node are aggregated into a node weight ($v_i$) using a $p$-norm method: $v_i = \sigma(\sum_j w_{ij}^p)^{1/p}$.
• Loss Function: The model is trained using a Weighted Mean Squared Error (WMSE) loss. This loss encourages nodes corresponding to target objects to have high weights (close to 1) and non-target nodes to have low weights (close to 0).
• Backpropagation: The supervision signal propagates back through the aggregation operation, implicitly encouraging the model to assign higher weights to edges incident to target objects.

3. Inference

During inference, the model computes edge weights for a given scene and task. The scene graph is pruned by retaining the incident edges with the highest weights for each node, generating a task-specific subgraph that preserves critical relations while discarding redundancy.

Key Contributions

1. Identification of Pruning Requirements: The authors demonstrate through qualitative and quantitative analysis that LLM spatial reasoning relies heavily on explicitly retained relations. They establish that effective pruning must preserve task-critical relations rather than just spatially proximal ones.
2. CAPruner Architecture: The proposal of a lightweight pruning model that combines fuzzy semantic matching with spatial proximity. This approach avoids the "all-or-nothing" nature of strict matching and the rigidity of fixed KNN strategies.
3. Novel Supervision Mechanism: The development of a training framework that uses aggregated node scores to supervise edge weights, eliminating the need for expensive relation-level annotations.
4. Empirical Validation: Extensive experiments showing that CAPruner preserves essential spatial relations, leading to significant performance gains in downstream 3D-VL tasks.

Experimental Results

The authors evaluated CAPruner on three benchmarks: ScanRefer (3D Visual Grounding), ScanQA (3D Visual Question Answering), and SQA3D (Situated 3D VQA).

• Performance Gains: Using the same backbone LLMs (Llama-3.2-1B and Llama-3-8B), CAPruner consistently outperformed the baseline 3DGraphLLM (which uses proximity-based KNN).
  • On ScanRefer, CAPruner + Llama-3-8B achieved 61.7% accuracy (A.@0.25), surpassing the baseline 3DGraphLLM-8B (60.2%).
  • On ScanQA, it achieved 56.0% accuracy, compared to 54.6% for the baseline.
  • On SQA3D, it reached 55.3%, improving upon the baseline's 54.7%.
• Token Efficiency: CAPruner demonstrated higher token efficiency. For instance, it achieved better performance with an average of 582 tokens compared to the baseline's 882 tokens, saving approximately 34% in token usage.
• Pruning Strategy: Experiments comparing different pruning policies (KNN vs. MST) on the learned edge weights showed that the KNN strategy (retaining top-weighted edges per node) performed best, validating the approach of prioritizing local task-relevant regions over global graph connectivity.
• Ablation Studies:
  • Fuzzy Matching: Consistently outperformed strict matching and BERT-based embeddings, highlighting the importance of robust semantic relevance in the presence of 3D feature alignment noise.
  • Proximity vs. Semantics: While proximity aids identification, semantic relevance was found to be the dominant factor, aligning with the Maxim of Relation.
  • $p$-norm: Accuracy improved as $p$ increased up to 3, stabilizing thereafter, suggesting that high-weight edges are more critical than the aggregate sum of weights.

Significance and Claims

The paper claims that CAPruner offers a scalable and effective strategy for enhancing LLM-based spatial reasoning in 3D-VL tasks. By shifting the pruning paradigm from "spatial proximity" to "conceptual adjacency" (task-specific semantic relevance), the method ensures that the limited token budget is allocated to relations that are actually useful for answering the query.

The authors emphasize that CAPruner achieves these improvements with negligible inference overhead (0.75 ms per sample) and without requiring costly relation-level annotations. They conclude that task-specific scene graph pruning is a viable path to scaling 3D-VL systems, allowing LLMs to reason effectively even in complex scenes with hundreds of objects.

Limitations Acknowledged: The authors note that scene graphs may still struggle to represent highly complex, multi-hop relations (e.g., "the fourth chair... in the fifth row"), suggesting that more versatile data structures may be needed for such scenarios in the future.