2D or 3D: Who Governs Salience in VLA Models? -- Tri-Stage Token Pruning Framework with Modality Salience Awareness

This paper proposes a tri-stage token pruning framework for multi-visual-modal Vision-Language-Action (VLA) models that dynamically captures 2D/3D modality salience differences to achieve up to a 2.55x inference speedup with minimal accuracy loss.

The Gist

The Big Picture: The Robot's Overloaded Brain

Imagine a robot that needs to learn how to do chores, like closing a fridge or picking up a banana. To do this, it uses a "Brain" called a VLA Model (Vision-Language-Action).

• The Old Way (2D Only): The robot used to look at the world like a person looking at a photograph. It saw flat images. This was okay, but it was hard for the robot to judge depth (how far away the fridge handle is).
• The New Way (2D + 3D): To get better, scientists gave the robot a second pair of eyes that sees in 3D (like a depth map or a point cloud). Now, the robot can see exactly where things are in space.

The Problem: Giving the robot 3D vision is like giving it a superpower, but it comes with a heavy price. The robot now has to process twice as much information (flat photos + 3D data). It's like trying to read a book while someone is shouting a second story in your ear. The robot gets overwhelmed, thinks too slowly, and can't move fast enough to be useful in the real world.

The Solution: The "Tri-Stage" Smart Filter

The authors of this paper realized that existing methods to speed up robots were like using a blunt knife: they just chopped out random pieces of information to save time. But this was dangerous because the robot might cut off the important 3D data it needed to grab a cup, or the important 2D data it needed to read a label.

They asked a simple question: "Who is the boss right now? Is the 2D image more important, or is the 3D depth map more important?"

They found that the answer changes depending on what the robot is doing and where it is in time. So, they built a Tri-Stage Token Pruning Framework. Think of this as a highly intelligent bouncer at a club who decides who gets to stay in the VIP room (the robot's brain) and who gets kicked out, based on three different rules.

---

The Three Stages of the "Bouncer"

Stage 1: The Data Preprocessing (The "Raw Material" Check)

• The Analogy: Imagine you are a chef preparing a meal. You have a pile of vegetables (2D images) and a pile of spices (3D data).
• The Insight: The researchers found that for some tasks, the vegetables are the main dish, and the spices are just a garnish. For other tasks, the spices are the main flavor.
• The Action: The bouncer looks at the raw ingredients. If the 3D data is just "noise" (like extra background clutter), the bouncer throws it away immediately. If the 2D image is blurry but the 3D shape is clear, the bouncer keeps the 3D and tosses the 2D.
• Result: They set two "thresholds" (rules) to decide: "Keep only 2D," "Keep only 3D," or "Keep both."

Stage 2: The Semantic Synthesis (The "Context" Check)

• The Analogy: Now the robot is looking at a specific scene, like a kitchen counter. The scene has three parts: the Background (the wall), the Robot Arm (the tool), and the Target Object (the banana).
• The Insight:
  • Background: The wall doesn't need 3D depth. It's just a flat wall. Throw away the 3D data here!
  • Robot Arm: The arm needs to know its own shape in 3D to avoid crashing. Keep the 3D!
  • Target (Banana): To pick up the banana, you need to see its color (2D) and its shape (3D). Keep both!
• The Action: The bouncer divides the scene into these three zones. In the "Background" zone, they aggressively cut out 90% of the data. In the "Target" zone, they protect everything.

Stage 3: The Action Iteration (The "Time" Check)

• The Analogy: Imagine the robot is reaching for the banana.
  • Step 1: The arm is far away. It mostly needs to see the general shape (3D).
  • Step 2: The arm is close. It needs to see the texture of the banana peel to know if it's ripe (2D).
  • Step 3: The arm grabs. It needs precise depth again.
• The Insight: The importance of 2D vs. 3D changes moment by moment. If the robot decides what to keep based only on the current second, it might get "jittery" (switching back and forth too fast).
• The Action: The bouncer uses a "Sliding Window" (like looking through a history book). It remembers what happened in the last few seconds. If the robot was relying on 3D a moment ago, it assumes it probably still needs it, smoothing out the decision so the robot doesn't panic and switch modes too quickly.

---

The Magic Result

By using this three-step smart filter, the robot can:

1. Throw away the junk: It removes up to 70-80% of the unnecessary data.
2. Keep the gold: It keeps the exact 2D or 3D data needed for the specific moment and specific object.
3. Run faster: The robot becomes 2.55 times faster.

The Bottom Line:
Before this paper, robots with 3D vision were too slow to be practical. This new method is like giving the robot a pair of sunglasses that automatically tint the right lens (2D or 3D) depending on what it's looking at. The robot is now fast enough to do real-time tasks without losing its ability to see the world clearly.

In short: They taught the robot when to look with its 2D eyes, when to look with its 3D eyes, and when to close one eye to save energy, all without dropping the banana.

Technical Summary

1. Problem Statement

Vision-Language-Action (VLA) models are the mainstream for embodied intelligence. While early models relied solely on 2D images (SVLA), state-of-the-art models now integrate 3D modalities (e.g., point clouds) to enhance spatial perception, forming Multi-Visual-Modal VLAs (MVLA).

However, this expansion creates a critical bottleneck:

• Computational Overhead: Integrating 3D data significantly increases the number of input tokens (e.g., doubling from 256 to 512), leading to high inference latency (3–5 Hz), which fails to meet real-time robotic requirements (20–30 Hz).
• Ineffectiveness of Existing Pruning: Current token pruning methods are designed for 2D-only models. They fail to account for the salience discrepancies between 2D and 3D modalities. Blindly pruning tokens based on 2D logic leads to severe performance degradation in MVLA models because 2D and 3D tokens contribute differently depending on the task stage and semantic context.

2. Methodology: Tri-Stage Token Pruning Framework

The authors propose a novel framework that analyzes and prunes tokens across the three distinct stages of the MVLA inference pipeline: Data Preprocessing, Semantic Synthesis, and Action Iteration.

A. Tri-Stage Salience Analysis

The framework first establishes when and which modality governs salience:

1. Data Preprocessing Stage:
   • Finding: 2D modality generally exhibits higher salience than 3D in raw feature representation.
   • Metric: Proposed a quantitative metric ($MS^1$) based on the $L_1$ norm of final-layer features to measure modality contribution.
2. Semantic Synthesis Stage:
   • Finding: Salience varies by semantic region. 3D modality becomes significantly more salient in Target Object and Robot Body regions (crucial for geometry/collision), while 2D dominates in Background regions.
   • Metric: Proposed a decomposition mechanism ($MS^2$) using attention scores to separate 3D features into "parallel" (redundant with 2D) and "orthogonal" (unique 3D information) components.
3. Action Iteration Stage:
   • Finding: Modality salience is temporally dynamic. The reliance on 2D vs. 3D shifts as the robot executes a task (e.g., needing 3D for grasping but 2D for navigation).
   • Mechanism: Requires a prediction mechanism to adapt pruning budgets over time.

B. The Pruning Framework (Algorithm)

Based on the analysis, the framework implements a cascaded pruning strategy:

1. Stage 1: Data Preprocessing (Dual-Threshold Candidate Determination)

   • Uses the feature norm ratio ($MS^1$) to classify patches.
   • Applies dual thresholds ($\tau_{2D}, \tau_{3D}$) to decide retention:
     • Low 3D ratio $\rightarrow$ Keep only 2D.
     • High 3D ratio $\rightarrow$ Keep only 3D.
     • Intermediate $\rightarrow$ Keep both.

2. Stage 2: Semantic Synthesis (Semantic-Baseline Candidate Determination)

   • Clusters attention scores into three semantic sets: Background ($S_{bg}$), Robot ($S_{rob}$), and Object ($S_{obj}$).
   • Background: Aggressively pruned (90% random drop) as it is low-salience.
   • Robot/Object: Retention is adaptive. If 3D orthogonal features exceed a baseline, both modalities are kept; otherwise, 2D is prioritized.

3. Stage 3: Action Iteration (Temporal Dynamics & Fusion)

   • EMA Smoothing: Uses an Exponential Moving Average (EMA) with a sliding window to smooth pruning decisions across frames, preventing flickering and leveraging temporal continuity.
   • Fusion: Intersects the candidates from Stage 1 (Modality) and Stage 2 (Semantic).
   • Conflict Resolution: If the intersection is empty (e.g., a critical semantic region is flagged for pruning by modality metrics), the system defaults to the semantic requirement to prevent feature loss.
   • Cold Start: Handles the first few frames without historical data by using raw observations before switching to EMA.

3. Key Contributions

1. Tri-Stage Analysis: First comprehensive analysis revealing that 2D/3D modality salience is not static but varies by data stage, semantic region, and temporal iteration.
2. Modality-Aware Pruning Framework: A novel framework that automatically selects optimal pruning configurations (2D-only, 3D-only, or both) based on the specific context of the MVLA inference.
3. Efficiency vs. Accuracy: Achieves massive speedups with minimal accuracy loss by intelligently removing redundant tokens rather than applying uniform pruning.

4. Experimental Results

The framework was evaluated on the RLBench simulation benchmark and real-world robotic tasks using the MLA model.

• Speedup: Achieved up to 2.55× inference speedup compared to the unpruned baseline.
• Accuracy: Maintained high task success rates (SR). At a 70% pruning rate, SR dropped by only 2.5% (from 48.8% to 46.3%), whereas naive pruning caused catastrophic failure (e.g., dropping to 6.7% SR).
• Comparison: Significantly outperformed SOTA baselines (SP-VLA, VLA-Pruner) which were adapted from 2D-only models. For example, on the "Close Box" task at 60% pruning, the proposed method achieved 60.0% SR vs. 16.7% for SP-VLA.
• Overhead: The pruning mechanism itself adds only ~5.8% overhead (61ms per step), which is negligible compared to the latency reduction.
• Real-World Validation: Tested on a Songling Piper robotic arm, confirming that 3D pruning rates decrease during key interaction phases (grasping), aligning with the theoretical analysis.

5. Significance

This paper addresses a critical gap in the deployment of advanced embodied AI. By recognizing that 2D and 3D modalities are not interchangeable and that their importance fluctuates dynamically, the authors provide a scalable solution to the "token explosion" problem in MVLA models. The framework enables real-time robotic control (20–30 Hz) without sacrificing the spatial understanding gained from 3D data, paving the way for more efficient and capable embodied intelligence systems.