Revisiting Parameter Redundancy in Vision-Language-Action Models: Insights from VLM-to-VLA Adaptation

Technical Summary: Revisiting Parameter Redundancy in Vision-Language-Action Models

Problem Statement
Vision-Language-Action (VLA) models have advanced embodied intelligence by adapting large-scale pre-trained Vision-Language Models (VLMs) to robotic control tasks. However, these models inherit massive parameter scales, creating significant computational and storage burdens. Current efficiency optimization efforts, particularly parameter pruning, face a critical bottleneck: VLA models exhibit extreme sensitivity to parameter removal. Even moderate pruning often leads to a "performance cliff," necessitating post-hoc recovery mechanisms like Supervised Fine-Tuning (SFT) or Low-Rank Adaptation (LoRA) to restore functionality.

The authors challenge the prevailing engineering paradigm that treats this performance degradation as an inevitable cost of pruning. They argue that the widespread reliance on recovery masks a fundamental issue: if a pruned model must re-learn to function, the removed parameters were likely not redundant but vital. This raises the question of whether current pruning criteria indiscriminately remove critical parameters undergoing functional reorganization during the VLM-to-VLA adaptation process.

Methodology
The paper proposes a diagnostic framework that reframes VLA pruning not as an optimization goal, but as a controlled intervention to analyze parameter functional roles. The methodology consists of three core components:

1. Quantifying Adaptation Signals: The authors define the relative parameter divergence ($\Delta W$) between the pre-trained VLM weights ($W^{VLM}$) and the adapted VLA weights ($W^{VLA}$). They visualize the spatial distribution of this divergence across different modules (Vision Encoders, Language Backbones, and Cross-Modal Projectors) to identify structured patterns rather than random noise.
2. Controlled Pruning as a Diagnostic Probe: To establish a causal link between divergence signals and functional importance, the authors introduce a "recovery-free" evaluation protocol. They apply pruning directly to the model without subsequent fine-tuning. They compare two strategies:
   • Highest-difference pruning: Removing parameters with the largest $|\Delta W|$.
   • Lowest-difference pruning: Removing parameters with the smallest $|\Delta W|$.
     By observing the immediate impact on task success rates (SR), they determine which divergence signals correlate with redundancy.
3. Multi-Module Joint Pruning Scheme: Based on the discovery that redundancy signals vary significantly across modules (modular heterogeneity), they design a differentiated pruning strategy. This scheme applies specific pruning rules to Vision Encoders, Language Backbones, and Projectors based on their unique divergence characteristics, aiming to reduce parameters without triggering performance collapse.

Key Contributions

1. Diagnostic Framework: The authors introduce a novel framework that treats pruning as a controlled intervention to causally link adaptation-induced divergence with parameter functional importance, decoupling redundancy identification from performance recovery.
2. Revelation of the "Recovery Trap": The study demonstrates that the prevailing reliance on post-pruning recovery conceals the indiscriminate removal of vital parameters. They show that $\Delta W$ signals exhibit strong module heterogeneity, meaning a uniform pruning strategy is ineffective.
3. Recovery-Free Pruning Strategy: They propose a multi-module joint pruning scheme that leverages adaptation signals to achieve significant compression (12%–30% parameter reduction) while maintaining approximately 90% of the original performance, entirely without post-pruning fine-tuning or compensation.

Experimental Results
The approach was evaluated on the LIBERO benchmark using two representative model pairs: OpenVLA (based on Prismatic/Llama-2) and $\pi0.5$ (based on PaLI-Gemma).

• Hypothesis Validation:
  • Hypothesis I: Experiments confirmed that models requiring recovery after pruning had likely lost vital parameters. Models pruned using "Lowest-diff" strategies collapsed immediately (e.g., 0.0% SR) but recovered after LoRA fine-tuning, proving the removed parameters were not redundant.
  • Hypothesis II & III: The utility of $\Delta W$ signals varies by module. For instance, in OpenVLA's DINOv2, pruning highest-difference attention heads caused collapse, while pruning lowest-difference FFN channels caused collapse. Conversely, in the Projector module, both strategies led to collapse, indicating it is a fragile bottleneck requiring protection.
  • Hypothesis IV: The proposed multi-module scheme successfully reduced parameters without recovery.
• Performance Metrics:
  • OpenVLA: The "Moderate" configuration reduced parameters from 7.5B to 6.2B (12.4GB memory) while maintaining a 62.3% average success rate on LIBERO, compared to a catastrophic failure (1.0% SR) for baseline methods like LLM-Pruner under the same recovery-free conditions.
  • $\pi0.5$: The "Moderate" configuration reduced memory from 7.3GB to 5.6GB, maintaining an 89.0% average success rate (down from a baseline of 96.9%), whereas existing criteria resulted in total performance collapse.

Significance and Claims
The paper claims to provide a new path for deploying efficient, robust robotic policies in resource-constrained environments. By shifting the focus from "how to prune more and recover" to "how to identify true redundancy via adaptation signals," the authors demonstrate that VLA models contain significant, structured redundancy that can be safely removed without expensive recovery mechanisms. The study suggests that understanding the parameter evolution from general perception to embodied action is crucial for efficient VLA deployment, offering a principled alternative to the current "prune-then-recover" paradigm.