Toward Low-Latency Vision-Language Models with Doubly-Correct Predictions in Egocentric Visual Understanding

Technical Summary: Toward Low-Latency Vision-Language Models with Doubly-Correct Predictions in Egocentric Visual Understanding

Problem Statement
The rapid adoption of Vision-Language Models (VLMs) in human-robot collaboration (HRC) and egocentric visual understanding faces a critical bottleneck: the high computational cost and inference latency of massive models hinder real-time, on-board processing. While weight pruning is a standard technique to reduce model size and computation, existing methods primarily optimize for prediction accuracy alone. This paper argues that in safety-critical physical robotics, accuracy is insufficient. A model must produce Doubly-Correct Predictions (DCP): outputs that are not only factually correct but also grounded in valid visual evidence (rationales).

The authors identify a specific failure mode in current pruning strategies: they often preserve the ability to identify valid evidence (e.g., the correct object location) but degrade the model's ability to use that evidence to make the correct prediction. This leads to "Right for the Wrong Reason" scenarios where a pruned model might correctly localize a wrench but fail to predict the "hand-over" action, or conversely, predict the correct action based on spurious correlations rather than the object itself. Such misalignment poses severe safety risks in embodied intelligence, potentially leading to task failure or human injury.

Methodology
The paper proposes a Rationale-Informed Pruning strategy designed to preserve DCP while reducing computational overhead. The methodology proceeds through the following steps:

1. Evaluation Protocol Definition: The authors define a spatio-temporal DCP evaluation protocol for egocentric videos.

   • Rationale Definition: Rationales are defined as the human-manipulated objects providing evidence for an action.
   • Dimensions: Correctness is evaluated in two dimensions: Spatial (localization of the object within a frame, measured via Relevant Mass Accuracy) and Temporal (identification of frames containing the object, treated as binary classification).
   • Metrics: The study introduces Prediction Trustworthiness (PT), Inference Reliability (IR), and the proportion of Doubly-Correct samples (RR) to diagnose pruning effects.

2. Diagnostic Findings: Experiments on existing post-training pruning methods (OMP, GMP, Kwon et al., ECoFLaP) reveal that while they may maintain high PT (valid rationales among correct predictions), they significantly degrade IR (correct predictions among valid rationales) and RR. Pruned models often retain valid evidence localization but fail to propagate this evidence to the final decision layer, resulting in incorrect predictions despite correct rationales.

3. Proposed Pruning Algorithm: To address this, the authors introduce a pruning framework that leverages the model's own generated rationales to guide weight selection without additional supervision.

   • Layer-wise Non-Uniformity: Pruning ratios are allocated non-uniformly across layers based on weight magnitudes, acknowledging that different layers contribute unequally to prediction.
   • Rationale-Guided Hessian: The core innovation modifies the standard Optimal Brain Surgeon (OBS) approach. Instead of using raw input activations to compute the Hessian matrix (which includes background noise), the method incorporates a rationale mask.
   • Semantic Modulation: The rationale mask is refined by computing the cosine similarity between the text embedding of the rationale object and the predicted class. This similarity score acts as a semantic factor to weight the mask, amplifying contributions from semantically aligned rationales and down-weighting noise.
   • Weight Scoring: The importance score for each weight is calculated using a second-order Taylor approximation of the loss, where the Hessian is constructed using the modulated rationale mask. This ensures that weights critical for connecting valid evidence to the correct decision are preserved.

Key Contributions

• Spatio-Temporal DCP Evaluation: The paper proposes a new evaluation protocol specifically for egocentric visual understanding that assesses both prediction accuracy and the validity of spatio-temporal rationales.
• Diagnostic Insight: Through extensive experiments, the authors uncover a previously unexplored risk in pruning: existing methods often decouple valid evidence from correct decisions, leading to unreliable VLM deployment in robotics.
• Rationale-Informed Pruning: A novel pruning algorithm that utilizes the model's own rationales to guide weight removal. This approach aligns evidence localization with decision-making, ensuring that reduced models retain the ability to produce doubly-correct predictions.

Experimental Results
The method was evaluated on standard egocentric video benchmarks: EPIC-KITCHENS VISOR and EgoExo4D, using ActionCLIP (ViT-B/32) as the base model.

• Performance: The proposed method consistently outperforms existing pruning baselines (OMP, GMP, Kwon et al., ECoFLaP) across all pruning ratios (20%, 25%, 30%).
• Accuracy: It achieves the highest prediction accuracy among all compared methods.
• Doubly-Correct Metrics: The method significantly improves Inference Reliability (IR) and the proportion of Doubly-Correct samples (RR). For instance, on EPIC-KITCHENS VISOR at a 30% pruning ratio, the method improved spatial IR by 13.62% and RR by 3.65% compared to the strong baseline ECoFLaP.
• Trustworthiness: The method successfully preserves Prediction Trustworthiness (PT), ensuring that correct predictions remain grounded in valid rationales.
• Ablation Studies: Experiments confirm that both the use of rationale masks and the semantic modulation of these masks are critical components; removing either leads to a significant drop in DCP performance.

Significance and Claims
The paper claims that achieving low-latency VLMs for robotics requires more than just computational efficiency; it demands safety and transparency. By focusing on doubly-correct predictions, the authors argue that their method bridges the gap between efficiency and reliability. The work suggests that pruning strategies must evolve to preserve the semantic link between evidence and decision. The authors aim to stimulate research into efficient VLMs that align accuracy-driven advances with the transparency, auditability, and safety required for responsible human-robot interaction and embodied intelligence. They emphasize that their findings challenge the assumption that pruning only affects efficiency and accuracy, highlighting a new dimension of risk where models become "right for the wrong reasons" or fail to utilize correct evidence.