Test-Time Perturbation Learning with Delayed Feedback for Vision-Language-Action Models

1. Problem Statement

Vision-Language-Action (VLA) models have shown remarkable success in sequential decision-making tasks (e.g., robotic manipulation). However, they exhibit brittleness when faced with subtle environmental shifts (e.g., slight changes in object pose or lighting).

The authors identify the root cause of this failure as Trajectory Overfitting:

• Spurious Correlation: VLAs over-attend to non-semantic correlations between actions and specific entities (e.g., background textures, gripper appearance) learned during training rather than understanding the true semantic goal.
• Memorization: Instead of reasoning, the model reproduces memorized action patterns. Even if the target object is masked or the environment changes slightly, the agent executes the same trajectory, leading to task failure.
• Limitations of Existing Solutions:
  • Verifier-based TTA: Requires expensive pretraining of verifiers and repeated rollouts, causing high computational overhead.
  • Verifier-free TTA (Self-supervised): Often relies on entropy minimization. If the model is overconfident in incorrect behaviors, entropy minimization can amplify these errors rather than correcting them.

2. Methodology: Perturbation Learning with Delayed Feedback (PDF)

The authors propose PDF, a verifier-free Test-Time Adaptation (TTA) framework. It improves decision performance without fine-tuning the base model parameters. The framework consists of two core components:

A. Uncertainty-Based Action Voting

To mitigate trajectory overfitting without incurring constant computational costs, PDF uses an adaptive data augmentation strategy:

1. Uncertainty Estimation: The model calculates the predictive uncertainty of its current decision using the normalized Shannon entropy of the action logits.
2. Adaptive Augmentation Budget: The number of augmented views ($N_t$) generated for a specific state is proportional to the estimated uncertainty ($N_t = N_{max} \cdot U_t$). High-uncertainty states trigger more augmentations.
3. Voting Mechanism: The model processes the original observation and the augmented views. The final action is selected via dimension-wise majority voting (voting on each action dimension independently) rather than voting on full action tuples. This allows for flexible deviations from the original policy.

B. Delayed Feedback-Guided Adaptation

To correct overconfident decisions and stabilize the policy, PDF introduces a lightweight Perturbation Head ($h_\theta$):

1. Architecture: A small, learnable module is added to the frozen VLA. It takes the multimodal features and outputs a perturbation to the action logits.
2. Delayed Feedback: After an episode completes, the agent receives a scalar feedback signal ($r$) indicating task success/failure or cumulative reward.
3. Optimization Objective: The Perturbation Head is updated using a custom loss function ($L_{PDF}$) that combines:
   • REINFORCE-style Term: Increases the likelihood of actions if the feedback $r$ exceeds a baseline $b$.
   • KL Regularization: Applied only when $r > b$ to prevent the model from drifting too far from the original policy, ensuring stability.
   • Gating: The KL term is gated by an indicator function $I[r > b]$, allowing flexible exploration when feedback is negative but enforcing stability when feedback is positive.

Key Constraint: Only the Perturbation Head ($h_\theta$) is updated during test time. The visual encoder, transformer, and LM head remain frozen, ensuring efficiency.

3. Key Contributions

1. Diagnosis of Trajectory Overfitting: Systematically identified and analyzed the phenomenon where VLAs rely on spurious correlations rather than semantic grounding, leading to failure under environmental shifts.
2. PDF Framework: Proposed a novel, verifier-free TTA method that combines uncertainty-based adaptive augmentation with delayed-feedback-guided logit perturbation.
3. Efficiency and Stability: Demonstrated a method that improves robustness without fine-tuning the massive base model, using only a lightweight perturbation module and minimal computational overhead.

4. Experimental Results

The method was evaluated on two diverse benchmarks: LIBERO (robotic manipulation) and Atari-57 (visual control games).

• LIBERO Benchmark:
  • PDF achieved a 7.4% increase in average success rate compared to the vanilla OpenVLA.
  • It outperformed all baselines, including other TTA methods (e.g., MG-Select) and strong pretrained models (e.g., TraceVLA, SFT-4LIBERO), despite having no fine-tuning of the base model.
  • Gains were most significant in long-horizon tasks (+4.1% over the best baseline).
• Atari Benchmark:
  • PDF improved the Human-Normalized Score (HNS) by 10.3% (from 0.97 to 1.07).
  • Performance improved in 47 out of 57 games.
  • Notable gains in games like Boxing (+60.25% relative improvement).
• Ablation Studies:
  • Delayed Feedback (DF): Critical for robustness; removing it caused a significant drop in performance, especially in object manipulation tasks.
  • Data Augmentation (DA): Essential for mitigating overfitting; however, excessive augmentation (budget > 3) introduced noise and degraded performance.
  • Loss Components: Both the REINFORCE term and KL regularization were shown to be complementary; removing either degraded performance.

5. Significance

• Reliability in Real-World Deployment: PDF addresses a fundamental weakness of VLAs (brittleness to distribution shifts) without requiring expensive retraining or external verifiers.
• Practicality: By keeping the base model frozen and only updating a tiny perturbation module, PDF offers a computationally efficient path to reliable test-time adaptation.
• Paradigm Shift: The paper moves away from "entropy minimization" (which can fail on overconfident models) toward "feedback-guided perturbation," offering a more robust mechanism for correcting decision-making errors in embodied AI.

The code is available at: https://github.com/zhoujiahuan1991/CVPR2026-PDF.