VADF: Vision-Adaptive Diffusion Policy Framework for Efficient Robotic Manipulation

1. Problem Statement

Current diffusion policies for robotic manipulation, while effective, suffer from two primary inefficiencies due to their reliance on uniform computational allocation:

• Training Inefficiency (Sample Imbalance): Standard diffusion training uses uniform sampling across timesteps and data instances. This fails to distinguish between "easy" samples (low information gain) and "hard negatives" (critical transitions or contact-rich trajectories). Consequently, the model wastes computation on trivial samples, leading to slow convergence.
• Inference Inefficiency (Rigid Scheduling): Deployment typically uses fixed denoising steps ($N_d$) and action horizons ($N_a$) regardless of task complexity. This "one-size-fits-all" approach causes redundant latency during simple phases (e.g., reaching) and insufficient precision during complex phases (e.g., insertion or fine manipulation), resulting in high latency and frequent inference timeouts.

The paper argues that robotic tasks exhibit inherent spatio-temporal heterogeneity (varying difficulty across time and space) that current methods fail to address dynamically.

2. Methodology: VADF Framework

The authors propose VADF (Vision-Adaptive Diffusion Policy Framework), a model-agnostic, plug-and-play dual-adaptive system that integrates vision-language understanding into diffusion policies. It operates in two distinct phases:

A. Training Phase: Adaptive Loss Network (ALN)

The ALN is designed to accelerate convergence by prioritizing difficult samples and timesteps.

• Learnable Timestep Sampler: Instead of uniform timestep sampling, ALN employs a lightweight MLP-based policy ($\pi_\phi$) that learns a non-uniform distribution over timesteps. It uses sinusoidal positional embeddings to predict which timesteps yield the highest information gain (high loss) for the current model state.
• Hard Negative Mining (Trajectory Re-weighting): ALN maintains a learnable weight vector for training trajectories. It calculates a "reward" based on the normalized loss of each trajectory. Trajectories with higher losses (hard negatives) are up-weighted using a WeightedRandomSampler, ensuring the model focuses optimization on critical transitions.
• Mechanism: The system uses a REINFORCE algorithm with entropy regularization to update the sampler policy, ensuring the sampling distribution adapts online to the evolving loss landscape without requiring auxiliary objectives.

B. Inference Phase: Hierarchical Vision Task Segmenter (HVTS)

The HVTS addresses inference latency by dynamically adjusting computational resources based on real-time visual context.

• Zero-Shot Task Decomposition: Leveraging a Vision-Language Model (VLM), HVTS decomposes high-level user instructions into a sequence of semantic sub-stages (e.g., "approach," "align," "push").
• Dynamic Scheduling ($N_d, N_a$): Based on the identified stage, HVTS adaptively assigns:
  • Simple Stages: Shorter denoising steps ($N_d$) and longer direct execution sequences ($N_a$) to reduce latency.
  • Complex Stages: Longer denoising steps ($N_d$) and shorter execution sequences to ensure high precision.
• Probabilistic Transition: To ensure smooth execution, the VLM evaluates the current scene against a pre-defined stage library and samples the active stage, updating the $(N_d, N_a)$ pair dynamically without abrupt policy jitter.

3. Key Contributions

1. Unified Adaptive Framework: VADF is the first framework to introduce vision-driven adaptive computation into diffusion policies, addressing both sample difficulty (training) and stage complexity (inference) simultaneously.
2. Model-Agnostic & Plug-and-Play: The design requires no modifications to the underlying diffusion architecture, no additional training stages for the base model, and no human annotations. It can be integrated into any existing diffusion policy (MLP-based or Transformer-based).
3. Dual-Module Design:
   • ALN: Accelerates training convergence by focusing on hard negatives.
   • HVTS: Enables zero-shot, dynamic inference scheduling to balance speed and precision.
4. Comprehensive Evaluation: Demonstrated across diverse benchmarks (RoboMimic, Kitchen, Adroit) and real-world hardware (ARX5 robotic arm).

4. Experimental Results

The authors evaluated VADF on multiple benchmarks, comparing it against vanilla Diffusion Policy (DP) and state-of-the-art variants (DDIM, DP3).

• Behavior Cloning (RoboMimic):
  • VADF achieved a 91.3% success rate on the MLP variant (vs. 85.4% baseline) and 86.8% on the Transformer variant (vs. 80.1% baseline).
  • Early Success Rate (success within fewer steps) improved significantly, indicating faster convergence to successful trajectories.
• Adroit Benchmark (Dexterous Manipulation):
  • Integrated with the 3D Diffusion Policy (DP3), VADF improved success rates by 10.0% (Door), 3.3% (Hammer), and 10.9% (Pen), demonstrating robustness under extreme data scarcity and occlusion.
• Efficiency Gains:
  • Training: VADF reached peak success rates with significantly fewer gradient steps compared to vanilla DP (see Fig. 3).
  • Inference: VADF achieved up to a 2.46× speedup in inference latency when combined with DDIM, reducing total latency from ~1995ms to ~1199ms on the Push-T task while maintaining or improving success rates.
• Ablation Studies:
  • Removing HVTS (keeping only ALN) provided modest gains on simple 2D tasks but failed to significantly improve complex long-horizon tasks (Kitchen).
  • Removing ALN (keeping only HVTS) showed that adaptive inference alone is insufficient for training efficiency.
  • The full framework demonstrated a synergistic effect, proving both modules are necessary for optimal performance.

5. Significance

VADF establishes a new paradigm for compute-efficient and interpretable robotic imitation learning.

• Scalability: By dynamically allocating resources, it allows diffusion policies to scale to more realistic, complex, and long-horizon manipulation scenarios without prohibitive computational costs.
• Generalization: Its model-agnostic nature makes it a versatile tool that can enhance existing diffusion-based systems without retraining the core policy.
• Bridging the Gap: It effectively bridges the gap between high-level semantic planning (via VLMs) and low-level dynamic execution, enabling robots to reason about task difficulty and adjust their "effort" in real-time, mimicking human motor control strategies.