Hyper-DP3: Frequency-Aware Right-Sizing of 3D Diffusion Policies for Visuomotor Control

Technical Summary: Hyper-DP3 (HDP3)

Problem Statement
Current diffusion-based visuomotor policies for robotic manipulation, particularly those extending to 3D point clouds (e.g., DP3), suffer from significant overdesign. These methods typically inherit architectures from image generation, utilizing heavy conditional decoders (such as U-Net or DiT) and requiring multiple denoising steps (often 10–100) at inference time. The authors argue this design is mismatched to the nature of robot action trajectories. Unlike perceptual signals (images) which contain dense high-frequency details, robot action trajectories are highly smooth and temporally coherent. Consequently, existing approaches incur unnecessary computational latency and parameter overhead without a corresponding benefit in control performance.

Methodology
The proposed solution, Hyper-DP3 (HDP3), rethinks the diffusion policy design from a frequency-domain perspective. The methodology consists of three core components:

1. Frequency-Domain Analysis:
   The authors analyze robot action trajectories using the Discrete Cosine Transform (DCT). They demonstrate that the energy of these trajectories is overwhelmingly concentrated in the lowest frequency modes. Specifically, in their experiments, the first two DCT modes account for nearly 98.5% of the total energy, with the first mode alone holding 93.2%. This indicates that robot actions lie on a low-dimensional manifold in the frequency domain.

2. Theoretical Justification for Two-Step Inference:
   Building on the observation of low-frequency dominance, the paper provides a theoretical bound on the optimal denoiser's error. The analysis shows that the mean squared error (MSE) of the optimal estimator is bounded by the dimension of the low-frequency subspace and the residual high-frequency energy. Because the high-frequency energy is negligible, the error saturates rapidly. The authors derive that for such data, a two-step DDIM (Denoising Diffusion Implicit Models) sampler is sufficient to drive the decoding error to a level that has a negligible effect on closed-loop execution. The remaining high-frequency error is naturally filtered out by the robot's low-level controller.

3. Lightweight Architecture (Diffusion Mixer):
   Motivated by the finding that the score field for action denoising is simpler than that for high-dimensional perceptual signals, HDP3 replaces the heavy U-Net/DiT decoders with a Diffusion Mixer (DiM) decoder.

   • Structure: The DiM is based on the MLP-Mixer architecture. It processes the noisy trajectory tokens by alternating between temporal MLPs (aggregating information across time) and channel MLPs (aggregating across feature dimensions).
   • Conditioning: It integrates observation conditions (3D point clouds and robot proprioceptive states) via FiLM (Feature-wise Linear Modulation) layers with gated residual connections.
   • Efficiency: This design allows for efficient information fusion with a drastically reduced parameter count, supporting two-step inference without requiring consistency distillation or MeanFlow training.

Key Contributions

• Frequency-Domain Insight: The paper establishes that robot action trajectories are strongly low-frequency-dominant, revealing a fundamental mismatch in existing 3D diffusion policies that rely on image-generation-style decoders and multi-step sampling.
• Theoretical Analysis: It provides a frequency-domain theoretical proof showing that denoising error saturates quickly for smooth trajectories, justifying the sufficiency of very few reverse steps (specifically two) for effective action generation.
• Model Design: The authors propose HDP3, a "pocket-scale" 3D diffusion policy featuring a lightweight DiM decoder. This model supports efficient two-step DDIM inference.
• Empirical Validation: The design is validated on synthetic data, three simulation benchmarks (RoboTwin2.0, Adroit, MetaWorld), and real-world tasks, demonstrating that high performance can be achieved with a fraction of the parameters of prior methods.

Experimental Results

• Performance: Across RoboTwin2.0, Adroit, and MetaWorld, HDP3 achieves state-of-the-art performance. For instance, on RoboTwin2.0, it achieves an average success rate of 63.2% (outperforming the previous best, DP3, by 8.0 percentage points). On Adroit and MetaWorld, it achieves a 78.4% average success rate, surpassing the strong baseline MP1 (76.7%).
• Efficiency: HDP3 uses fewer than 1% of the parameters of prior 3D diffusion-based policies (e.g., 2.52M parameters vs. 255.8M for DP3).
• Latency: Due to the compact model and two-step inference, HDP3 achieves a significantly lower inference latency (4.50 ms) compared to prior methods (e.g., DP3 at 51.4 ms). While some one-step methods (Flow Policy, MP1) have low latency, they retain massive parameter counts (256M).
• Real-World Transfer: In real-world experiments on an AgileX Piper robot, HDP3 outperformed the DP3 baseline by an average of 15.3% in success rate, validating its sim-to-real transferability despite the simplified architecture.

Significance and Claims
The paper claims that the design of diffusion policies for robotics has been unnecessarily constrained by image-generation paradigms. By recognizing the intrinsic low-frequency nature of robot trajectories, HDP3 demonstrates that:

1. Complexity Reduction is Possible: The score field for action denoising is substantially simpler than for perceptual signals, allowing for lightweight architectures.
2. Sampling Acceleration is Natural: For smooth trajectories, multi-step denoising offers diminishing returns; two steps are theoretically and empirically sufficient.
3. Efficiency Does Not Sacrifice Quality: A model with <1% of the parameters of prior SOTA methods can achieve superior performance and lower latency, making real-time, on-device deployment on resource-constrained robotic platforms feasible.

The authors conclude that HDP3 represents a shift toward "right-sizing" diffusion policies, aligning model capacity and inference steps with the actual information content of robotic control tasks.