Technical Summary: Fourier Features for Point Cloud Imitation Learning

Problem Statement

High-precision robotic manipulation requires fine-grained spatial reasoning to execute tasks such as inserting a peg into a socket or grasping small objects. While RGB-based policies are common due to semantic richness and pretrained encoders, they suffer from depth ambiguity and perspective scale issues, requiring the policy to implicitly infer 2D-to-3D mappings. Conversely, policies leveraging 3D information directly (e.g., point clouds) offer stronger geometric priors but often fail to achieve consistent high-precision performance across diverse tasks.

The authors hypothesize that this discrepancy stems from the spectral bias of neural networks. Multi-Layer Perceptrons (MLPs), which underpin most point cloud architectures, exhibit a tendency to learn low-frequency functions first, while high-frequency components (essential for sharp decision boundaries and fine geometric details) converge slowly or are not learned at all. This bias is particularly detrimental when policies must condition on slow-moving Cartesian features where neighboring points have nearly identical coordinates but require distinct actions.

Methodology

The paper proposes mapping point cloud coordinates from Cartesian space into a high-dimensional Fourier space to equip point cloud encoders with direct access to high-frequency features. This approach is adapted from Neural Radiance Fields (NeRFs) but applied systematically to diffusion-based Imitation Learning (IL) on point clouds.

Core Components

1. Fourier Feature Mapping:
   Instead of feeding raw Cartesian coordinates $(x, y, z)$ into the network, the authors apply a sinusoidal encoding function $\gamma: \mathbb{R} \to \mathbb{R}^{2L}$. For a coordinate $x$, the mapping generates a vector of sine and cosine terms with varying wavelengths $\lambda_k$:
   $$\gamma_k(x) = \left[ \sin\left(\frac{2\pi x}{\lambda_k}\right), \cos\left(\frac{2\pi x}{\lambda_k}\right) \right]^T$$
   The wavelengths are log-spaced between a maximum ($\lambda_{max}$) and minimum ($\lambda_{min}$). This transforms the input into a higher-dimensional space where subtle geometric differences between neighboring points are amplified, allowing the network to distinguish them more easily.

2. Integration with Diffusion Policies:
   The method is integrated into a score-based diffusion framework (Elucidated Diffusion Models). The policy $\pi_\theta(a|o, g)$ learns to denoise action sequences conditioned on scene observations. The Fourier mapping is applied to the point coordinates before they enter the encoder (e.g., PointPatch, PointTransformer, DP3).

3. Data Augmentation (VariableJitter):
   To prevent overfitting to specific frequency bands and ensure robustness, the authors employ VariableJitter. Unlike standard uniform jitter, this method samples a noise scale $\sigma$ from a uniform distribution for each point cloud, training the network to ignore frequencies that do not contain useful information without requiring careful per-task wavelength tuning.

4. Architectural Flexibility:
   The approach is designed to be architecture-agnostic. It is tested on various point cloud encoders, including:

   • PointPatch: Groups points into patches and tokenizes them.
   • PointPatch-attn: Uses attention pooling to aggregate tokens.
   • PCM (PointCloudMatters): Uses max pooling across patches.
   • DP3: Creates a single global token via max pooling.
   • PointTransformer: Uses hierarchical self-attention.
   • Hybrid Models: Combines point clouds with RGB streams.

Key Contributions

1. Systematic Evaluation: The paper provides a comprehensive evaluation of Fourier feature mappings across a broad range of point cloud architectures and benchmarks, demonstrating that this simple non-parametric mapping resolves spectral bias in diverse encoders.
2. Robust Performance Gains: Experiments show consistent improvements in success rates across simulation (RoboCasa, ManiSkill3) and real-world robot setups. The method improves performance without requiring additional regularization or complex architectural changes.
3. Insight into Learning Dynamics: Through spectral analysis and parameter studies, the authors demonstrate that Fourier features not only help extract fine geometric details but also improve learning dynamics, even in the absence of fine geometry. The method is shown to be robust to hyperparameter choices (e.g., number of frequency bands, minimum wavelength).

Experimental Results

The authors evaluated their method on three benchmarks:

• RoboCasa (Simulation): On 16 high-precision manipulation tasks (e.g., inserting mugs, turning knobs), Fourier features significantly boosted performance. For instance, the PointPatch encoder's success rate on the CloseDrawer task increased from 34% to 72%, and the overall average success rate rose from 13% to 34%.
• ManiSkill3 (Simulation): On four grasping and tool-use tasks, Fourier features provided consistent, though sometimes more modest, improvements (e.g., +7% average improvement), likely due to the relative simplicity of these tasks compared to RoboCasa.
• Real-World Robot: On four challenging real-world tasks (Drawer, Cup-Stacking, Arranging, Folding), policies with Fourier features achieved a normalized score improvement from 14.8% to 40.2%. Notably, the method proved robust to real-world noise, occlusion, and camera miscalibration. Qualitative results indicated that policies trained with Fourier features moved more decisively and imitated expert demonstrations more closely, whereas baseline policies often hesitated or failed to perceive fine geometric details.

Spectral Analysis:
Graph Fourier spectrum analysis revealed that while standard networks are biased toward low frequencies, adding Fourier features increases the network's sensitivity to high-frequency components (eigenvalues $\lambda > 1.0$). This shift allows the network to learn sharper decision boundaries. Furthermore, the benefit of Fourier features correlated with the richness of geometric detail in the point cloud; denser point clouds yielded larger performance gains.

Significance and Claims

The paper claims that Fourier features serve as a general-purpose tool for point cloud-based imitation learning. By counteracting the inherent spectral bias of MLPs, these features enable policies to leverage geometric details more effectively than standard Cartesian features.

The authors emphasize that this improvement is achieved with minimal cost:

• Simplicity: It requires only a change in input encoding, not a redesign of the policy architecture.
• Robustness: The method is insensitive to hyperparameter choices and does not require task-specific tuning of wavelength ranges.
• Universality: It benefits diverse encoder architectures and multimodal setups (combining RGB and point clouds).

The work suggests that for high-precision robotic tasks, encoding coordinate inputs via Fourier features should be considered a standard practice over raw Cartesian features, particularly when fine-grained spatial reasoning is required.