Rotation-Aware Point-Cloud Embeddings for Vision-Based In-Hand Reorientation

Technical Summary: Rotation-Aware Point-Cloud Embeddings for Vision-Based In-Hand Reorientation

Problem Statement
The paper addresses the challenge of dexterous in-hand reorientation using point-cloud goals. While point clouds offer a natural way to specify a desired 3D geometry without requiring object-specific pose frames or 6D pose estimation, raw point-cloud goal conditioning is poorly suited for policy learning. The fundamental difficulty lies in the nature of point clouds: they are unordered, independently sampled, and visibility-dependent. Consequently, the discrepancy between a current observation and a goal cloud entangles object rotation with permutation, resampling, and unstable correspondence structures.

Prior approaches have attempted to bypass this by adding external structure, such as:

• Explicit pose or relative-pose inputs (e.g., Geometry-Dex).
• Dense flow features indicating point movement (e.g., HACMan).
• Distillation from privileged teachers who have access to ground-truth state.

These methods rely on brittle test-time computations (pose estimation, registration) or require expert supervision. The authors posit that generic point-cloud encoders (trained for reconstruction or shape identity) fail to organize task-relevant state in a way that facilitates control, often discarding the specific rotational signals needed for reorientation.

Methodology
The proposed solution is a rotation-aware point-cloud embedding that calibrates the Euclidean latent distance between embeddings to the SO(3) geodesic error between object orientations. This allows a model-free Reinforcement Learning (RL) policy to learn directly from current and goal point-cloud embeddings without explicit pose, flow, or teacher supervision.

1. Pose-Aware Encoder ($\phi$):

   • Architecture: A PointNet++ based encoder that maps colored point clouds ($P \in \mathbb{R}^{N \times 6}$) to 128-dimensional embeddings.
   • Training Objective: The encoder is trained using a metric-learning objective in a Siamese style. For two clouds of the same object at orientations $q_1$ and $q_2$, the Euclidean distance between their embeddings is regressed to the SO(3) geodesic rotation error:
     $$L_{metric} = \mathbb{E}[(\|\phi(P_1) - \phi(P_2)\|_2 - 2 \arccos |\langle q_1, q_2 \rangle|)^2]$$
   • Curriculum: Training utilizes an occlusion curriculum on YCB dataset textured meshes, progressively removing points to simulate partial observations. A fine-tuning stage uses an IsaacLab dataset with hand-like occlusions to match policy learning conditions.

2. Policy Learning (In-Hand Reorientation):

   • Framework: The task is formulated as a goal-augmented Markov Decision Process (MDP) solved using PPO with an asymmetric actor-critic architecture.
   • Observations (Actor): The policy receives:
     • Learned embeddings of the current and goal point clouds ($z_{obj}, z_{goal}$).
     • Proprioception (joint positions, velocities, fingertip poses).
     • Geometric metadata (centroids of current and goal clouds) to supplement translation information.
   • Critic: The critic has access to privileged state information (ground-truth pose, velocity, contact forces) to facilitate faster convergence.
   • Reward: The reward function minimizes in-hand position drift and rotational error (measured via geodesic distance on SO(3)), with action regularization and a success bonus.

Key Contributions

• Representation Learning for Control: The paper demonstrates that a representation itself can make point-cloud goal conditioning usable for standard policy learning, removing the need for external scaffolding like pose estimators or dense flow fields.
• SO(3)-Calibrated Embeddings: By explicitly training the encoder to align latent distance with rotational error, the method creates a smooth control signal where the difference in embeddings directly reflects the rotation needed to reach the goal.
• Model-Free RL Success: The authors show that a model-free policy using these embeddings can learn stably and match the performance of distillation-based baselines and privileged-state methods, despite lacking access to ground-truth pose or teacher actions during training.
• Diagnostic Evidence: The paper provides evidence that generic self-supervised pretraining (e.g., Point-MAE) is insufficient for this task because it preserves shape features but fails to organize the latent space according to rotation axes and magnitudes, whereas the proposed metric-learning approach recovers this structure.

Results
Experiments were conducted in simulation using the Shadow Hand and four YCB objects (Peach, Rubik's Cube, Pear, Tuna Can).

• Performance: The proposed method achieved 100% success rates across all tested objects, matching the performance of the "Teacher" (privileged state) and "Distilled Student" baselines.
• Efficiency: While the proposed method succeeded, it required more timesteps on average to reach the goal compared to the privileged teacher (e.g., 85.29 steps vs. 25.40 for the Peach). The authors attribute this to the model-free nature of the learning (optimizing from reward rather than per-step actions) and note that longer training runs could further improve speed.
• Comparison to Baselines:
  • Methods relying on pose estimators (e.g., FoundationPose) or registration (Open3D) showed significantly lower success rates (e.g., 33-58% for the Rubik's Cube and Pear) and much higher failure times, highlighting the brittleness of these intermediate modules in deployment.
  • Flow-based methods using simulated state performed well, but those relying on registration failed significantly.
  • Generic Encoders: A policy using a fine-tuned Point-MAE encoder failed to learn entirely, confirming that generic shape features do not provide the necessary rotation-aware signal.
• Representation Analysis: Latent space visualizations (surface plots and t-SNE) confirmed that the proposed encoder organizes embeddings by rotation axis and magnitude, creating a structured landscape similar to that learned by a distillation-based student, but without requiring action supervision.

Significance and Claims
The paper claims that point-cloud goals become practical for dexterous control when the representation itself encodes the task-relevant geometry of rotation, rather than relying on external modules to compute pose or flow.

• The work suggests that the "scaffolding" of pose estimation and registration is not strictly necessary if the visual representation is correctly calibrated to the task's geometric constraints.
• It highlights a limitation of generic visual pretraining: preserving shape identity is insufficient for control tasks requiring precise geometric comparison; the representation must be explicitly structured for the specific transformation group (SO(3) in this case).
• The authors note that while the evaluation is in simulation, the use of occlusion curricula, point jittering, and domain randomization provides evidence that the representation is robust to the perturbations expected in real-world deployment, motivating future real-robot validation.

Limitations
The authors acknowledge that the current embedding primarily preserves rotation-relevant state and is not explicitly constrained to preserve object shape, which limits the ability to learn a single general policy across different objects. Future work is suggested to explore embeddings that jointly preserve shape and enable control-friendly comparisons, as well as extending the metric objective to the full SE(3) group for general manipulation.