Technical Summary: AnyMug – Pose-Agnostic Robotic Functional Grasping via Observation-Action Canonicalization

1. Problem Formulation

The paper addresses the challenge of functional robotic grasping, specifically targeting the task of grasping mug handles. Unlike generic grasping, functional grasping requires the robot to interact with specific parts of an object (the handle) to enable subsequent tasks (e.g., pouring). This task is complicated by:

• Geometric Variability: Handles are thin, curved, and vary significantly across different mug instances.
• Pose Sensitivity: Mugs can appear in different positions, handle orientations, and two distinct pose categories: upright and inverted.
• Limitations of Existing Methods:
  • Grasp Pose Detection (Open-loop): Methods that predict a 6-DoF or 7-DoF grasp pose are highly sensitive to estimation errors, particularly on thin handle structures.
  • Learned Visuomotor Policies (Closed-loop): Standard policies must implicitly learn to handle the coupled variation between visual appearance and action direction for every possible placement, limiting generalization and requiring the learning of redundant, placement-dependent strategies.

The goal is to learn a single closed-loop policy that generalizes across diverse mug geometries and pose categories while maintaining precise contact with the handle.

2. Methodology: AnyMug Framework

The authors propose AnyMug, a reinforcement learning (RL) framework trained entirely in simulation (Isaac Sim) and deployed zero-shot on a real Franka Panda robot. The core innovation is Observation-Action Canonicalization.

2.1 Observation-Action Canonicalization

Instead of training a policy to handle raw, variable inputs, AnyMug transforms both the sensory input and the control output into a shared, mug-centric canonical frame.

• Observation Canonicalization:
  • The raw top-view depth image ($D_t$) is processed to identify the mug center and handle center.
  • An affine warp is applied to the image to: (1) center the mug in the image, and (2) rotate the handle to a fixed canonical direction (e.g., pointing left).
  • This results in a canonical depth image ($D^c_t$) where the mug always appears in the same location and orientation, regardless of its physical placement.
• Action Canonicalization:
  • The policy predicts a 7D end-effector action ($a^c_t$) in the canonical frame, consisting of translational displacement ($\Delta p^c$), rotational displacement ($\Delta \omega^c$), and a gripper command ($u_g$).
  • Before execution, the spatial components of the action are rotated back to the world frame using the inverse of the handle's rotation angle ($\theta_t$). The gripper command remains unchanged.
  • This allows the policy to learn a single "grasp strategy" that is reused across all mug placements and handle orientations.

2.2 Handle-Aware Reward Function

To ensure the policy learns precise functional grasping rather than just generic reaching, a dense, handle-aware reward function is employed:

• Approach Reward ($r_{pos}$): Encourages the gripper to reach the handle center.
• Orientation Reward ($r_{rot}$): Encourages the gripper to align with a target orientation relative to the mug.
• Finger Opposition Reward ($r_{opp}$): A critical task-specific term that rewards the two fingertips being on opposite sides of the handle before closure. This is gated by proximity to ensure the fingers are close to the handle.
• Action Penalty ($r_{act}$): Penalizes large, jerky movements to encourage smooth trajectories.

2.3 Training Strategy

• Algorithm: Proximal Policy Optimization (PPO).
• Pose Curriculum: Training begins with a bias toward upright mugs (70%) to establish reliable approach behavior, then transitions to a balanced distribution (50/50) for upright and inverted mugs.
• Domain Randomization: Randomization of mug positions, handle yaws, and mug geometries. Crucially, Gaussian noise is added to pose estimates to simulate real-world perception errors, calibrated from a state-of-the-art pose estimator (CPPF++).

3. Key Contributions

1. Observation-Action Canonicalization: The paper formulates mug grasping as a canonicalized problem where both depth observations and end-effector actions are represented in a shared mug-centric frame. This reduces pose-induced variation in both perception and control, allowing a single policy to reuse a shared strategy.
2. Closed-Loop Handle-Aware Grasping: The authors train a single RL policy using canonicalized visual-proprioceptive feedback and a specialized reward function. This enables precise approach, alignment, and closure timing across diverse geometries and poses without explicit motion planning.
3. Comprehensive Evaluation: The framework is benchmarked against motion planning and diffusion-based baselines in simulation and on hardware. The study includes ablation studies isolating the contributions of canonicalization, reward shaping, and curriculum learning.

4. Experimental Results

4.1 Simulation Performance

Evaluated on unseen mug instances with randomized placements:

• Success Rate (SR): AnyMug achieved 93.18% on upright mugs and 94.19% on inverted mugs.
• Comparison:
  • Motion Planning (Open-loop): Performance degraded sharply with increased pose estimation noise (dropping to ~60% SR at $\sigma=0.030$m), highlighting the sensitivity of open-loop execution.
  • Diffusion Policy (DP): Achieved ~36% SR, failing primarily due to empty grasps and timeouts, indicating difficulty in learning precise alignment from demonstrations alone.
  • Equivariant Baselines (EquiDiff, EquiBot): Improved over standard DP but remained significantly below AnyMug (54% and ~12-19% SR, respectively), suggesting that equivariance alone is insufficient for fine-grained handle geometry reasoning.

4.2 Real-World Zero-Shot Transfer

Deployed on a real Franka Panda robot without fine-tuning, tested on 5 physically distinct, unseen mugs:

• Success Rate: Achieved 80% success rate for both upright and inverted categories (8/10 trials each).
• Failure Modes: Failures were split between "tip/fall" and "empty grasp," indicating the policy generally approaches safely but struggles with alignment under real-world perception noise.
• Baselines:
  • Motion Planning: 70% (upright) and 60% (inverted) SR.
  • Diffusion Policy: 20% (upright) and 10% (inverted) SR.
• Conclusion: AnyMug demonstrated superior robustness to real-world perception errors compared to open-loop and standard learning-based baselines.

4.3 Ablation Studies

• Finger Opposition Reward: Removing this term resulted in 0% success rate and 100% empty grasps, proving it is essential for functional grasping.
• Canonicalization: Removing observation-action canonicalization caused success rates to drop from ~93% to ~31% (upright) and ~41% (inverted), confirming it is the dominant factor for generalization.

5. Significance and Claims

The paper claims that AnyMug successfully solves the problem of pose-agnostic functional grasping by decoupling the task strategy from the object's pose. By canonicalizing both the input (depth) and output (action), the system learns a single, reusable closed-loop policy that generalizes across instances and poses.

The authors emphasize that:

• Canonicalization is key: It removes the need for the policy to learn placement-dependent motor commands, significantly improving generalization.
• Closed-loop feedback is necessary: Unlike open-loop methods, the closed-loop nature of AnyMug compensates for residual pose estimation errors and contact variations.
• Sim-to-Real Transfer: The framework achieves robust zero-shot transfer to a real robot, outperforming both analytical planners and state-of-the-art learning baselines without real-world fine-tuning.

Limitations Acknowledged:

• The current work is limited to mug objects and two pose categories (upright/inverted).
• It assumes a clear top-down depth observation; performance may degrade in cluttered scenes, under severe occlusion, or with reflective surfaces where handle direction estimation fails.
• Future work is needed to extend the framework to broader object classes, additional pose modes (e.g., tilted), and downstream tasks like pouring.