Technical Summary: First Plan, Then Evaluate (FPTE)

Problem Statement

Autonomous multi-finger grasping is a fundamental capability for robotic manipulation, particularly when dealing with novel objects using only a single fixed camera's pointcloud. While optimization-based approaches are state-of-the-art, they are often sensitive to initialization and computationally expensive due to the non-linear nature of optimizing over neural network classifiers.

To address speed, the "generator-evaluator-planner" paradigm has emerged. In this traditional pipeline:

1. A generator proposes a batch of grasp candidates in the object frame.
2. An evaluator ranks these candidates based on predicted success likelihood.
3. A motion planner attempts to find a collision-free trajectory to the highest-ranked grasp.

If the planner fails to reach the top-ranked grasp (due to unreachability or collisions), it attempts the next-best grasp. This approach presents a "lose-lose" trade-off:

• Strict Planning: Enforcing strict proximity to the target grasp ensures accurate success estimation but leads to frequent planning failures, forcing the robot to execute lower-ranked, less likely-to-succeed grasps.
• Relaxed Planning: Relaxing proximity thresholds allows for easier trajectory computation but results in inaccurate grasp success estimates because the learned evaluator is highly sensitive to the end-effector (EE) pose.

Furthermore, grounding grasps in the object frame (to ensure generalization across environments) often generates targets that are unreachable by the physical robot due to joint limits or environmental collisions, a problem that persists even with large-scale simulation datasets.

Methodology: First Plan, Then Evaluate (FPTE)

The authors propose a framework that reverses the order of evaluation and planning to resolve the trade-off between planning feasibility and success estimation accuracy. The pipeline consists of three stages:

1. Generation: A learned generator takes a partial pointcloud (encoded via Basis Point Set) and proposes a batch of $K$ grasp candidates ($G^T$) in the object frame. These serve as targets for the motion planner.
2. Parallel Planning: A vectorized motion planner computes trajectories to all $K$ target grasps simultaneously. Crucially, the system does not discard trajectories that fail to reach the exact target pose due to unreachability or obstacles. Instead, it retains the resulting collision-free trajectories and their terminal configurations.
3. Terminal Evaluation & Execution: A learned evaluator estimates the grasp success likelihood based on the terminal configuration of the planned trajectories (the actual state the robot reaches), rather than the initial generator targets. The robot then executes the trajectory corresponding to the highest estimated success likelihood.

Key Technical Components

• Representation: Objects are encoded as $O \in \mathbb{R}^{4096}$ using Basis Point Sets (BPS). Grasps are parametrized as $G = (T_{OH}, \theta_p, \theta_g)$, representing hand pose, pre-grasp configuration, and in-grasp configuration.
• Generator: Trained exclusively on positive grasps, it outputs a batch of grasp samples.
• Evaluator: Trained on both positive and negative samples (including hard negatives generated by perturbing positive samples), it outputs a grasp success likelihood. It evaluates the actual reachable state, not the proposed target.
• Data Collection: A dataset of 28.9 million grasp attempts was synthesized in simulation using full-robot trajectories (avoiding the "floating end-effector" issue). This yielded approximately 180k successful grasps for training.

Key Contributions

1. Framework Inversion: The primary contribution is the shift from "Generate-Evaluate-Plan" to "Generate-Plan-Evaluate." By evaluating grasps at the terminal configuration of executable trajectories, the system avoids the sensitivity of the evaluator to unreachable targets.
2. Robustness to Unreachability: The approach explicitly handles cases where the highest-ranked generator target is unreachable. Instead of discarding the trajectory or downgrading to a suboptimal target, it evaluates the actual reachable configuration.
3. Generalization: The method demonstrates that planning to multiple targets in parallel (using vectorized planners like Curobo or Fabrics) is computationally efficient and significantly improves success rates across different object shapes, generator architectures, and motion planners.

Experimental Results

The authors evaluated FPTE in both simulation and the real world using a KUKA LBR4 arm with an Allegro hand.

• Simulation: FPTE consistently achieved higher grasp success rates than the traditional "trad" method across various generators (cVAE, Diffusion, MDN) and motion planners (Curobo, Fabrics, RRT).
  • The traditional method's performance degraded significantly as Inverse Kinematics (IK) error increased, whereas FPTE remained robust.
  • FPTE-Fabrics outperformed FPTE-Curobo, likely due to the data collection pipeline using Fabrics, reducing the sim-to-real gap.
• Real World:
  • Tested on 11 novel objects across 5 locations each.
  • Success Rate: FPTE achieved an 80% success rate compared to 22% for the traditional baseline.
  • Failure Analysis: The baseline failed to find any trajectory in 23 out of 55 attempts. When considering only executed trajectories, the baseline success rate was 38%, while FPTE's was significantly higher.
  • Confidence: The predicted success likelihood for executed grasps using FPTE averaged 90%, compared to 26% for the baseline.
• Generalization: The system successfully generalized to novel environments with varying table heights, shelves, and clutter, despite being trained only on isolated objects at a fixed height in simulation.

Significance and Claims

The paper claims that FPTE offers a practical and effective solution to the inherent trade-off between motion planning accuracy and grasp success estimation. By decoupling the target generation from the success evaluation, the framework allows the robot to execute the trajectory most likely to succeed, even if it differs from the initially proposed target.

The authors emphasize that this improvement is framework-independent, persisting regardless of the specific learner (generator) or planner used. They conclude that the subtle change of evaluating grasps at the end of the trajectory, rather than at the start, is critical for bridging the gap between simulation and real-world performance, particularly in handling unreachable targets and environmental constraints.

Limitations noted by the authors:

• Limited diversity in training objects.
• Segmentation inaccuracies in the real world can lead to incorrect collision modeling.
• The system operates in an open-loop manner based on a single RGB-D reading.