AdaptPNP: Integrating Prehensile and Non-Prehensile Skills for Adaptive Robotic Manipulation

1. Problem Statement

Robotic manipulation typically relies on prehensile (P) actions (grasping and lifting). However, many real-world scenarios render grasping infeasible or insufficient due to object geometry (e.g., thin cards, flat objects), environmental constraints (e.g., objects tucked in corners), or collision risks. In these cases, non-prehensile (NP) skills (pushing, poking, sliding, rotating) are required.

The core challenges addressed by this paper are:

• Unified Framework: Existing methods often handle P and NP skills separately. There is a lack of a unified framework that can dynamically decide when to switch between grasping and pushing, and how to sequence them.
• Combinatorial Complexity: The diversity of NP primitives creates a massive search space for skill selection and sequencing.
• Physical Feasibility vs. Semantic Plausibility: Vision-Language Models (VLMs) can generate semantically reasonable plans but often fail to account for physical constraints (e.g., friction, collision, stability), leading to plans that are impossible to execute.
• Generalization: Current methods struggle to generalize to novel objects, environments, and complex hybrid tasks without extensive hand-crafted domain knowledge.

2. Methodology: AdaptPNP Framework

AdaptPNP is a Vision-Language Model (VLM)-empowered Task and Motion Planning (TAMP) framework. It operates in a closed-loop, three-stage process to integrate high-level semantic reasoning with low-level physical execution.

A. Core Components

1. VLM Task Planner & Reflector:

   • Planner: Takes an RGB image and a natural language instruction to generate a Plan Skeleton. This is a sequence of action primitives (e.g., Push, Rotate, Grasp, MoveTo, Release) paired with discrete parameters (target objects/regions). It uses Chain-of-Thought (CoT) reasoning to infer spatial relations.
   • Reflector: Analyzes execution failures. If a step fails, it receives an error message and a new visual observation. It reasons about the root cause and prompts the Planner to re-generate the plan skeleton, creating an iterative refinement loop.

2. Digital Twin "Mental Rehearsal" (Sub-goal Pose Generation):

   • To bridge the gap between high-level semantics and low-level physics, the framework uses a Digital Twin (a 3D reconstruction of the environment without the robot arm).
   • Pose Sampling: For each action in the plan skeleton, the system samples candidate 6D object poses (position + orientation) within the digital twin.
   • Feasibility Filtering: It simulates the action in the twin to discard physically impossible outcomes (e.g., objects falling off the table).
   • VLM Selection: The VLM is presented with rendered images of the top candidate poses and selects the most semantically and physically appropriate one as the Sub-goal Pose. This serves as a precise intermediate representation.

3. Low-Level Execution & Feedback:

   • The system maps the selected 6D sub-goal poses to robot trajectories using heuristic policies (e.g., PID controllers for pushing along a vector, AnyGrasp for grasping).
   • Crucially, the interface is object-centric: the system sends target object poses to the execution environment, not specific robot joint commands. This abstracts away robot dynamics, facilitating transfer across different robots.
   • Execution feedback (success/failure and error types) is fed back to the VLM Reflector to trigger replanning if necessary.

B. Action Primitives

The framework classifies skills into:

• Non-Prehensile (NP):
  • Push: Planar translation and in-plane rotation (SE(2)).
  • Rotate: Out-of-plane rotation (tilting/flipping) about edges.
• Prehensile (P):
  • Grasp, Moveto, Release.

3. Key Contributions

1. Adaptive Hybrid Framework: Proposes AdaptPNP, the first generalizable framework that adaptively selects and coordinates both prehensile and non-prehensile skills based on visual and physical constraints, enabling flexible skill composition.
2. 6D Object Pose Intermediate Representation: Introduces a digital-twin-based layer that generates physically informed 6D target poses. This bridges the semantic planning and motion execution gap, ensuring that sub-goals are both semantically correct and physically feasible.
3. Closed-Loop Reflection Mechanism: Implements an iterative feedback loop where execution failures trigger VLM-based reasoning to refine the plan, resolving multi-modal ambiguities and physical infeasibilities.
4. Comprehensive Evaluation: Demonstrates superior performance across diverse scenarios including object alignment, extrinsic dexterity (using environment edges), and tool usage.

4. Experimental Results

The authors evaluated AdaptPNP on 8 simulated tasks and 4 real-world tasks (using a UFactory xArm6 robot).

• Baselines: Compared against Model Predictive Control (MPC), Reinforcement Learning (RL), hierarchical VLM planners (MoKA, MolmoAct), and end-to-end Vision-Language-Action models (OpenVLA).
• Simulation Performance:
  • AdaptPNP achieved the highest success rates across all tasks (e.g., 9/10 on Box, 7/10 on Book, 9/10 on Slot).
  • Baselines like MPC and RL failed on long-horizon tasks requiring non-monotonic strategies (e.g., moving an object away from a target to go around an obstacle).
  • End-to-end models (OpenVLA) and 2D-point-based planners (MoKA) performed poorly due to a lack of physical grounding and inability to handle fine-grained 3D rotations.
• Ablation Studies:
  • Removing the 6D Pose Representation (reverting to 2D keypoints) caused significant performance drops, especially in extrinsic dexterity tasks, proving the necessity of full 3D spatial reasoning.
  • Removing the Reflection Mechanism (one-shot planning) resulted in near-total failure, highlighting the critical need for iterative error correction.
• Real-World Transfer:
  • AdaptPNP successfully transferred from simulation to the real world with minimal performance degradation (e.g., 8/10 on Box, 5/10 on Tool Hook).
  • The object-centric interface (transferring poses rather than robot commands) effectively bypassed the sim-to-real gap.

5. Significance

AdaptPNP represents a significant step toward general-purpose, human-level robotic manipulation. By unifying semantic reasoning (via VLMs) with physical validation (via Digital Twins) and adaptive replanning, it overcomes the limitations of current methods that are either too rigid (hand-crafted TAMP) or too brittle (end-to-end learning). The framework demonstrates that robots can effectively handle complex, unstructured environments by leveraging non-prehensile skills as a strategic complement to grasping, rather than as a fallback. This approach is crucial for deploying robots in dynamic, real-world settings where objects are often ungraspable or require manipulation via environmental interaction.