Visual Prompting for Robotic Manipulation with Annotation-Guided Pick-and-Place Using ACT

Technical Summary: Visual Prompting for Robotic Manipulation with Annotation-Guided Pick-and-Place Using ACT

Problem Statement
Robotic pick-and-place tasks in convenience store environments face significant hurdles due to dense object arrangements, frequent occlusions, and high variability in object properties (color, shape, size, texture). Traditional approaches relying on predefined heuristics, structured environments, or exhaustive scene segmentation often lack the adaptability required for novel items and dynamic layouts. Furthermore, conventional step-by-step planning can be error-prone in long-horizon tasks, leading to accumulated errors and failure.

Methodology
The authors propose a perception-action pipeline that combines annotation-guided visual prompting with Action Chunking with Transformers (ACT), an imitation learning algorithm.

• Visual Prompting: Instead of requiring the robot to perform complex, exhaustive scene parsing, the system utilizes bounding box annotations to provide structured spatial guidance. These annotations explicitly identify the target object for picking and the destination for placement. The system employs two Intel RealSense cameras (D435i on the wrist and D415 on the table) to capture RGB and depth data. The RGB images, augmented with visual prompts (bounding boxes), serve as direct inputs to the neural network.
• Action Chunking with Transformers (ACT): The core control algorithm is ACT, which replaces rigid, step-by-step planning with the prediction of "chunked" action sequences.
  • Architecture: The model uses a Transformer-based architecture. A CVAE encoder processes human demonstration data (actions and observations without images) to generate a latent variable ($z$). A Transformer decoder then predicts a sequence of future actions (a "chunk") based on the current state, the latent variable, and the visual input (RGB image with bounding boxes).
  • Training: The system is trained via imitation learning using human demonstrations collected via teleoperation (3D space mouse). The training objective minimizes the reconstruction error between predicted and actual action chunks, regularized by a KL divergence term.
• Experimental Setup: The system was implemented on a Universal Robots UR5e arm equipped with a Robotiq 2-Finger Gripper. Experiments were conducted in a simulated and real-world convenience store setting using six distinct product categories (e.g., noodle bowls, choco boxes, tea bottles) representing diverse shapes, textures, and materials.

Key Contributions

1. Annotation-Guided Visual Prompting: The introduction of bounding box-based visual prompting to guide robotic manipulation. This approach reduces the complexity of scene understanding while ensuring task execution by providing lightweight, effective spatial cues.
2. ACT-Based Adaptive System: The implementation of ACT to enable the robotic arm to execute smooth, chunked action sequences derived from human demonstrations, rather than relying on rigid, incremental planning.
3. Systematic Evaluation Framework: A structured experimental analysis across three progressive levels of task complexity (Simple, Complex, and More Complex scenarios) to evaluate how visual prompting and object diversity influence robotic performance.

Results
The system was evaluated across three scenarios involving 3x3 arrangements of products:

• Simple Scenario: Nine similar boxes with one annotated target. The system achieved a 90% success rate, demonstrating reliability in uniform environments.
• Complex Scenario: Nine diverse products with one annotated target. Initial success rates dropped to 70% due to variations in object properties (reflectivity, slipperiness). After increasing the demonstration data by 20% specifically for the failure cases, the system's performance improved significantly, achieving a 100% success rate across all object categories in this scenario.
• More Complex Scenario: Nine diverse products in varying positions with both pick and place locations annotated. Initial success was 70%. Due to the increased difficulty, the researchers collected double the amount of human-annotated data for each product. Following this data increase, performance improved, though specific object categories (e.g., reflective tea bottles, slippery jars) still showed lower success rates (80%) compared to rigid boxes (90%).

Failure Analysis
The study identified specific failure modes, including finger misalignment, weak grasp power leading to slippage (particularly on slippery surfaces), and placement misalignment. Attention heatmaps revealed that the ACT model successfully shifts focus from the pick location to the place location, adapting its strategy based on visual prompts. However, objects with reflective surfaces or soft textures presented persistent challenges, leading to misalignment and grasp failures.

Significance and Claims
The paper claims that combining visual prompting with action chunking allows robots to efficiently interpret and act upon minimal yet informative annotations for real-world manipulation tasks. The proposed method represents a step forward in creating adaptive robotic systems capable of handling complex object variations with improved autonomy and robustness.

The authors maintain a modest stance regarding the limitations of their work. They explicitly acknowledge that the system is data-demanding, relying heavily on diverse and high-quality human demonstration data. Consequently, the paper does not claim universal generalization without sufficient training data. Future work is identified as focusing on data augmentation to artificially expand datasets, rather than claiming immediate solutions to data scarcity. The study concludes that while the approach enhances grasp stability and placement accuracy, the effectiveness remains tied to the quality and quantity of the training demonstrations.