💻 computer science

Self Supervised Learning from Automatically Generated Demonstrations for Visual Robotic Manipulation

This paper presents a self-supervised visual manipulation method that utilizes automatically generated demonstrations in simulation to train a convolutional network for relative pose correction from wrist-mounted RGB images, enabling a UR5e robot to achieve successful grasps with reduced setup effort and improved planar accuracy in both simulated and real-world environments.

Original authors: Andres Rivas, Anselmo R. Cukla, Rodrigo S. Guerra, Bruna V. Guterres, Ricardo B. Grando

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Andres Rivas, Anselmo R. Cukla, Rodrigo S. Guerra, Bruna V. Guterres, Ricardo B. Grando

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to teach a robot how to pick up a coffee mug. In the old days, engineers had to write thousands of lines of code, measure the mug's exact shape, and calibrate the robot's eyes with laser precision. It was like trying to teach someone to ride a bike by giving them a manual on tire pressure and aerodynamics instead of just letting them hop on and wobble. This made robots great at factory jobs but terrible at handling the messy, unpredictable world of our homes.

Recently, scientists discovered a better way: "Learning from Demonstration." Instead of coding every move, you let a human show the robot how to do it, and the robot learns by watching. But there's a catch: getting a human to physically guide the robot's arm or control it with a joystick is slow, boring, and hard to scale up. What if the robot could teach itself? What if it could just look at an object, guess where to grab it, realize it's slightly off, and then practice fixing its own mistakes over and over again until it gets it right? This is the frontier of "Self-Supervised Learning," where the robot acts as both the student and the teacher, generating its own practice problems without needing a human to hold its hand.

This paper introduces a clever new method where a robot does exactly that. The researchers built a system where a robot with a camera on its wrist automatically creates its own "demonstrations." Instead of a human showing it how to grab an object, the robot starts at a random spot, looks at the object, and then moves to the perfect grabbing position. It records this move as a lesson: "When I saw the object this way, I needed to move that way to get to the target." By repeating this thousands of times around different objects, the robot builds a massive library of "image-to-move" lessons without a single human label or complex camera calibration.

The team tested this idea in two ways. First, they ran it in a high-tech video game simulation called Isaac Sim. They taught the robot to approach an object roughly and then fine-tune its position. The results were promising: the robot's final aim got much sharper. In the simulation, the "spread" of where the robot ended up (how far off it was from the perfect spot) shrank from 9.69 mm down to just 5.38 mm after the fine-tuning step. It was like going from throwing a dart that lands in the general neighborhood to hitting the bullseye.

Next, they took the system to the real world using a UR5e robot arm equipped with a gripper and a standard USB camera. They didn't use any special rulers or laser guides to calibrate the camera; the robot just learned from the pictures it took. They tested it on three different physical objects with different shapes and textures. The robot tried to grab them without rotating the object first, and it succeeded 66.6% of the time for one object and 63.6% for another. When they added a twist—literally rotating the objects so the robot had to recognize them from a new angle—the success rates dropped (to 36.4% and 55.5% respectively), but the robot still managed to grab them sometimes.

The paper suggests that while this self-teaching method works well for getting the robot close and refining its aim on flat surfaces, it still struggles a bit with guessing how deep an object is (depth prediction) and gets confused when objects are turned in weird ways. However, the core idea holds up: robots can indeed generate their own training data to learn visual manipulation, skipping the need for expensive human teachers or perfect lab setups. It's a step toward robots that can just look at a cluttered table, figure out how to grab a cup, and learn from their own mistakes, all without needing a human to write the rules.

Technical Summary: Self-Supervised Learning from Automatically Generated Demonstrations for Visual Robotic Manipulation

Problem Statement
Robotic manipulation in unstructured or dynamic environments remains challenging due to the reliance on object-specific programming, manual data annotation, and calibrated perception pipelines. While Learning from Demonstration (LfD) offers a direct alternative, traditional approaches often depend on time-consuming human teleoperation or kinesthetic teaching. Furthermore, many existing visual manipulation systems require explicit camera-to-robot extrinsic calibration, which complicates deployment. The paper addresses the need for a method that automates demonstration collection, eliminates manual labeling, and operates without explicit calibration while maintaining robustness in real-world settings.

Methodology
The proposed framework is a self-supervised pipeline built on ROS 2 and validated in both Isaac Sim and on a physical UR5e collaborative robot. The core approach involves the robot automatically generating demonstrations around a target pose ( $P_0$ ) to learn relative pose corrections directly from wrist-mounted monocular RGB images.

Automatic Data Generation: Instead of human teleoperation, a "Robot Commander" node moves the robot through predefined trajectories around a reference pose. For every RGB image captured, the system computes the relative transformation required to return to the target pose ( $P_0$ ) using tf2. This creates labeled image-pose pairs without manual annotation or explicit camera calibration.
Dataset Complexity: Separate datasets were generated for three physical objects with varying geometries. The data collection utilized four trajectory families to progressively increase difficulty:
- 2-DOF (Planar): Zig-zag on the x-y plane.
- 3-DOF (Planar + Rotation): Planar movement with in-plane rotation.
- 3-DOF (Cone): Vertical displacement combined with planar positions for coarse approach.
- 4-DOF (Cone + Rotation): Vertical displacement with in-plane rotation.
Visual Policy Learning: A convolutional neural network (following the architecture by Johns [1]) is trained to regress relative pose corrections (3 translation values and a 4-component quaternion) from single-frame RGB observations. The loss function combines Mean Squared Error (MSE) for translation and rotation, weighted to prioritize translation accuracy ( $L = MSE_{trans} + 0.01 \cdot MSE_{rot}$ ).
Coarse-to-Fine Control Strategy: During execution, the system employs a two-stage controller:
- Coarse Approach: Uses a model trained on cone-based (3D) data to approach the object from varying heights.
- Planar Refinement: Uses a model trained on planar data to fine-tune the final alignment on the x-y plane.

Key Contributions
The paper outlines four primary contributions:

Self-Supervised Data Collection: A method where the robot automatically generates demonstration-like image-pose pairs without manual labeling or human intervention.
Calibration-Free Visual Policy: A learning model that predicts relative pose corrections from monocular RGB observations without requiring explicit camera-to-robot extrinsic calibration.
Coarse-to-Fine Control: A strategy that decouples the 3D approach from planar refinement, improving final alignment in closed-loop execution.
Real-World Validation: A comprehensive evaluation on a UR5e robot with a gripper and monocular camera, demonstrating performance across multiple physical objects with varying appearances.

Results
The framework was evaluated in simulation and on real hardware using three distinct objects.

Simulation: The coarse-to-fine strategy successfully reduced the final planar dispersion. The radius of the enclosing circle for the final position decreased from 9.69 mm (after the coarse stage) to 5.38 mm (after the refinement stage). However, depth prediction remained challenging, with the 3D sphere radius only decreasing from 33.39 mm to 31.55 mm, and some trajectories showing unstable height estimation.
Real-World Experiments: The system performed end-to-end grasp attempts. Success rates varied by object and condition:
- Without Object Rotation: Object 1 achieved 66.6% and Object 2 achieved 63.6% success. Object 3 had a lower rate of 35.2%.
- With Object Rotation: Performance decreased across the board. Object 1 dropped to 36.4%, Object 2 maintained the highest rate at 55.5%, and Object 3 fell to 25.0%.
- The results indicate that while the system is robust to some variations, performance is object-dependent and degrades when the object's orientation deviates significantly from the training distribution.

Significance and Claims
The authors claim that automatically generated demonstrations can support practical visual manipulation with limited setup effort, effectively removing the need for manual annotation and explicit calibration. The work demonstrates that self-supervised demonstration generation can be extended from simulated or low-dimensional settings to real-world visual manipulation with closed-loop grasp validation.

However, the paper maintains a modest tone regarding its limitations. It acknowledges that the method remains sensitive to depth prediction, object geometry, and object rotation. The authors conclude that while the approach is viable for practical manipulation, future work must address depth estimation challenges (potentially via RGB-D or stereo sensing) and improve generalization across different object orientations. The paper does not claim to solve all generalization problems but rather presents a functional pipeline that reduces engineering effort in setting up visual manipulation tasks.

Technical Summary: Self-Supervised Learning from Automatically Generated Demonstrations for Visual Robotic Manipulation

More like this