💻 computer science

Utilizing Inpainting for Keypoint Detection for Vision-Based Control of Robotic Manipulators

This paper presents a novel visual servoing framework for robotic manipulators that utilizes an inpainting-based data collection pipeline to generate automatically labeled, markerless training images and a runtime inpainting model combined with an Unscented Kalman Filter to achieve robust, model-free keypoint detection and control under both full visibility and partial occlusion.

Original authors: Sreejani Chatterjee, Venkatesh Mullur, Abhinav Gandhi, Berk Calli

Published 2026-04-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sreejani Chatterjee, Venkatesh Mullur, Abhinav Gandhi, Berk Calli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot arm to pick up a cup and stack it. Usually, to do this, engineers have to put bright, high-contrast stickers (like QR codes or ArUco markers) all over the robot's joints so the camera can see exactly where the "elbows" and "shoulders" are. It's like putting neon tape on a dancer so a director can track their moves.

But what if you can't put stickers on the robot? Maybe the robot is made of soft, squishy material, or the stickers would get in the way of the task. Or maybe the robot is moving so fast the stickers blur.

This paper presents a clever solution: Teach the robot to see itself without any stickers at all, even when things are blocking the view.

Here is how they did it, broken down into simple steps with some analogies:

1. The "Magic Eraser" Training (Data Collection)

To teach the robot what its own body looks like without stickers, the researchers used a two-step trick:

Step A: They temporarily stuck the bright markers on the robot and took thousands of photos as the robot moved around.
Step B: They used a special AI tool called Inpainting (think of it as a "Magic Eraser" or Photoshop's "Content-Aware Fill") to digitally paint over the markers and fill in the missing spots with the robot's actual skin and joints.

The Result: They now have a massive library of photos where the robot looks natural (no stickers), but the computer knows exactly where the joints are because it remembers where the stickers used to be. They used this library to train a "Keypoint Detective" AI.

2. The "Keypoint Detective" (Keypoint Detection)

Once trained, this AI can look at a photo of the robot and instantly point out its "joints" (like shoulders, elbows, and wrists) just by looking at the natural curves and shadows of the metal arm. It's like a human looking at a silhouette and knowing exactly where the person's elbows are, even without seeing the skin.

3. The "Imagination Engine" (Handling Occlusion)

Here is the tricky part: What if a box, a person, or another part of the robot blocks the camera's view? The "Keypoint Detective" might get confused and say, "I can't see the elbow!"

To fix this, the researchers added a second AI, an Imagination Engine.

The Problem: If a hand is hidden behind a cup, a normal camera sees a cup.
The Solution: This AI looks at the visible parts of the robot and the surrounding scene, then hallucinates (reconstructs) what the hidden part should look like. It fills in the missing robot body so the "Keypoint Detective" can still see the joints.
The Analogy: Imagine you are looking at a friend standing behind a fence. You can only see their head and feet. Your brain automatically "fills in" the rest of their body so you know they are standing there. This AI does that mathematically and instantly.

4. The "Smooth Operator" (The Kalman Filter)

Sometimes, the "Imagination Engine" might guess wrong, or the "Keypoint Detective" might get a little jittery. To stop the robot from shaking or jerking, they added a Kalman Filter.

The Analogy: Think of this as a predictive GPS. If the robot's arm is moving smoothly to the right, and for one split second the camera glitches and says the arm is suddenly on the left, the GPS knows, "That's impossible! The arm was moving right, so it must still be moving right." It smooths out the errors and keeps the movement fluid.

5. The "Blind Pilot" (Control)

Finally, the robot uses all this information to move itself. It doesn't need a 3D map, it doesn't need to know the exact physics of its own joints, and it doesn't need a ruler. It just looks at the camera, sees where its "natural joints" are (or where they should be), and adjusts its motors to get to the target.

Why is this a big deal?

No Stickers Needed: You can use this on soft robots, cheap robots, or robots in messy environments where stickers would fall off.
No 3D Cameras Needed: It works with a single, cheap 2D camera (like a webcam).
Robust: It keeps working even when things block the view, which is crucial for real-world tasks like stacking cups or assembling parts in a cluttered factory.

In a nutshell: The researchers taught a robot to recognize its own body parts using a "Magic Eraser" to create training data, an "Imagination Engine" to see through obstacles, and a "Smooth Operator" to keep the movements steady. This allows the robot to control itself using only a simple camera, without needing expensive sensors or physical markers.

1. Problem Statement

The paper addresses the limitations of current Image-Based Visual Servoing (IBVS) systems for robotic manipulators. Traditional approaches often rely on:

External Markers: Fiducial markers (e.g., ArUco, AprilTags) attached to the robot, which can be occluded, cause motion blur, or be impractical in certain applications.
Explicit Models: Precise knowledge of robot kinematics, camera calibration, and depth sensors (LiDAR/stereo) to generate training data or compute control laws.
Occlusion Sensitivity: Failure when visual features are partially or fully occluded by objects in the environment or self-occlusion.

The authors aim to develop a model-free, markerless, and purely vision-based control framework that:

Uses natural features (keypoints) along the robot's body rather than artificial markers.
Operates without explicit robot kinematic models or accurate camera calibration.
Maintains robust control performance under partial occlusion in an eye-to-hand configuration.
Enables 3D (out-of-plane) motion control using only a single monocular camera.

2. Methodology

The proposed pipeline consists of four interconnected stages:

A. Automated Data Collection & Marker Removal (Training Phase)

To generate a dataset of "natural" robot images with ground-truth keypoints without manual labeling:

Marker Placement: ArUco markers are temporarily attached to the robot body at specific locations.
Data Capture: The robot is moved through its workspace (both planar and out-of-plane) to capture images.
Inpainting for Dataset Generation: A modified LaMa (Large Mask Inpainting) model (a GAN-based approach) is used to remove the ArUco markers from the images and reconstruct the occluded robot regions.
- The centers of the removed markers serve as the ground-truth keypoint labels.
- The inpainted images serve as the training data.
Result: A large dataset of markerless robot images with automatically generated keypoint annotations.

B. Keypoint Detection Model

Architecture: A modified Keypoint R-CNN (ResNet50 FPN backbone) is trained on the generated dataset.
Function: This model detects natural keypoints along the robot's body in real-time. It is customized for specific configurations (e.g., 5 keypoints for 3 planar joints, 8 keypoints for 3 planar + 1 spatial joint).

C. Real-Time Occlusion Handling (Runtime Phase)

To handle occlusions during operation without requiring pre-defined masks:

Mask-Free Inpainting: An Attention U-Net based GAN (trained with Wasserstein GAN with Gradient Penalty - WGAN-GP) is employed.
- Unlike traditional methods, this model does not require an explicit binary mask of the occlusion.
- It learns to distinguish between robot structures and occluding objects, reconstructing the hidden parts of the robot directly from the occluded image.
Process: Occluded images $\rightarrow$ Inpainting Model $\rightarrow$ Cleaned Image $\rightarrow$ Keypoint R-CNN.

D. Control & Filtering

Unscented Kalman Filter (UKF): Integrated to refine keypoint estimates over time.
- It maintains a state vector (position, velocity, acceleration) for each keypoint.
- If the detector fails or returns noisy data due to severe occlusion, the UKF predicts the location based on the motion model, ensuring continuity in the control loop.
Adaptive Visual Servoing:
- The system uses an adaptive Jacobian estimation technique (window-based least squares) to map image feature errors to joint velocities.
- This eliminates the need for a pre-calibrated interaction matrix or robot model.

3. Key Contributions & Novelty

Markerless Data Generation: A novel pipeline using inpainting to automatically generate labeled datasets for natural keypoints, removing the need for manual annotation or complex simulation-to-real transfer.
Model-Free Control: The system operates without explicit robot kinematic models, camera calibration, or proprioceptive sensors (encoders), making it suitable for soft, underactuated, or low-cost robots.
Mask-Free Occlusion Handling: Introduction of a mask-free GAN inpainting model specifically designed for real-time robotic control, capable of reconstructing occluded robot parts without prior knowledge of the occlusion shape.
3D Monocular Control: Successful demonstration of out-of-plane (3D) motion control using a single camera and 2D keypoints, a significant advancement over previous eye-to-hand IBVS methods that often required depth sensing or stereo setups.
Robustness via UKF: Integration of a UKF to handle residual detection errors and missing keypoints, ensuring stable control even when inpainting is imperfect.

4. Experimental Results

The framework was tested on a Franka Emika Panda robot with an Intel RealSense D435i camera.

Keypoint Detection Accuracy:
- Achieved ~100% detection rate within a 10-pixel margin under full visibility for various configurations (2D and 3D).
- Average localization error was under 2.2 pixels.
Occlusion Robustness:
- Small Occlusions: Minimal degradation in control performance (rise/settling times comparable to full visibility).
- Large Occlusions: While trajectories became noisier and overshoot increased, the system successfully converged to the goal. The combination of Inpainting + UKF significantly reduced failure rates compared to using UKF alone.
Control Performance:
- 2D (Planar): Comparable rise and settling times to prior methods that relied on explicit models.
- 3D (Out-of-Plane): Successfully controlled 4 joints (3 planar + 1 spatial) using 6 keypoints, demonstrating stable convergence despite depth variations.
Real-World Demonstration: The system successfully performed a cup-stacking task under partial occlusion, maintaining continuous control throughout the sequence.

5. Significance and Limitations

Significance:
This work represents a paradigm shift toward accessible, robust, and flexible robotic control. By removing dependencies on expensive sensors, precise calibration, and manual marker placement, it lowers the barrier for deploying vision-based control in unstructured environments. It is the first known eye-to-hand IBVS system to achieve 3D control using only monocular 2D keypoints and natural features.

Limitations & Future Work:

Pose Ambiguity: If the end-effector is completely occluded, the inpainting model may hallucinate incorrect poses (e.g., duplicating the end-effector), leading to control errors that the UKF cannot fully correct.
Visual Similarity: Reconstruction fails when occluding objects have colors/textures similar to the robot or are highly reflective/bright, causing structural inaccuracies.
Dynamic Occlusion: Experiments were limited to static occlusions; dynamic moving obstacles were not evaluated.
Future Direction: The authors plan to move from image reconstruction to direct keypoint reconstruction using Graph Neural Networks (GNNs) to better model spatial relationships between keypoints and resolve ambiguity without relying on full image inpainting.