💻 computer science

Hydra: Marker-Free RGB-D Hand-Eye Calibration

This paper introduces Hydra, a marker-free RGB-D hand-eye calibration method utilizing a Lie algebra-based iterative closest point algorithm that achieves significantly higher convergence rates, faster processing times, and improved task-space accuracy compared to existing baselines, while providing open-source code and ROS 2 integration.

Original authors: Martin Huber, Huanyu Tian, Christopher E. Mower, Lucas-Raphael Müller, Sébastien Ourselin, Christos Bergeles, Tom Vercauteren

Published 2026-09-02

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Martin Huber, Huanyu Tian, Christopher E. Mower, Lucas-Raphael Müller, Sébastien Ourselin, Christos Bergeles, Tom Vercauteren

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots that move with their own hands and eyes are a staple of science fiction, but in the real world, giving a machine the ability to see where its arm is in space is a surprisingly difficult engineering puzzle. This challenge, known as hand-eye calibration, is the process of teaching a robot how to translate the data from its camera into the language of its own joints. Without this translation, a robot might see an object clearly but reach for the wrong spot, or fail to understand how its own movements change what it sees. For decades, the standard solution to this problem has relied on physical markers—special patterns or stickers placed on the robot or the environment that the camera can easily recognize. While effective, these markers are fragile; if they are hidden, damaged, or simply not in the camera's view, the calibration fails. This limitation makes it hard to deploy robots in messy, unpredictable environments where such markers cannot always be guaranteed.

A team of researchers has now developed a new way to solve this problem without needing any markers at all. Their method, which they call Hydra, allows a robot to calibrate itself simply by looking at its own arm. Instead of relying on special stickers, the system uses a camera to capture the shape of the robot's metal links and joints, then matches that visual shape against a digital model of the robot. The researchers tested this approach on three different types of robotic arms and two different cameras, finding that the system could successfully align the robot's vision with its movement in nearly all cases. Remarkably, the new method works with as few as three random positions of the robot arm, achieving a level of accuracy that rivals methods using markers, but with a speed that is two orders of magnitude faster than other marker-free attempts.

The core idea behind Hydra is to treat the robot's arm not as a collection of moving parts, but as a single, solid 3D object that can be matched to a cloud of points captured by a camera. In the past, researchers tried to solve this by training complex computer programs to recognize specific points on a robot, or by using heavy computational techniques that simulated how light would bounce off the robot's surface. These older methods were often slow and prone to getting stuck in local errors, meaning the robot would think it was calibrated when it was actually slightly off. The Hydra team took a different path. They used a modern image-analysis tool to isolate the robot from the background in a video feed, creating a clean 3D map of just the arm. They then compared this map to the robot's known digital blueprint, sliding the two shapes against each other until they fit perfectly.

To ensure this fitting process was robust, the researchers formulated a mathematical approach that focuses on the surfaces of the robot rather than just individual points. Imagine trying to fit two pieces of a puzzle together; looking at just the corners might be misleading if the pieces are slightly worn, but looking at how the flat edges slide against each other provides a much more stable connection. The team's method uses this principle of surface alignment to find the correct position. They also built in a safety mechanism that ignores stray data points, such as shadows or reflections, which often confuse other systems. This allows the robot to find the correct alignment even when the camera view is imperfect or the robot is in a cluttered space.

The results of their experiments were striking. When tested on three different robotic arms—a large medical robot, a compact desktop arm, and a lightweight industrial model—the system succeeded in calibrating the robot in about 90% of attempts using only three random positions. In terms of speed, the process took less than a second to complete, which is significantly faster than other marker-free methods that can take over two minutes. The accuracy of the calibration was also impressive, with the robot's hand landing within 5 millimeters of the intended target in the real world. This is a significant improvement over older, marker-free techniques, which often resulted in errors of 7 millimeters or more. While methods that use physical markers can sometimes be slightly more precise, the new approach removes the need for those markers entirely, making the process much simpler for users who do not want to attach stickers to their equipment.

One of the most practical aspects of this work is that it does not require the robot to be a specific brand or model. Because the system relies on the general shape of the robot rather than pre-programmed knowledge of its joints, it can be applied to almost any serial manipulator, which is the technical term for the common type of robot arm with a series of linked segments. The researchers made their software and the data from their experiments available to the public, allowing other engineers to test and improve upon the method. They also integrated the system into a standard robotics software platform, meaning that developers can now plug this capability into their own robots with minimal effort.

The study does have some limitations. The system currently requires a camera that can measure depth, meaning it cannot work with a standard 2D camera alone. It also relies on a human to briefly point out the robot in the initial video frame, though this interaction is minimal. Furthermore, the method was tested in static environments where the camera and robot do not move during the calibration, so it is not yet ready for robots that are constantly on the move. Despite these constraints, the work represents a significant step forward in making robots more autonomous and easier to deploy. By removing the dependency on physical markers and drastically reducing the time needed for setup, this new approach brings the vision of flexible, self-calibrating robots closer to reality. The researchers suggest that future work will focus on adapting the method for moving setups and further refining the accuracy to match the precision of the best marker-based systems.

Technical Summary: Hydra: Marker-Free RGB-D Hand-Eye Calibration

Problem Statement
Hand-eye calibration is a fundamental requirement for robotic autonomy, enabling the transformation of joint space into a Cartesian reference frame. While classical approaches rely on physical markers (e.g., AprilTags) to establish correspondences, these methods impose strict visibility constraints and increase user complexity. Existing marker-free alternatives often suffer from significant limitations: some rely on supervised learning requiring dedicated models for specific robots, while others utilize differentiable rendering (DR) which incurs high computational overhead and risks converging to local minima due to unknown initialization. There is a need for a calibration method that is accurate in task space, robust, applicable to any serial manipulator, and computationally efficient without requiring markers.

Methodology
The proposed method, Hydra, addresses these challenges by directly registering robot meshes to point clouds derived from RGB-D images, bypassing the need for differentiable rendering or dedicated keypoint detectors.

Data Acquisition and Segmentation:
- The system utilizes instance segmentation on RGB images using the foundation model SAM 2 to identify the robot.
- To reduce correspondence complexity and favor the surface sliding mechanism of the objective function, the segmentation boundaries are eroded.
- These eroded masks are applied to depth maps to extract relevant point clouds, which are then projected into camera coordinates using camera intrinsics.
- Multiple robot configurations are fused into a combined point cloud to enhance the optimization landscape.
Optimization Framework (Lie Algebra ICP):
- The core of Hydra is a novel Iterative Closest Point (ICP) algorithm formulated on the Lie algebra $se(3)$ to ensure the transformation $\Theta$ remains within the Special Euclidean group $SE(3)$ .
- Objective Function: The method minimizes a robust Point-to-Plane (PTP) objective. Given correspondences between mesh vertices ( $\tilde{m}_i$ ) and observed points ( $\tilde{o}_i$ ), the algorithm minimizes the distance between the transformed observed points and the mesh planes.
- Linearization: The exponential map is linearized for small increments, converting the optimization into a standard least-squares problem solvable via matrix inversion.
- Robustness: To handle outliers in the point cloud, the method employs Iteratively Reweighted Least Squares (IRLS) with a Huber loss function. The noise standard deviation is estimated dynamically using the Median Absolute Deviation (MAD) of residuals.
Initialization and Execution:
- An initial estimate is computed by aligning the centroids of the mesh and point cloud for each configuration using the Kabsch-Umeyama algorithm.
- The algorithm iterates through an outer loop (finding correspondences) and an inner loop (updating the transformation via the Lie group update $\Theta_{j+1} = \Theta_j \exp_{Lie}(\delta\theta)$ ).

Key Contributions

Novel Algorithm: A computationally efficient ICP algorithm incorporating a robust PTP objective formulated on a Lie algebra, applicable to any serial manipulator without requiring robot-specific training.
Efficiency and Robustness: The method avoids the runtime overhead of differentiable rendering and the initialization sensitivity of DR-based approaches.
Open Ecosystem: The authors provide an open-source implementation (Apache 2.0), a comprehensive benchmark dataset, and a ROS 2 integration with robot abstraction for seamless deployment.

Experimental Results
The method was evaluated on three serial manipulators (KUKA LBR Med 7, Mecademic Meca500, UFACTORY xArm 7) and two RGB-D cameras (Intel RealSense D435, Stereolabs ZED 2i) across 270 unique samples.

Convergence Rate: Hydra achieves approximately 90% successful calibrations using only three randomly chosen robot configurations. This represents a 2–3× higher convergence rate to the global optimum compared to both marker-based and marker-free baselines (including EasyHeC).
Computational Speed: The method demonstrates a convergence time of 0.8 ± 0.4 seconds for 9 robot configurations, which is two orders of magnitude faster than the marker-free EasyHeC method (132.7 ± 17.4 s).
Accuracy: In task space, Hydra achieves an accuracy of 5 mm, significantly improving upon classical marker-based approaches (7 mm). While the PnP method (relying on known end-effector transformations) showed lower errors (e.g., 0.6–1.8 mm), it is noted as less practical for variable end-effectors.
Repeatability: Hydra maintains a 100% success rate when the robot is observed in sufficiently varying configurations, whereas marker-free baselines like EasyHeC* (adapted with SAM 2) dropped to 33% success rates under similar conditions.

Significance and Limitations
The paper claims that Hydra is the first work to demonstrate that image-space segmentation using a foundation model, combined with a Lie algebra-based PTP ICP, provides a viable, marker-free abstraction for hand-eye calibration across diverse hardware. It offers a simpler alternative to differentiable rendering, relying solely on matrix inversion for the linearized objective.

The authors note specific limitations:

The method relies on RGB-D cameras and SAM 2, which requires limited user interaction for prompting.
It is currently designed for static setups and does not support eye-in-hand configurations or dynamic motion without recalibration.
Performance on the Meca500 robot was slightly reduced due to sparse mesh data and lower effective camera resolution, occasionally leading to local minima.

Despite these constraints, the work positions Hydra as a significant step forward in making hand-eye calibration more accessible, robust, and deployable for general-purpose robotics.

Technical Summary: Hydra: Marker-Free RGB-D Hand-Eye Calibration

More like this