🤖 machine learning

Bi-Manual Joint Camera Calibration and Scene Representation

This paper introduces Bi-JCR, a framework that leverages 3D foundation models to enable marker-free, joint calibration of bi-manual robot cameras and the construction of a unified, scale-consistent 3D scene representation directly from RGB images, thereby facilitating downstream bimanual coordination tasks.

Original authors: Haozhan Tang, Tianyi Zhang, Matthew Johnson-Roberson, Weiming Zhi

Published 2026-07-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haozhan Tang, Tianyi Zhang, Matthew Johnson-Roberson, Weiming Zhi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine two robot arms, like a pair of mechanical friends, trying to work together on a table. Usually, before they can high-five or pass a tool, they need to know exactly where their eyes (cameras) are attached to their hands and where they are standing relative to each other. Traditionally, getting this information is a boring, messy chore: you have to tape a special checkerboard pattern to the wall, move the robots around to take pictures of it, and do the math to figure out the angles. It's like trying to learn to dance by staring at a mirror with a ruler taped to your forehead.

This paper introduces a new trick called Bi-JCR (Bi-Manual Joint Calibration and Representation). Instead of using those annoying checkerboards, Bi-JCR lets the robots just look at the cluttered table full of random objects—spoons, tape rolls, toolboxes—and figure everything out on their own.

How it works: The "Magic Brain" and the Puzzle
The secret sauce is a "3D foundation model." Think of this as a super-smart, pre-trained brain that has seen millions of images and knows how to guess the 3D shape of a room just by looking at flat photos. When the robots take pictures of the table, this brain builds a rough 3D map. But here's the catch: the brain doesn't know the real size of things (is that spoon 5 inches long or 5 feet long?) and it doesn't know exactly where the robot's base is.

Bi-JCR acts like a detective connecting the dots. It takes the robot's own movement logs (we know exactly how the robot moved its arm) and compares them to the "guesses" made by the magic brain. By solving a giant math puzzle, it figures out three things at once:

Where the camera is on the hand: It aligns the robot's eye with its hand.
Where the two robots are standing: It figures out how far apart the two robot bases are.
The real-world scale: It calculates a "magic number" to turn the brain's fuzzy 3D guess into a real-world, metric-scale map (so a spoon is actually a spoon, not a giant or a tiny one).

What the paper says is NOT the answer
The authors are very clear: they are not using checkerboards, AprilTags, or any special markers. They are also not relying on the robots to have perfect depth sensors (like LiDAR) or pre-existing 3D models of the objects. They explicitly argue against the old way of doing things, which requires stopping the robot, taping things down, and calibrating each arm separately before trying to fuse them together. Their method does it all in one go, using only the standard RGB cameras already on the robots.

How sure are they? (The Evidence)
The team didn't just dream this up; they tested it in the real world. They set up two real robot arms (AgileX Piper 6-degree-of-freedom manipulators) with cheap USB webcams. They ran experiments in three different lighting conditions with different numbers of objects (9 or 10 items).

The Calibration: They compared their method against other popular tools (like COLMAP and Ray Diffusion). The results showed that Bi-JCR was much more consistent. When they used just 4 images per robot, their method still worked, while others often failed to find a solution. Even with more images (up to 9), their error rates stayed low. For example, the "rotational error" (how much the angle was off) was around 0.0769 with 4 images, dropping to 0.0612 with 9 images.
The Scale: They measured real objects like a spoon, a tea lid, and a battery. With just 8 images per robot, the reconstructed 3D models were incredibly accurate. The biggest error they saw was 2.98% (for a joystick), and the median error was just 1.48%. This means if a real spoon is 10 inches, the robot's map says it's roughly 10.15 inches.
The "Refinement" Step: They tested if running a "gradient descent" (a mathematical fine-tuning process) helped. They found that when images were scarce (only 4 views), this step made a huge difference, cutting the error significantly. But when they had plenty of images (8 views), the extra step helped only a tiny bit.
The Brain Choice: They tried different "foundation models" (the magic brains). They found that one called DUSt3R worked the best, beating out other models like MASt3R and VGGT in almost every test.

The Grand Finale: Can they actually do stuff?
The ultimate test wasn't just numbers; it was action. Once the robots calibrated themselves, they used the 3D map to do real tasks.

Joint Grasping: They picked up heavy, awkward objects like a toolbox and a battery pack together.
Hand-overs: They successfully passed smaller items (a wrench, a spoon, a tape roll) from one robot to the other.

The paper concludes that because the robots knew exactly where they were and how big the objects were, they could coordinate perfectly without any human help or special markers. The authors suggest that in the future, they could use the "confidence" scores from the AI brain to tell the robots exactly which new photos to take to make the map even better, but for now, they have proven that this marker-free, joint calibration works in the real world.

Technical Summary: Bi-Manual Joint Camera Calibration and Scene Representation (Bi-JCR)

Problem Statement
Bimanual robot manipulation typically requires multiple cameras rigidly mounted to robot end-effectors. Before these systems can generate motion or build environmental representations, the cameras must be calibrated to determine their extrinsic transformations relative to the end-effectors and the relative poses between the manipulator bases. Traditional "hand-eye" calibration relies on offline procedures using external markers (e.g., checkerboards or AprilTags), which is cumbersome and often violated in dynamic lab settings. Extending these methods to dual-arm systems is non-trivial, as it requires fusing two separate coordinate frames, often necessitating a secondary manipulator to hold a marker. This paper addresses the challenge of calibrating dual manipulators with wrist-mounted cameras without any external calibration markers, while simultaneously recovering a metric-scale 3D representation of the shared workspace.

Methodology
The proposed framework, Bi-Manual Joint Calibration and Representation (Bi-JCR), leverages 3D foundation models to perform marker-free calibration and scene reconstruction. The process involves the following key steps:

Data Collection: Two manipulators (primary and secondary) are commanded through a sequence of $N$ distinct end-effector poses, capturing RGB images at each pose. The forward kinematics (base-to-end-effector transforms) are known, but the camera-to-end-effector transforms ( $T^E_C$ ), the relative pose between the two robot bases ( $P^{b_1}_{b_2}$ ), and the global scale are unknown.
3D Foundation Model Inference: The collected RGB image sets from both arms are fed into a pre-trained 3D foundation model (e.g., DUSt3R). This model outputs:
- Relative camera poses in an arbitrary, unscaled coordinate frame.
- Dense 3D point sets corresponding to the scene geometry.
- Confidence maps for the reconstructed points.
Joint Optimization Formulation: Bi-JCR formulates a joint optimization problem to recover the unknown transformations and the scale factor ( $\lambda$ $λ$ ).
- Initial Solution: The problem is decomposed into solving for rotation components (invariant to scale) using linear algebra on the $SO(3)$ manifold, followed by a least-squares solution for translation and the scale factor, treating the problem as a Similarity Scale Recovery Problem (SRP).
- Refinement: The initial estimates are refined via gradient descent on the manifold of transformation matrices ($SE(3)$). The cost function minimizes the discrepancy between the observed end-effector motion and the motion inferred from the foundation model's camera poses, adhering to the classical hand-eye calibration equation $AX = XB$.
Metric-Scale Reconstruction: Once the scale factor $\lambda$ and the transforms from the foundation model's frame to the robot base frames are recovered, the unscaled point clouds are scaled and transformed into a unified, metric-scale coordinate frame (the primary manipulator's base frame). This yields a dense 3D representation suitable for downstream tasks like collision checking and semantic segmentation.

Key Contributions

Bi-JCR Framework: A novel method that simultaneously calibrates wrist-mounted cameras on dual manipulators and constructs a unified 3D environment representation without using external markers.
Joint Optimization Problem: A formulation that recovers camera-to-end-effector transforms, inter-arm relative poses, and the metric scale factor from a single set of RGB images, solving for all transformations in a unified optimization process.
Empirical Validation: Rigorous evaluation on real-world tabletop environments demonstrating the method's ability to produce accurate calibrations and metric-accurate 3D reconstructions, validated against ground truth and baseline methods.

Experimental Results
The authors evaluated Bi-JCR using two AgileX Piper manipulators with USB webcams across various lighting conditions and object densities.

Calibration Accuracy: Bi-JCR consistently achieved lower residual errors in rotation and translation compared to baselines like COLMAP (which struggled with correspondence in cluttered scenes) and Ray Diffusion (which suffered from inconsistent pose regression). The method demonstrated robustness even with a low number of input images (as few as 4 per arm).
Scale Recovery: The recovered metric scale was highly accurate. With 8 images per manipulator, the error in reconstructed object dimensions was at most 2.98% (median 1.48%), confirming the system's ability to recover real-world metric units.
Foundation Model Selection: An ablation study compared different 3D foundation models (DUSt3R, MASt3R, VGGT). DUSt3R consistently yielded the best calibration residuals, outperforming MASt3R and VGGT across all view counts.
Gradient Descent Refinement: The refinement step via gradient descent significantly improved rotational accuracy, particularly in sparse-view scenarios (fewer images), though the marginal benefit decreased as the number of views increased.
Downstream Tasks: The framework successfully enabled complex bimanual tasks, including the coordinated grasping of heavy objects (e.g., toolboxes) and the handover of small objects (e.g., spoons, wrenches) between the two arms, demonstrating the utility of the unified coordinate frame and dense 3D representation.

Significance and Claims
The paper claims that Bi-JCR provides a unified solution that eliminates the need for cumbersome, marker-based offline calibration procedures. By leveraging 3D foundation models, the framework enables immediate fusion of visual data across dual viewpoints, facilitating downstream bimanual coordination tasks such as grasping and object passing. The authors emphasize that their approach produces a dense, size-accurate 3D representation that lives in a common coordinate frame, directly supporting collision checking and semantic segmentation without reliance on external depth sensors or markers. Future work is suggested to leverage confidence masks from foundation models to actively guide image collection and continuously refine the scene reconstruction and calibration.

Technical Summary: Bi-Manual Joint Camera Calibration and Scene Representation (Bi-JCR)

More like this