💻 computer science

BayesContact: Uncertain Pose Estimation via Visuo-Tactile Proposals and Simulation-based Inference

BayesContact is a simulation-based inference framework that enhances contact-rich pose estimation for tasks like peg-in-hole insertion by fusing depth and visuo-tactile observations to maintain a particle belief, thereby significantly improving observability and insertion success rates compared to vision-only methods.

Original authors: Aditya Kamireddypalli, Matias Mattamala, Joao Moura, Russell Buchanan, Sethu Vijayakumar, Subramanian Ramamoorthy

Published 2026-07-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Aditya Kamireddypalli, Matias Mattamala, Joao Moura, Russell Buchanan, Sethu Vijayakumar, Subramanian Ramamoorthy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a giant, three-dimensional puzzle in a dark room. You have a flashlight, but it only lights up the surface of the pieces, and sometimes the pieces look exactly the same from different angles. This is the daily reality for robots trying to do delicate tasks like plugging a USB cable into a port or screwing a bolt into a tight spot. In the world of robotics, this is called "contact-rich manipulation." It's the art of making things touch, fit, and lock together. The problem is that robots often rely too much on their "eyes" (cameras). Just like you might struggle to tell if a key is the right way up just by looking at it from above, a robot can get confused by shadows, weird angles, or parts of the object that are hidden inside a hole.

To fix this, scientists have been teaching robots to "feel" their way through tasks. Instead of just guessing where a hole is based on a picture, the robot can gently poke around with a tool, feeling for resistance. If the robot feels a bump on the left, it knows the hole is probably to the right. This paper, titled BayesContact, introduces a clever new way for robots to combine what they see with what they feel. It uses a method called "Simulation-Based Inference," which is like a robot running thousands of tiny, invisible movies in its head to guess where an object is. It asks, "If the hole were here, what would my camera see? What would my fingers feel?" Then, it compares those imaginary movies to reality to figure out the truth. This is a big deal because it helps robots stop guessing and start knowing, making them much better at assembling things without breaking them.

The Robot's "Gut Feeling" Upgrade

Meet BayesContact, a new brain upgrade for robots that are trying to perform the tricky "peg-in-hole" dance. You know the move: a robot holds a peg (like a dowel or a plug) and tries to slide it into a matching hole. It sounds simple, but if the robot is off by even a tiny fraction of a millimeter, the peg hits the edge, gets stuck, or jams.

The researchers behind BayesContact realized that relying only on cameras is like trying to find a lost coin in a dark room using only a flashlight that flickers. You might see the general area, but you can't be sure exactly where the coin is or which way it's facing. So, they gave the robot a second sense: touch. But not just any touch—a very smart kind of touch that uses physics simulations to "feel" the future.

The Magic of "What-If" Movies

Here is how BayesContact works, step-by-step:

The Particle Cloud: Imagine the robot doesn't just guess one spot where the hole might be. Instead, it creates a cloud of thousands of tiny "ghost" guesses (called particles). Each ghost says, "I think the hole is here," or "I think it's there," or "I think it's tilted this way."
The Virtual Reality Test: For every single ghost guess, the robot runs a mini-movie in its computer brain.
- The Visual Movie: It uses a graphics engine to ask, "If the hole were actually at this ghost's location, what would the camera see?" It compares this fake image to the real photo the robot just took.
- The Tactile Movie: It uses a physics engine to ask, "If the hole were here, and I poked it with my tool, what kind of force would I feel?" It compares this fake feeling to the real force sensor data.
The Scorecard: The robot gives a score to each ghost. If a ghost's "what-if" movie matches the real world perfectly, that ghost gets a high score and becomes more "real." If a ghost's movie doesn't match (e.g., the ghost thought the hole was on the left, but the robot felt a bump on the right), that ghost gets a low score and fades away.
The Active Probe: This is the coolest part. Once the robot has a cloud of ghosts, it doesn't just sit there. It asks, "Which poke will teach me the most?" It picks a spot to probe that is most likely to clear up the confusion. If the robot is unsure whether the hole is rotated 90 degrees or 180 degrees, it will poke in a way that gives a totally different answer for each possibility, instantly narrowing down the truth.

Why This Matters: The "Teeth" Problem

The researchers tested this on some tricky shapes, including ones with "teeth" or weird curves that look very similar from different angles. In these cases, a camera alone gets totally confused.

The results were impressive. In their computer simulations, BayesContact improved the robot's ability to find the right spot by 30% compared to using just a camera. When they tested it on a real robot arm (a KUKA iiwa14) in the real world, the improvement was just as strong. The robot using BayesContact was able to successfully insert the peg 20% to 30% more often than the robot that only used its eyes.

The "Guessing Game" vs. The "Smart Probe"

The paper also compared different ways for the robot to choose where to poke.

The "Best Guess" (MAP): The robot looks at its cloud of ghosts, picks the single most likely one, and pokes there.
The "Curious Explorer" (Information Gain): The robot looks at the whole cloud and asks, "Where is the biggest confusion?" It then pokes exactly where it will learn the most, even if that spot isn't the most likely location.

The "Curious Explorer" strategy won. It solved the puzzle faster and with fewer pokes. It showed that by keeping a "multimodal" belief (remembering that the hole could be in several different places at once) and actively probing to rule them out, the robot becomes much more reliable.

What This Paper Doesn't Say

It's important to note what BayesContact doesn't do. The authors are careful to say this isn't a magic wand that solves every robot problem.

It doesn't work for objects the robot has never seen before; it needs to know the shape of the peg and the hole in advance.
The biggest wins were seen in simulations and specific real-world tests. While the results are strong, the paper suggests this is a step forward in making robots more reliable, not a final solution to all robotic manipulation.
It explicitly argues against methods that try to learn everything from scratch using massive amounts of data (like some deep learning methods). Instead, BayesContact uses the laws of physics and geometry to reason its way through the problem, which means it doesn't need to be retrained every time the environment changes slightly.

In short, BayesContact gives robots a better way to think. Instead of just staring at a picture and hoping for the best, they run mental simulations, feel the world, and ask smart questions to find the truth. It's a shift from "I think I see it" to "I know where it is."

Technical Summary: BayesContact

Problem Statement
Contact-rich manipulation tasks, such as peg-in-hole insertion, require precise pose estimation that often exceeds the capabilities of depth-only sensing. Vision-based methods suffer from partial observability due to occlusions, sensor noise, and geometric ambiguities (e.g., hidden surfaces inside a hole). While existing learning-based approaches (Reinforcement Learning, Imitation Learning) can address these challenges, they typically require costly offline training and retraining for new environments or geometries. Furthermore, decoupling state and dynamics uncertainty remains a challenge for sample efficiency and Sim2Real transfer. The core problem is estimating the pose of a target object (specifically a hole) when the state is only indirectly observed through vision, proprioception, and contact, where raw force/torque (F/T) data does not directly reveal contact points or object pose without geometric context.

Methodology
The authors propose BayesContact, a Simulation-Based Inference (SBI) framework for online visuo-tactile pose estimation. The system maintains a particle belief over the object's pose ( $x \in SE(3)$ ) and updates this belief sequentially using depth observations and F/T-derived contact evidence.

Simulation-Based Likelihoods: Instead of relying on learned sensor models, BayesContact uses forward simulators to approximate observation likelihoods:
- Depth Likelihood: A graphics renderer predicts depth measurements for each pose hypothesis. The likelihood is scored using a pixel-wise Laplace inlier model mixed with a constant outlier model to handle noise and missing data.
- Contact Likelihood: A physics simulator predicts contact outcomes for a given pose hypothesis and probing action. F/T sensor data is converted into contact evidence on the peg surface. The likelihood is computed using a Chamfer-distance score between simulated and observed contact points, conditioned on the geometry and robot state.
- Joint Likelihood: During the contact phase, a joint log-likelihood fuses depth, contact evidence, and camera trajectory data.
Sequential Monte Carlo (SMC) Inference: The belief is represented as a weighted set of particles to handle multimodal distributions caused by occlusion or symmetry.
- Vision Phase: The system initializes a posterior using SMC with Metropolis-Hastings (MH) rejuvenation based on depth observations.
- Contact Phase: The vision posterior serves as a warm start. The system executes guarded probing actions (vertical dips with planar/yaw offsets). For each action, particles are proposed, simulated, and re-weighted using the joint likelihood. No MH rejuvenation is applied here to avoid the computational cost of repeated physics simulations.
Active Sensing: The framework leverages the particle belief to select "next-best" probing actions. It employs a myopic approximation of Information Gain (IG) to choose actions that maximize the expected reduction in posterior entropy, effectively turning contact into an active sensing mechanism to disambiguate pose hypotheses.

Key Contributions

Bayesian SBI Framework: Introduction of a framework for online pose estimation in contact-rich tasks that combines visual and tactile sensing without requiring offline training or retraining for new geometries.
Modular Likelihoods: Proposal of rendering-based depth and simulation-based contact likelihoods that allow flexible integration of graphics and physics engines for scoring pose hypotheses.
Empirical Validation: Demonstration through simulation and real-robot experiments that fusing depth vision and F/T contact significantly improves pose observability and insertion success across multiple geometries compared to vision-only inference.

Results
The authors evaluated BayesContact on five peg-in-hole geometries (varying in ambiguity) in both simulation (Drake) and on a physical KUKA iiwa14 robot.

Observability Improvement: In simulation, the Information-Gain (IG) probing strategy (BC-IG) reduced average ADD-S, position error, and orientation error by 35.8%, 46.3%, and 67.1% respectively, compared to vision-only SMC (BCv-SMC). The improvements were most pronounced in geometrically ambiguous scenarios (e.g., "Rectangle-teeth" and "Ellipse-teeth").
Real-Robot Performance: On the physical robot, BC-IG reduced ADD-S error by approximately 34–36% compared to vision-only baselines and improved insertion success rates by 20–30%.
Comparison to Parametric Methods: BayesContact (particle-based) significantly outperformed a Manifold Unscented Kalman Filter (Man-UKF), particularly in orientation error reduction (up to 73.4% improvement). This highlights the advantage of maintaining multimodal beliefs to resolve geometric ambiguity before contact provides disambiguating evidence.
Active Sensing: IG-based probing strategies achieved convergence to a target accuracy (ADD < 2mm) with fewer probes and higher insertion success rates compared to Maximum A Posteriori (MAP) and Thompson Sampling strategies.

Significance and Claims
The paper claims that BayesContact offers a robust alternative to learning-based methods by utilizing simulation tools for online inference without the need for retraining. The core significance lies in demonstrating that force-torque sensing becomes spatially informative when interpreted through geometry and simulation, transforming contact from a mere control feedback signal into a source of Bayesian evidence. By maintaining a multimodal particle belief, the system can preserve geometric ambiguity until physical interaction resolves it. The work further establishes that this belief representation enables effective information-gain-based active sensing, allowing the robot to actively probe the environment to resolve residual uncertainty, thereby improving the reliability of contact-rich manipulation tasks.