💻 computer science

ContactFusion: Stochastic Poisson Surface Maps from Visual and Contact Sensing

This paper introduces ContactFusion, a method that fuses visual point clouds with force-based contact sensing to generate Stochastic Poisson Surface Maps, thereby improving pose estimation and assembly success in tight-tolerance robotic tasks like peg-in-hole insertion.

Original authors: Aditya Kamireddypalli, Joao Moura, Russell Buchanan, Matias Mattamala, Sethu Vijayakumar, Subramanian Ramamoorthy

Published 2026-07-20

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Aditya Kamireddypalli, Joao Moura, Russell Buchanan, Matias Mattamala, Sethu Vijayakumar, Subramanian Ramamoorthy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to perform a delicate task, like sliding a peg into a tiny hole. To a human, this seems simple, but for a robot, it's a nightmare of precision. If the robot's "eyes" (cameras) are even a little bit blurry or confused by shadows, the peg might miss the hole by a fraction of a millimeter, causing the robot to jam or break the part. This is the world of robotic manipulation, where machines must understand the 3D shape of objects to interact with them. Usually, robots rely on cameras to build a mental map of the world, but cameras can be fooled by bad lighting or reflections. To fix this, scientists have started teaching robots to "feel" their way around, using sensors that detect when a robot's hand touches something. The big question is: how do you combine what a robot sees with what it feels to build a perfect, super-accurate map of an object, especially when both the eyes and the touch sensors make mistakes?

This is exactly the puzzle tackled by a new method called ContactFusion, created by researchers at the University of Edinburgh and the University of Waterloo. They realized that while cameras give a broad view, they can be noisy, and while touch sensors are precise, they only know about the tiny spot they are touching. To solve this, the team built a system that acts like a super-smart detective, merging visual clues with tactile hints to create a "Stochastic Poisson Surface Map" (or SPSMap). Think of this map not as a rigid drawing, but as a living, breathing cloud of possibilities. It doesn't just say, "The wall is here"; it says, "The wall is probably here, but we are 90% sure, and here is exactly how much we are guessing."

The core of their invention is a clever way to translate the robot's "ouch" or "push" signals (force and torque) into a guess about where the touch happened. If a robot arm bumps into a peg, the sensors measure the twist and push. The ContactFusion system uses math to work backward from that twist to figure out, "Ah, the robot must have touched the peg right here." It then fuses this "touch guess" with the "sight data" from the camera. The result is a map that gets sharper and more accurate every time the robot looks or touches. In their tests, this method was able to reconstruct the shape of objects up to 30–35% more accurately than previous methods that only used cameras or standard maps. Even better, because the map knows where it is unsure, the robot can use that uncertainty to decide where to look or touch next, effectively teaching itself to find the missing pieces of the puzzle faster.

The Problem: When "Seeing" Isn't Enough

Robots are great at moving, but they are terrible at guessing. When a robot tries to put a peg into a hole, it needs to know the exact shape and position of both objects. If the robot's camera sees the hole but the image is a bit fuzzy, or if the robot's hand bumps the peg slightly off-center, the whole task can fail. This is because real-world sensors are noisy; they make mistakes. A camera might think a hole is in one spot when it's actually a millimeter away. A millimeter might not sound like much, but for a robot trying to fit parts together, it's the difference between success and a jammed-up mess.

Traditionally, robots have tried to solve this by just looking harder or by using "compliant control," which is a fancy way of saying the robot moves its arm loosely so it doesn't break things if it hits the wrong spot. But this is a trade-off: if the robot is too loose, it can't move precisely; if it's too stiff, it breaks things. The researchers behind ContactFusion asked a different question: What if we could combine the robot's vision with its sense of touch to build a map that knows exactly where it is unsure?

The Solution: A Map That Knows What It Doesn't Know

The team introduced ContactFusion, a system that builds a special kind of map called an SPSMap. To understand what makes this special, imagine you are trying to draw a map of a cave in the dark. You have a flashlight (the camera) that gives you a general idea of the walls, but the light is flickering. You also have a long stick (the robot arm) that you use to tap the walls. When the stick hits something, you know exactly where the wall is at that one point, but you don't know what the rest of the cave looks like.

Old methods would just take the flashlight picture and the stick taps and try to mash them together, often resulting in a jagged, confusing map. ContactFusion, however, uses a mathematical trick called Stochastic Poisson Surface Reconstruction (SPSR). Instead of drawing a single, hard line for the wall, it draws a "cloud" of probability. It says, "Here is the most likely shape of the wall, but here is also how much we are guessing."

This "uncertainty" is the secret sauce. Because the map knows where it is unsure, the robot can use that information to take better actions. If the map is fuzzy in one corner, the robot knows to go touch that specific spot or look at it from a new angle. This is called active perception: the robot isn't just passively waiting for data; it's actively hunting for the information it needs to make its map perfect.

How It Works: Turning "Ouch" into "Aha!"

The magic happens in how the system handles the robot's touch. When a robot's hand touches an object, it doesn't just get a simple "contact" signal. It gets a complex mix of force (how hard it pushed) and torque (how much it twisted). The researchers built a contact location estimator to decode this.

Imagine the robot arm is a lever. If you push on the end of the lever, the base twists. The robot's sensors measure that twist. The ContactFusion system uses physics to work backward: "If the base twisted this much, the touch must have happened there." It turns the force and torque numbers into a list of "hypotheses" (guesses) about where the contact happened on the object's surface.

Once it has these guesses, it fuses them with the camera data. The camera says, "The object is roughly here," and the touch sensor says, "But I definitely touched it right here." The SPSR algorithm blends these two sources of information, updating the "cloud" of probability. Every time the robot looks or touches, the map gets clearer, and the "cloud" of uncertainty shrinks.

The Results: Smarter Maps, Better Robots

The researchers tested their system in two ways: in a computer simulation and on a real robot arm (a KUKA IIWA) equipped with a camera and a force sensor. They set up a classic "peg-in-hole" challenge and compared ContactFusion against other popular mapping methods, like Occupancy Maps (which just say "occupied" or "empty") and GPIS (another type of 3D map).

The results were clear. In the simulations, the ContactFusion maps were 30% more accurate than the other methods. When they tested it on the real robot, the improvement was even more impressive, reaching up to 35%. The maps produced by ContactFusion were not only more accurate but also "geometrically consistent," meaning the shapes looked smooth and logical, rather than jagged and broken.

One of the most interesting findings was how the system handled uncertainty. Because the map knew where it was unsure, the robot could use an active reconstruction strategy. Instead of just randomly poking the object, the robot would look at the map, see a blurry spot, and decide, "I need to touch that spot to clear up the confusion." This allowed the robot to identify the target shape much faster and more efficiently than if it had just followed a pre-set list of moves.

Why It Matters

This work suggests that the future of robotic assembly isn't just about building better cameras or stronger arms. It's about building robots that can think about what they don't know. By creating a map that explicitly models uncertainty, robots can make smarter decisions about how to explore their environment.

The researchers noted that this method does come with a cost: it takes more time to compute the map because it has to solve complex math equations for every new piece of data. However, the trade-off seems worth it for tasks that require high precision. As the authors suggest, future work could focus on speeding up these calculations, perhaps by using parallel computing, to make these smart, uncertainty-aware maps real-time for even the most complex assembly tasks.

In the end, ContactFusion shows that when robots learn to trust their "feel" as much as their "sight," and when they learn to ask "Where am I guessing?" instead of just "What do I see?", they become much better at the delicate, precise work that humans have always done best.

Technical Summary: ContactFusion: Stochastic Poisson Surface Maps from Visual and Contact Sensing

Problem Statement
Robotic assembly tasks, particularly peg-in-hole insertion, demand precise geometric reasoning and relative pose estimation between interacting objects. However, real-world sensor noise often exceeds the tight tolerances required for successful insertion. Vision-based pose estimation alone is frequently insufficient due to sensor noise, environmental occlusions, and robot-induced occlusions, leading to misalignment and task failure. While compliant control strategies can mitigate contact forces, they introduce a trade-off with trajectory tracking. The authors posit that complementing global visual information with fine-grained local contact information is a necessary approach to achieve the required precision.

Methodology: ContactFusion
The paper proposes ContactFusion, a probabilistic mapping framework that fuses depth sensing (vision) and force–torque (F/T) measurements to estimate the geometry of insertion targets. The core of the method is the construction of a Stochastic Poisson Surface Map (SPSMap), an uncertainty-aware implicit surface representation.

Stochastic Poisson Surface Reconstruction (SPSR):
The framework builds upon the SPSR method, which extends standard Poisson Surface Reconstruction by placing a Gaussian process prior over the implicit function. This allows the system to maintain a posterior field $f(x) | O \sim \mathcal{N}(f_{SPSR}(x), \sigma^2(x))$ , providing not only a surface estimate but also a variance map $\sigma^2(x)$ . This enables closed-form geometric queries, such as occupancy probability $P(f_{SPSR}(x) \leq 0)$ and surface probability $P(f_{SPSR}(x) = 0)$ .
Sequential Fusion:
ContactFusion implements an online sequential estimation procedure. It updates the implicit field iteratively using the operator defined by SPSR:
$f^i_{SPSR}(x), \sigma^i(x) = SPSR(O_i, \sigma^i_O, f^{i-1}_{SPSR}(x))$
where $O_i$ represents the oriented point set from the current measurement iteration.
Multi-Modal Inputs:
- Vision: Depth observations from an RGB-D camera are converted into oriented point sets (3D points and surface normals estimated via K-Nearest Neighbors).
- Contact: The authors introduce a contact location estimator that transforms raw force–torque measurements into spatial hypotheses. Assuming a single-point contact and rigid-body mechanics, the estimator computes candidate contact locations on the object surface that satisfy the measured wrench (force and torque). Candidates inconsistent with the measured torque are rejected (via a residual function), while consistent candidates form an oriented point set with normals aligned to the measured force direction.
Reconstruction Strategies:
The system supports two strategies for acquiring measurements:
- Passive Reconstruction: Executes a pre-determined sequence of random vision and probing actions.
- Active Reconstruction: Utilizes the uncertainty estimates ( $\sigma^2(x)$ ) from the SPSMap to guide a "Next-Best-Action" approach. The system selects vision viewpoints or probing locations that intersect areas of highest uncertainty, thereby accelerating the identification of the target geometry.

Key Contributions

Probabilistic Mapping Framework: A novel integration of vision and contact sensing using SPSR to create an uncertainty-aware representation (SPSMap) suitable for online sequential updates.
Contact Location Estimator: A method to convert F/T sensor data into spatial likelihoods over candidate contact locations, enabling the fusion of non-geometric force data with geometric depth data.
Active Perception: Demonstration that the uncertainty estimates inherent in SPSMap can drive active reconstruction strategies to improve efficiency.

Experimental Results
The framework was evaluated in simulation and on a real robotic system (KUKA IIWA with Realsense D435 and ATI F/T sensor) using peg-in-hole scenarios with ARCH and RECTANGLE geometries.

Reconstruction Accuracy: Compared to baselines including Gaussian Process Implicit Surfaces (GPIS) and Occupancy Maps (OccMap), SPSMap demonstrated superior performance.
- In simulation (Exp. 1), SPSMap achieved up to 30% improvement in F-score over baselines at a 2mm tolerance.
- In real-world experiments (Exp. 3), SPSMap showed up to 35% improvement in F-score.
- SPSMap produced smoother, more geometrically consistent reconstructions, particularly in sparsely observed regions, whereas OccMap often produced fragmented results.
Active Reconstruction: In active scenarios (Exp. 2), the uncertainty-guided strategy using SPSMap consistently outperformed an occupancy-map-based frontier exploration baseline, achieving higher F-scores with fewer measurements.
Computational Cost: The authors acknowledge a significant computational overhead. Updating the SPSMap takes approximately 9,900 ms per measurement (due to solving a Partial Differential Equation), compared to 5,800 ms for GPIS and 200 ms for Occupancy Maps.

Significance and Claims
The paper claims that ContactFusion addresses the critical gap between sensor noise and the tight tolerances required for robotic assembly. By explicitly modeling uncertainty and fusing visual and tactile modalities, the system produces more accurate and geometrically consistent reconstructions than existing methods. The authors emphasize that the ability to perform statistical queries (occupancy and surface probability) and the provision of uncertainty estimates are key differentiators that enable active perception strategies. While the method incurs higher computational costs, the authors argue that the resulting accuracy and geometric coherence justify the overhead for precision manipulation tasks. The work does not claim to solve all manipulation challenges but specifically targets the improvement of geometric state estimation for insertion tasks through multi-modal fusion.