💻 computer science

Learning to Predict Contact Force Distributions from Vision Leveraging Object Geometry Priors

This paper proposes a vision-based approach that leverages object geometry priors to predict smooth 3D contact force distributions rather than noisy point forces, significantly improving prediction accuracy and downstream manipulation performance while demonstrating strong generalization from simulation to the real world.

Original authors: Ryo Hanai, Yukiyasu Domaea, Ixchel G. Ramirez-Alpizar, Abdullah Mustafa, Floris Erich, Tetsuya Ogata

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ryo Hanai, Yukiyasu Domaea, Ixchel G. Ramirez-Alpizar, Abdullah Mustafa, Floris Erich, Tetsuya Ogata

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to pick up a toy from a messy pile of toys. If the robot just grabs blindly, it might knock over the whole tower, crushing the toy it wanted to save. To avoid this, the robot needs to "feel" the pile without actually touching it first. This is the world of robotic manipulation, where machines try to understand the physical world just by looking at it. The big challenge here is contact force: the invisible push and pull that happens when objects touch each other. In the real world, when a box sits on a table, it doesn't touch at just one tiny dot; it rests on a whole surface. But for a long time, computer simulations—the virtual test drives robots use to learn—only showed these forces as sharp, lonely dots. This made it hard for robots to learn how to move gently in a messy, real-world pile where things touch along edges and flat faces. Scientists care about this because if robots can predict these invisible forces, they can become better at sorting trash, helping in warehouses, or even just tidying up a bedroom without breaking everything.

This paper introduces a clever new way to teach robots to "see" these invisible pushes. The researchers started by building a virtual world using a rigid-body simulator, a tool that calculates how solid objects bounce and stack. However, they noticed a problem: the simulator only gave them "point forces," like a map showing only single dots where objects touch. In reality, a box resting on a table touches along a whole bottom surface, not just a few dots. If a robot tried to learn from these sharp dots, it would get confused and make messy predictions.

To fix this, the team came up with a method called geometry-aware statistical smoothing. Think of it like this: if you drop a handful of glitter (the sharp dots from the simulator) onto a piece of paper, it looks scattered and messy. But if you gently blow on the glitter, it spreads out into a soft, smooth cloud that matches the shape of the paper underneath. The researchers did this digitally. They took the sharp, noisy dots from the simulator and "smoothed" them out, but with a special twist: they used the shape of the objects as a guide. If two objects were touching along a flat surface, the "glitter" spread out to cover that whole flat area. If they touched at a sharp corner, the "glitter" stayed tight around that corner. This created a smooth, 3D map of forces that looked much more like how humans intuitively understand contact.

They trained a robot brain (a deep learning model) using only these smoothed, simulated pictures. The goal was to see if the robot could look at a single photo of a messy pile and guess where the forces were pushing, even for objects it had never seen before. The results were promising. When they tested the robot in a real-world lab with actual piles of boxes and cans, the model worked surprisingly well. It successfully predicted where to pull an object out of a pile without knocking everything over. Specifically, when using their new "shape-guided" smoothing method, the robot caused less disturbance to the surrounding objects compared to older methods that just smoothed the dots evenly in all directions.

The paper suggests that while the robot wasn't trained on real-world data, it learned a general rule about how forces work that transferred perfectly to the real world. In their experiments, the new method helped the robot pick objects with a success rate of about 97.9% to 99.2% in simulations, and it performed consistently well in real-world trials, causing less "maximum velocity" (speed of movement) and less "maximum contact force" (hardness of bumps) to the surrounding items than the older methods. The researchers found that if they smoothed the forces too much, the robot got confused about where exactly to pull, but with the right amount of "geometry-aware" smoothing, it found the sweet spot.

Ultimately, the paper shows that you don't need a perfect, physics-accurate simulation to teach a robot. Instead, by teaching the robot to look for smooth, shape-based patterns of force rather than sharp, noisy dots, you can give it a "rough but useful" intuition. This allows a robot trained entirely in a computer game to walk into a real kitchen, look at a pile of dishes, and figure out how to grab a plate without crashing the whole stack. The authors admit that their method assumes objects are solid and doesn't handle squishy, deforming things perfectly, but for rigid objects like boxes and cans, this "visual force prediction" is a significant step toward robots that can handle the messy, unpredictable world of everyday life.

Technical Summary: Learning to Predict Contact Force Distributions from Vision Leveraging Object Geometry Priors

1. Problem Statement

Robots require the ability to make rough physical predictions about unknown objects and novel scenes based on visual information to adapt their manipulation strategies flexibly. While vision-based force prediction offers non-contact estimation advantages, the problem is inherently ill-posed; high accuracy is difficult to achieve, and predictions can be easily misled by objects with identical appearances but different physical properties or hidden contents.

A specific challenge arises when using physics simulators to generate training data. Standard rigid-body simulators output sparse point forces at contact points. However, in the real world, contact often occurs over surfaces or lines, not just points. Directly predicting these sparse point forces from images leads to overfitting and poor generalization to novel objects and scenes because the predictions are highly sensitive to fine contact states and mesh structures. Furthermore, previous attempts to bridge this gap using isotropic statistical smoothing failed to account for specific contact patterns (e.g., surface vs. point contact), often spreading forces incorrectly into neighboring regions or along normal directions where they do not belong.

2. Methodology

2.1 Overall Framework

The authors propose an end-to-end deep learning framework that predicts a 3D force distribution from a single RGB image of a piled scene. The system operates in three stages:

Data Generation: Paired data (RGB images and contact forces) are generated using a rigid-body simulator (Isaac Sim).
Label Processing: The raw point forces from the simulator are converted into smooth, consistent distributions using a novel Geometry-Aware Statistical Smoothing technique.
Training & Application: A ResNet-based encoder-decoder model is trained on synthetic data (using domain randomization) to predict these smoothed distributions. The trained model is then applied to real-world scenes to plan lifting directions that minimize disturbance to surrounding objects.

2.2 Geometry-Aware Statistical Smoothing (GAFS)

To address the gap between simulated point forces and real-world surface/line contacts, the authors propose a smoothing method that incorporates object geometry as a prior. Unlike previous isotropic Kernel Density Estimation (KDE) methods, GAFS attenuates weights based on the distance from the contacting objects.

The force density $\tilde{f}(p_j)$ at a voxel $p_j$ is calculated as:
$\tilde{f}(p_j) = \sum_i \alpha_i f_i \tilde{K}(p_j, c_i)$
Where:

$f_i$ is the magnitude of the contact force at point $c_i$ .
The kernel $\tilde{K}$ includes terms that decay based on the distance from the voxel to the surfaces of the two contacting objects ( $A_i$ and $B_i$ ).
This ensures that smoothing spreads primarily along the contact surface (where distance to both objects is near zero) but decays rapidly in the normal direction (where distance to at least one object increases).
For point contacts, the smoothing effect decays even near the contact point, keeping the distribution localized.

This method utilizes a Voxel-based Signed Distance Function (SDF) derived from visual meshes to efficiently compute distances.

2.3 Network Architecture and Training

Model: A ResNet-based encoder-decoder (ResNet50 encoder, Residual U-Net decoder).
Input: Single RGB image.
Output: A 3D voxel map representing force magnitude. The direction is approximated later from the magnitude gradient.
Loss Function: Mean Squared Error (MSE) on log-scaled, normalized force values. This structure-aware loss penalizes small errors more heavily, which is crucial for planning.
Data: The dataset consists of piled scenes using YCB objects and scanned convenience store items. Training uses only synthetic data with domain randomization (camera parameters, lighting, object color).

3. Key Contributions

Geometry-Aware Smoothing: A new method for smoothing force labels that incorporates object geometry priors to better model surface and line contacts, preventing the undesired spread of forces into neighboring regions.
Analysis of Smoothing Effects: An investigation into how smoothing methods and strength affect prediction accuracy on unseen scenes composed of unseen objects.
Downstream Task Evaluation: A novel evaluation framework for the utility of predicted force distributions using a "lifting direction planning" task. This includes metrics for physical disturbance (total distance, max velocity, max contact force) and physical interpretation metrics (Gravity Support, Torque Balance, Low Resistance Direction Agreement).
Sim-to-Real Generalization: Demonstration that a model trained solely on synthetic data with rigid-body simulation generalizes effectively to real-world piled scenes for downstream manipulation tasks.

4. Experimental Results

4.1 Prediction Accuracy (RQ1)

Comparison: The proposed Geometry-Aware Force Smoothing (GAFS) was compared against Isotropic Force Smoothing (IFS) with varying strengths and raw point forces.
Findings: GAFS achieved lower prediction errors near object surfaces compared to IFS. Specifically, GAFS produced distributions that aligned better with actual contact geometries (elongated for line contacts, flat for surface contacts).
Metric: Normalized Mean Squared Error (MSE) and surface-shell errors ( $L(d)$ ) showed that incorporating geometry awareness further reduced errors, particularly as the evaluation region narrowed to the object surface.

4.2 Downstream Task Performance (RQ2)

Task: Planning a lifting direction to extract a target object from a pile while minimizing disturbance to surrounding objects.
Simulation Results:
- Methods using smoothed forces significantly reduced maximum velocity (23–24%) and maximum contact force (9–14%) compared to a baseline "No-Force (UP)" strategy.
- GAFS outperformed IFS in reducing disturbances, particularly in complex scenes.
- Physical Interpretation: GAFS showed better Gravity Support (GS) and Low Resistance Direction Agreement (LRDA) for objects with surface contacts (e.g., flat boxes, scissors). IFS performed better for objects where precise point contact localization was critical (e.g., bowls), but GAFS provided more consistent performance across diverse scenes.
- Observation: Excessive smoothing (high $\sigma_f$ ) in GAFS degraded performance, suggesting a balance is needed between geometry weighting and force locality.

4.3 Sim-to-Real Transfer (RQ3)

Setup: Real-world experiments using a UR5e robot and RealSense cameras on seven piled scenes with unseen objects.
Evaluation: Disturbance was measured via 6DoF pose tracking of surrounding objects (using FoundationPose).
Results:
- GAFS consistently achieved the smallest disturbances in most real-world scenes compared to IFS and the baseline.
- GAFS demonstrated lower variance in disturbance metrics, indicating more consistent lifting direction planning.
- Visualizations showed GAFS correctly identified surface constraints (e.g., between a can and a box) to find safe lifting paths, whereas IFS sometimes suggested directions that would push adjacent objects.
- The model successfully generalized to real scenes despite being trained only on synthetic rigid-body data.

5. Significance and Claims

The paper claims that predicting smooth force distributions rather than raw point forces is a viable strategy for enabling robots to make coarse yet useful physical predictions. The significance lies in:

Bridging the Sim-to-Real Gap: By using geometry-aware smoothing, the method mitigates the discrepancy between sparse simulator outputs and continuous real-world contacts, allowing a model trained on rigid-body simulation to perform effectively in the real world.
Practical Utility: The predicted distributions are not just theoretical; they directly improve downstream manipulation tasks (lifting) by identifying directions that minimize disturbance to cluttered environments.
Generalization: The approach demonstrates that robots can learn to anticipate physical interactions in unknown scenes using only visual input and simulation-derived priors, without requiring extensive real-world force data collection.

The authors acknowledge limitations, including the assumption of rigid bodies (handling only small deformations via smoothing), fixed friction coefficients, and the challenge of predicting forces for transparent objects or those with hidden internal contents. They suggest future work could involve continuous force prediction during lifting and leveraging additional modalities (tactile, language) for more complex physical reasoning.