💻 computer science

Differentiable Mesh State Estimation via Factor Graph Inference for Deformable Object Reconstruction

This paper proposes a novel factor graph-based framework that unifies physics priors, noisy sensor data, and temporal constraints to perform accurate, probabilistic state estimation and reconstruction of deformable objects represented as tetrahedral meshes.

Original authors: Lidia Al-Zogbi, Fangjie Li, Samuel Tobin, James Ferguson, Nithesh Kumar, Alejandro Chara, Kuan-I Chung, Mingxing Rao, Ayberk Acar, Susheela Sharma Stern, Robert Webster, Daniel Moyer, Alan Kuntz, Cale

Published 2026-09-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lidia Al-Zogbi, Fangjie Li, Samuel Tobin, James Ferguson, Nithesh Kumar, Alejandro Chara, Kuan-I Chung, Mingxing Rao, Ayberk Acar, Susheela Sharma Stern, Robert Webster, Daniel Moyer, Alan Kuntz, Caleb Rucker, Tucker Hermans, Jie Ying Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long been masters of the rigid world, able to move with precision through factories filled with steel beams and unyielding gears. But the real world is often soft, squishy, and unpredictable. When a robot tries to interact with a living body, a piece of fabric, or a lump of clay, the object changes shape the moment it is touched. To navigate this fluid environment, a robot cannot simply rely on a static map; it must constantly guess how the object is deforming, even when parts of it are hidden from view. This challenge sits at the intersection of geometry, physics, and sensing, where the goal is to build a digital twin of a soft object that updates in real time as the object moves and bends. If a robot cannot accurately track these changes, it risks damaging delicate tissues or failing to complete a task.

A team of researchers has developed a new way to solve this problem, creating a system that can estimate the shape of a deforming object by combining what it sees with how it knows the object should physically behave. Instead of treating the object as a collection of separate points or a simple surface, the researchers represent it as a three-dimensional mesh made of tiny tetrahedrons, which are pyramid-like shapes that fill the volume of the object. This mesh acts like a digital skeleton that can stretch and twist. The system works by constantly adjusting the position of every point in this mesh to satisfy three different demands at once: it must match the noisy, imperfect data coming from sensors; it must follow the laws of physics that govern how soft materials deform; and it must move smoothly from one moment to the next without jumping erratically.

The researchers tested this approach in two ways: first, by running computer simulations of a deforming cube, and second, by performing experiments on a physical model of a human airway blockage. In the airway experiment, they used a sheep trachea with a piece of chicken breast inserted to mimic a tumor, creating a scenario where a robotic tool pushed and deformed the tissue. Because the tool and the surrounding anatomy often block the view, the sensors could only see parts of the surface. The system had to infer the shape of the hidden parts based on the visible ones and the physical properties of the tissue. The results showed that by blending the sensor data with a physics-based model, the system could reconstruct the entire shape of the object with high accuracy, even in areas where no sensors could see.

In the computer simulations, the researchers found that relying on sensor data alone led to massive errors, with the reconstructed shape deviating by hundreds of millimeters from the true shape. Adding a rule that the object should move smoothly over time helped, but the biggest improvement came from including the physics of the material. When the system used a model that understood how the material resisted stretching and bending, the error dropped to less than one millimeter. This held true even for the parts of the object that were completely hidden from the sensors. The physics model acted as a guide, allowing the system to "feel" how the hidden parts must have moved based on how the visible parts were deforming.

The real-world tests on the airway model confirmed these findings. The system successfully tracked the deformation of the tissue as a robotic tool pushed against it. While the sensor data alone was precise where it could see, it was sparse and often missed the full picture. The physics model provided a global understanding of the tissue's behavior, filling in the gaps. By combining the two, the system achieved a more complete and accurate picture than either method could provide on its own. The researchers noted that this approach allows a robot to maintain a reliable estimate of the object's state, even when the view is partially blocked or the data is noisy.

This work suggests a path forward for robots that need to operate in delicate, changing environments. By treating the estimation of a soft object's shape as a problem of finding the most likely configuration that satisfies both observation and physical law, the researchers have created a method that is robust and principled. The system does not just guess; it calculates the most probable shape by weighing the evidence from sensors against the constraints of physics. While the current experiments were limited to specific scenarios and did not yet address the extreme complexity of cutting or tearing tissue, the results demonstrate that this probabilistic framework can effectively track deformable objects. This capability is a crucial step toward enabling robots to perform complex medical procedures, such as removing blockages from airways, with the safety and precision required for working inside the human body.

Technical Summary: Differentiable Mesh State Estimation via Factor Graph Inference for Deformable Object Reconstruction

Problem Statement
Estimating the state of deformable objects remains a fundamental challenge in robotics and simulation, particularly when updating time-varying tetrahedral meshes from partial, noisy point cloud observations. Existing approaches face significant limitations: Kalman-based methods struggle with large deformations due to local linearization and quadratic scaling; online filters fail to exploit global temporal structure or revise past estimates; physics-based variational methods often require accurate constitutive parameters and suffer from poor conditioning under noise; and data-driven approaches lack guarantees of physical plausibility. The core challenge lies in balancing measurement fidelity with physics-based regularization in a high-dimensional, non-convex state space.

Methodology
The authors propose a novel factor graph-based framework that formulates mesh state estimation as a unified probabilistic inference problem. The method represents the deformable object as a tetrahedral mesh where the state variables are the 3D coordinates of all vertices ( $X_t$ ). The estimation is posed as a nonlinear least-squares optimization problem solved via the Levenberg–Marquardt algorithm, utilizing automatic differentiation (PyTorch) to handle differentiable residuals.

The framework integrates three primary factor types within the graph:

Physics Prior Factor: Enforces physically plausible nodal displacements using a finite element stiffness matrix ( $K_t$ ) as the information matrix. This factor models temporal evolution as a probabilistic transition where a deformable simulator predicts the next state, penalizing deviations from mechanical consistency.
Point-to-Surface Measurement Factor: Penalizes the error between noisy sensor measurements and the mesh surface. This involves projecting measurement points onto the closest surface triangles and computing residuals in barycentric coordinates. This factor is differentiable with respect to vertex positions.
Temporal Smoothness Factor: Connects consecutive states to encourage temporal consistency, penalizing large inter-frame displacements.

The optimization minimizes a composite cost function comprising the weighted sum of residuals from these factors (physics, measurement, and temporal), allowing for joint spatiotemporal optimization.

Key Contributions

Unified Probabilistic Formulation: The paper introduces a factor graph representation that treats mesh estimation as a structured inference problem, enabling the fusion of heterogeneous data sources (geometric measurements, mechanical priors, and temporal constraints) into a single optimization objective.
Differentiable Mesh Inference: By defining residuals that are differentiable with respect to the mesh state itself, the approach transforms mesh reconstruction from a procedural registration step into a differentiable inference process. This allows gradients from point-to-surface errors, temporal regularization, and stiffness-based priors to jointly update the mesh.
Handling Partial Observability: The framework is designed to infer the state of unobserved surface regions by propagating information from observed areas through the physics-based stiffness prior, addressing the issue of partial observability common in robotic manipulation.

Experimental Results
The method was evaluated through two primary experimental setups:

Simulation-Based Studies: Using a deforming cube model, the authors conducted ablation studies comparing different factor configurations. Results indicated that the "measurement-only" configuration yielded high errors (RMSE ~280 mm). Adding temporal smoothness reduced errors but left the system poorly constrained. The inclusion of the physics-based simulation prior resulted in the lowest reconstruction errors (RMSE ~0.73 mm), demonstrating that the physics prior is the principal contributor to accuracy in this setting. The full model (combining simulation, measurement, and temporal factors) achieved comparable performance to the simulation-plus-measurement configuration.
Ex Vivo Studies: The framework was tested on a central airway obstruction (CAO) model using a sheep trachea and a chicken breast tumor mimic, manipulated by a robotic system. Ground truth was obtained via intra-operative CT scans.
- The factor graph approach achieved a lower Chamfer distance ( $d_{CD}$ ) and 95th-percentile distance ( $d_{H95}$ ) compared to tissue simulation alone, particularly in unobserved regions.
- While Metric Monocular Depth Estimation (MDE) provided high local precision (lowest $d_{RMSE}$ ) in observable regions, it suffered from sparse coverage at grazing angles.
- The factor graph successfully leveraged the MDE's local precision while using the tissue-mechanics prior to constrain the global deformation state, resulting in better overall surface coverage and robustness.

Significance and Claims
The paper claims that this probabilistic approach offers a principled, measurement-driven method for mesh state estimation that overcomes the limitations of purely geometric or purely physics-based methods. The authors state that the framework demonstrates reliable and accurate reconstruction under both rigid motion and deformation.

The significance lies in the ability to maintain accurate estimates of airway and tumor geometry during tissue manipulation and cutting, even when observations are sparse, partial, or noisy. The authors position this work as a prototype that highlights the potential of unifying measurements and physics within a factor graph framework to transform how robots perceive and interact with deformable environments. They modestly note that future work is required to address sensor noise models beyond identity-weighted matrices, extend validation to more physical experiments, and handle complex cases like mesh topology changes during cutting.

Technical Summary: Differentiable Mesh State Estimation via Factor Graph Inference for Deformable Object Reconstruction

More like this