🤖 machine learning

Deformable State Estimation for Autonomous Surgical Tissue Retraction Under Partial Observability

This paper proposes a learned, geometry-regularized state estimator that reconstructs full deformable tissue states from sparse, noisy observations to enable effective autonomous surgical retraction planning under partial observability, achieving near-oracle performance in simulations.

Original authors: Everest Yang, Skye Thompson, George D. Konidaris

Published 2026-07-16

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Everest Yang, Skye Thompson, George D. Konidaris

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots are learning to perform delicate surgery, but they face a tricky problem: they are trying to move soft, squishy things like skin or organs, which don't behave like rigid blocks of metal. These tissues are "deformable," meaning they stretch, fold, and twist in complex ways, much like a wet towel or a piece of dough. To do this safely, a robot needs to know exactly where every part of the tissue is at all times. However, in a real surgery, the robot's "eyes" (cameras) often get blocked by the tissue itself or other instruments, and the images they see are fuzzy and full of static. This is called "partial observability." It's like trying to guess the shape of a giant, invisible sheet by only being able to poke four random spots on it while wearing thick gloves. If the robot guesses wrong about the shape, it might pull too hard and cause damage. So, the big question for scientists is: Can we teach a robot to figure out the full, hidden shape of this squishy tissue just by looking at a few noisy, scattered dots?

This paper, titled "Deformable State Estimation for Autonomous Surgical Tissue Retraction Under Partial Observability," tackles exactly that challenge. The authors, Everest Yang, Skye Thompson, and George D. Konidaris from Brown University, propose a clever new way for robots to "see" the invisible parts of the tissue. Instead of trying to simulate the physics of the tissue from scratch every time (which is slow and computationally heavy), they built a "learned state estimator." Think of this estimator as a super-smart detective who has studied thousands of examples of how this specific type of tissue moves. When the robot sees only 40 noisy dots out of a possible 900 (about 4.4% of the surface), the detective uses a mathematical trick called Principal Component Analysis (PCA) to compress the complex shape into a simple, 12-dimensional "secret code." The robot then uses a neural network (a type of AI brain) to translate those 40 noisy dots into that secret code, and finally, expands the code back into a full, smooth 2D map of the tissue.

To make sure this AI detective doesn't come up with impossible shapes (like a tissue that stretches like rubber or folds in ways that defy physics), the researchers added "geometry-aware regularization." You can think of this as a strict teacher who constantly checks the student's work, ensuring that the tissue remains smooth and that the distance between connected points doesn't stretch too far, just like a real piece of cloth. They trained this system in a computer simulation where a robot had to pull a sheet of "tissue" to reveal a hidden target underneath. The results were impressive: in a single pull, the AI was slightly better than a basic guesser and almost as good as a perfect "oracle" (a robot that could see the entire tissue perfectly). But the real magic happened in multi-step planning. When the robot had to pull, look, pull again, and look again, the AI's performance was 98.1% as good as the perfect oracle. This suggests that by using this learned, geometry-regularized approach, robots might soon be able to perform complex, multi-step surgical tasks without needing expensive, slow physics simulations for every single move, even when their view is messy and incomplete. The authors note that while these results are from simulations, they are a strong step toward making autonomous surgical robots more reliable in the real world.

Technical Summary: Deformable State Estimation for Autonomous Surgical Tissue Retraction Under Partial Observability

Problem Statement

Autonomous surgical tissue retraction is a critical task in minimally invasive surgery, requiring the manipulation of soft, deformable tissue to expose underlying anatomical targets. This task is complicated by two primary factors: the high-dimensional, nonlinear behavior of deformable tissue and the constraints of surgical perception. In realistic operating conditions, sensing is partial and noisy; decision-makers typically have access only to sparse surface observations (e.g., a few visible vertices) rather than dense visual data or full state access.

Existing approaches often rely on full state access combined with repeated physics-based simulations (finite-element or spring-based models) or learned policies trained on rich visual observations. However, these methods either incur prohibitive planning-time costs due to repeated simulation or operate under observation regimes (dense images, stereo reconstruction) that do not match the sparse, noisy point observations typical of constrained surgical views. The core challenge addressed by this work is achieving effective planning for deformable manipulation when accurate state estimation must be derived directly from sparse, noisy observations.

Methodology

The authors propose a system that integrates a learned state estimator with a planning loop to handle partial observability. The tissue is modeled as a cloth-like sheet discretized on a $30 \times 30$ grid of vertices (1,800-dimensional state). At each planning step, the system observes only 40 randomly selected vertices (4.4% of the total), with coordinates corrupted by Gaussian noise ( $\sigma = 0.02$ ).

A. Learned State Estimator

The core of the approach is a learned estimator that reconstructs the full deformable mesh state from the sparse observations:

Latent Representation: The authors utilize Principal Component Analysis (PCA) to compress the high-dimensional mesh states into a low-dimensional latent space. Using 5,000 randomly generated mesh states, a 12-dimensional latent space was identified, capturing approximately 83.7% of the variance.
Network Architecture: A three-layer Multilayer Perceptron (MLP) maps the noisy, partial observations (vertex coordinates and their indices) to this 12-dimensional latent state.
Reconstruction: The full mesh configuration is reconstructed from the latent state via inverse PCA and provided to the planner.

B. Geometry-Regularized Training

To ensure the estimator produces physically plausible deformations, the training objective incorporates geometry-aware regularization. The loss function is defined as:
$L = L_{recon} + \lambda_s L_{smooth} + \lambda_e L_{stretch}$
Where:

$L_{recon}$ is the mean squared error between the predicted and ground-truth mesh vertices.
$L_{smooth}$ penalizes differences between neighboring vertices in the reconstructed mesh to encourage smoothness.
$L_{stretch}$ penalizes deviations of edge lengths in the reconstructed mesh from the undeformed reference configuration.
Regularization weights are fixed at $\lambda_s = \lambda_e = 0.1$ .

Training data is generated on-the-fly by sampling random mesh configurations and applying the same observation model (40 noisy vertices) used during inference.

C. Planning Framework

The system evaluates both single-step and multi-step retraction planning. Actions are defined as patch-based pulling operations parameterized by a grasp point and a pull displacement. The planner selects actions by sampling 200 candidate pulls and choosing the one that maximizes the predicted exposure score (the fraction of the target area not covered by the sheet). Deformation resulting from actions is modeled using a diffusion-based relaxation process.

Experimental Results

Experiments were conducted in a 2D deformable sheet simulation over 200 episodes. The system was compared against an "oracle" (which has full state access) and a naive baseline (which fills unobserved vertices with undeformed reference positions).

Single-Step Performance: The learned estimator achieved a mean exposure improvement of 0.1055 with a 50.0% success rate. This outperformed the oracle (0.0989 improvement, 47.0% success) and the naive baseline (0.0951 improvement, 45.5% success).
Multi-Step Performance: In a two-step planning scenario, the learned estimator achieved a mean improvement of 0.0941 with a 45.0% success rate. This corresponds to 98.1% of the oracle's performance (0.0958 improvement).
Oracle Analysis: The authors validated that the oracle is effectively near-optimal even with low computational budgets (200 candidate actions). Statistical analysis ( $p=0.835$ ) showed that increasing the action budget from 200 to 2000 yielded negligible gains (2.7% improvement), confirming that the primary bottleneck is state estimation, not action search.
Success Rate Context: The authors note that approximately 57% of episodes began with the target already fully exposed, making them unimprovable under the defined success threshold ( $\Delta > 0.05$ ). On the remaining improvable episodes, the oracle succeeded in 100% of cases, reinforcing that the planner is near-optimal and the challenge lies in estimation.

Key Contributions and Significance

The paper demonstrates that a learned state estimator, operating directly on sparse and noisy observations, can enable effective planning for autonomous tissue retraction. By combining a PCA-based latent representation with geometry-aware regularization (smoothness and stretch constraints), the system achieves multi-step performance statistically equivalent to a full-state oracle.

The significance of this work lies in its demonstration that planning-sufficient learned representations can replace expensive physical simulations for deformable manipulation under realistic perception constraints. The results suggest that accurate state estimation under partial observability is achievable without relying on dense visual data or computationally intensive repeated simulations, offering a viable pathway for autonomous surgical systems operating in constrained environments.

The authors conclude modestly, noting that future work will focus on validating these methods on real tissue data and investigating robustness under varying observation densities and noise levels.