Topology-Agnostic Mesh Reconstruction of Deformable Objects from Sparse Touch

Technical Summary: Topology-Agnostic Mesh Reconstruction of Deformable Objects from Sparse Touch

Problem Statement
Estimating the full shape of deformable objects (ropes, cloths, soft bodies) is a critical prerequisite for robotic manipulation tasks such as folding, routing, and grasping. While most existing perception systems rely on vision, many practical scenarios—such as reaching into opaque containers, operating in darkness, or manipulating objects hidden behind a gripper—render vision unavailable or degraded. In these settings, touch is the only reliable sensor. However, tactile data is inherently sparse, local, and costly to acquire. The paper addresses two coupled challenges: (1) Can a robot reconstruct a full 3D mesh of a deformable object using only a handful of sparse, noisy touch observations without vision? (2) Can the system actively decide where to touch next to maximize information gain per contact?

Methodology
The authors propose a unified framework comprising a topology-agnostic estimator and a learned active sensing policy.

• Topology-Agnostic Estimator:
  The core of the system is a single neural network architecture based on permutation-invariant cross-attention.

  • Input Encoding: Each touch observation is encoded as a token containing the noisy 3D coordinates and the normalized grid index of the touched vertex.
  • Architecture: The tokens are embedded and processed via a multi-head self-attention layer to contextualize observations. A fixed bank of query tokens, corresponding to every vertex in the output mesh, then performs cross-attention against these contextualized observations.
  • Topology Agnosticism: The model handles 1D (rope), 2D (cloth), and 3D (volumetric soft body) objects using the exact same weights. The only variable is the dimensionality of the query grid: $(H, 1)$ for ropes, $(H, W)$ for cloth, and $(D, H, W)$ for volumes. This design preserves locality, allowing each output vertex to read specific observations rather than relying on a pooled summary.
  • Uncertainty Estimation: A deep ensemble of five independently initialized estimators provides a per-vertex belief (mean position) and epistemic uncertainty (variance), which is high where observations under-constrain the shape.

• Learned Active Sensing:
  To determine the next touch location, the authors train an acquisition policy.

  • Oracle: An expected-error-reduction (EER) oracle serves as an upper bound by tentatively adding candidate touches and measuring the resulting true error reduction against ground truth.
  • Policy Training: A small Multi-Layer Perceptron (MLP) is trained via regression against the oracle using on-policy rollouts with $\epsilon$-greedy exploration. The policy scores candidate vertices based on belief features (ensemble mean, variance, grid position, fraction observed, and distance to nearest observation).
  • Efficiency: Unlike the oracle, which requires multiple forward passes per candidate, the learned policy amortizes this lookahead, requiring only a single ensemble reconstruction and a cheap scoring pass at deployment.

Key Contributions

1. Single Topology-Agnostic Estimator: A cross-attention network that reconstructs full vertex meshes for 1D, 2D, and 3D deformable objects from sparse touch alone, without requiring vision.
2. Learned Active Policy: An acquisition policy that utilizes ensemble uncertainty to select touch locations, demonstrating gains over random touching and Gaussian-process baselines, particularly under sparse budgets and high self-occlusion.
3. Delimitation of Active Sensing Value: A negative result showing that active touch placement provides negligible benefit when vision is available, as the combination of a camera and a learned prior sufficiently constrains the hidden state.

Experimental Results
Experiments were conducted in the MuJoCo flex simulator on three object types: a 20×20 cloth, a 40-vertex rope, and a 6×6×6 volumetric soft body.

• Estimation Performance: The learned estimator significantly outperforms non-learned baselines. Compared to Inverse-Distance Weighting (IDW) and Gaussian-Process Implicit Surface (GPIS) reconstruction, the proposed method reduces reconstruction error by approximately two-thirds (e.g., +66% to +77% error reduction on ropes and cloths at low touch budgets). The cross-attention mechanism proves superior to global-pool set encoders, with the performance gap widening as more touches are observed, highlighting the importance of locality.
• Active Sensing Performance: The learned policy reduces error by roughly 5–7% compared to random touching across various budgets, outperforming a GP uncertainty baseline at sparse budgets. However, the average gain is modest. The ensemble-variance proxy (a heuristic baseline) was found to degrade below random performance as budgets increased, as it concentrated touches on persistent outliers rather than informative locations.
• Impact of Occlusion: The benefit of active sensing is highly dependent on the regime. The error reduction of the learned policy increases from +6.0% on least-occluded configurations to +11.8% on the most-occluded configurations.
• Vision vs. Touch: When a noisy depth camera is added to the system, the advantage of active touch placement vanishes. Random and oracle touch placements yield nearly identical results, indicating that vision and the learned prior already resolve the hidden state, leaving little for additional touches to improve.

Significance and Claims
The paper argues that the primary leverage in tactile deformable estimation lies in the estimator itself rather than the strategy for where to touch. A locality-aware cross-attention model recovers most of the recoverable information from sparse random touches. Active sensing is a "real but bounded add-on" that offers modest improvements specifically under self-occlusion and vanishes once vision is available. The authors conclude that while full-mesh estimates are valuable for downstream planning, the current algorithmic gap lies in closing the divide between deployable active policies and the ideal oracle, particularly under realistic probing conditions. All experiments are simulation-based, utilizing a position-only touch model compatible with standard robotic arms.