TacSE3: Equivariant SE(3) Motion Estimation from Low-Texture Visuotactile Images for In-Gripper Tracking and Compensation

Technical Summary: TacSE3 – Equivariant SE(3) Motion Estimation from Low-Texture Visuotactile Images

Problem Statement
Robotic in-hand manipulation frequently encounters scenarios where visual tracking fails due to occlusion or the lack of salient visual features. While Vision-based Tactile Sensors (VBTS) offer a promising alternative for contact-rich interaction, conventional motion estimation pipelines relying on image texture, marker tracking, or dense optical flow struggle with low-texture or symmetric objects (e.g., polished spheres). In these conditions, subtle rotational motions induce minimal appearance changes, making correspondence-based methods unreliable. Furthermore, single-sensor setups often suffer from ambiguity between translational and rotational motions, as both can produce similar local deformation patterns. The paper addresses the challenge of constructing a reliable SE(3) motion estimate directly from low-texture tactile images without relying on explicit texture matching.

Methodology: TacSE3 Pipeline
The proposed method, TacSE3, reframes the tactile image not as a textured visual input but as a geometric measurement of contact. The pipeline proceeds through the following stages:

1. Decoupled 3D Force Field Construction:
   Instead of tracking image features, the method processes raw tactile observations to reconstruct a dense 3D force field. Using a dense color-pattern VBTS (specifically DM-Tac), the system decomposes the tactile response into tangential (shear) and normal components. This is achieved via Helmholtz–Hodge decomposition of the planar displacement field, separating curl-free (divergence) and divergence-free (curl) components to derive spatial distributions of normal and shear forces.

2. Contact Point and Motion Proxy Generation:
   The system identifies active contact regions based on significant normal response. For each valid pixel, a 3D contact point is constructed in the local tactile frame. The tangential and normal force responses are calibrated to serve as a local velocity proxy, approximating the instantaneous motion of the contact point.

3. SE(3) Rigid-Body Estimation:
   The local velocity proxies are modeled using a rigid-body twist equation ($v = \omega \times q + t$). The estimation process employs a specific decoupling strategy:

   • Translation: Planar translation is derived strictly from the motion of the contact centroid on the sensor plane, rather than being jointly fitted with rotation.
   • Rotation: Rotation is estimated separately by fitting the shear-related motion constraints to the rigid-body model.
     This separation avoids the ill-conditioning often found in coupled translation-rotation least-squares problems and provides a physically interpretable signal.

4. Temporal Integration and Equivariance:
   The estimated instantaneous twist is integrated over time using the exponential map on the SE(3) manifold to produce a continuous pose trajectory. The method ensures equivariant coordinate consistency: the estimated motion is defined in the end-effector frame, ensuring that the same underlying object motion yields consistent estimates regardless of changes in the robot's global pose or the specific location of contact on the sensor.

5. Dual-Sensor Fusion:
   To resolve translation-rotation ambiguity, the system utilizes two tactile sensors mounted on opposing gripper fingers. By applying a sign-matrix transformation to account for the mirrored coordinate frames, the system fuses the twist estimates. This allows the system to distinguish between translational motion (which produces consistent force directions across sensors) and rotational motion (which produces opposing force patterns).

6. Residual Control Integration:
   The estimated SE(3) motion is treated as a tactile geometric residual. It is fused with an existing base policy (e.g., Action Chunking with Transformers) as a lightweight post-processing signal ($u_{final} = u_{policy} + \Delta u_{tactile}$) to refine robot actions without retraining the base policy.

Key Contributions

• Geometric State Formulation: The paper formulates tactile-to-SE(3) estimation for low-texture sensing as a geometric state estimation problem centered on local rigid-body motion, moving away from texture-dependent correspondence matching.
• Force-Field Decoupling: It proposes a pipeline that decouples raw tactile observations into a 3D force field and reconstructs rigid-body twist via a physically grounded SE(3) model, specifically separating centroid-based translation from shear-based rotation.
• Dual-Sensor Ambiguity Reduction: The work demonstrates that dual-sensor configurations significantly reduce the ambiguity between translation and rotation compared to single-sensor setups.
• Lightweight Compensation: The method provides a rotation signal that can be attached as a post-processing module to improve disturbance tolerance in downstream manipulation tasks without modifying the base control policy.

Experimental Results
Experiments were conducted using a UR5 robot equipped with dual DM-Tac sensors:

• Rotation Accuracy: Under controlled rotations of a grasped ball (30°, 60°, 90°) across three axes ($r_x, r_y, r_z$), the method achieved mean angular errors ranging from ~3° for 30° rotations to ~12° for 90° rotations. The error growth is attributed to integration drift and contact nonlinearity over larger angles.
• Single vs. Dual Sensor: In pure translational motion tests, the single-sensor configuration exhibited significant rotation drift (approx. 9–10°), whereas the dual-sensor configuration reduced this drift to approximately 1.1–1.5°, demonstrating effective decoupling.
• Generalization: The method maintained low error rates (2.6°–4.0°) across diverse objects with varying geometries (spheres, cylinders, irregular shapes) and surface textures (smooth, textured, wood, plastic), confirming robustness to texture-poor conditions.
• Policy-Level Performance: In tasks involving human-induced disturbances (Drawing, Gear Insertion, Peg-in-Hole), enabling the tactile post-processor improved success rates under interference compared to the base policy alone (e.g., Gear Insertion improved from 20% to 35% success under interference).

Significance and Claims
The paper claims that TacSE3 offers a practical geometric bridge between low-texture visuotactile sensing and contact-rich in-gripper manipulation. By prioritizing contact geometry over image appearance, the method achieves reliable motion estimation where traditional visual or texture-based tactile methods fail. The authors emphasize that the approach is not intended to replace global vision or provide full object-level state estimation for novel objects over long horizons. Instead, it serves as a robust, interpretable, and lightweight module for local motion tracking and compensation, particularly valuable for smooth, symmetric, or occluded objects where vision is unreliable. The equivariant nature of the output ensures the signal remains consistent for downstream control regardless of the specific contact location or robot pose.