Vi-TacMan: Articulated Object Manipulation via Vision and Touch

1. Problem Statement

Autonomous manipulation of articulated objects (e.g., cabinets, ovens, doors) in unstructured human environments is a fundamental challenge for robotics.

• The Core Difficulty: Everyday objects exhibit vast variability in appearance, geometry, and kinematics, making precise a priori modeling impractical.
• Limitations of Existing Approaches:
  • Vision-only methods: Can infer hidden kinematics globally but often yield imprecise estimates on unfamiliar objects, leading to execution failures.
  • Tactile-only methods: Achieve robust control through contact feedback but require accurate initialization (a stable grasp and a coarse motion direction) to function.
• The Gap: There is no systematic framework that exploits the synergy between vision (global guidance) and touch (local precision) to enable generalized manipulation without explicit kinematic models.

2. Methodology: The Vi-TacMan Framework

Vi-TacMan proposes a hierarchical framework where vision provides coarse guidance (grasp proposals and interaction directions) to seed a tactile controller that handles precise execution via real-time contact regulation.

A. System Overview

The system operates in two stages:

1. Vision-Based Initialization: The robot uses RGB-D data to detect movable/holdable parts, propose a grasp, and estimate a coarse interaction direction.
2. Tactile Execution: A contact-regulating controller takes the initial state and iteratively adjusts the end-effector pose to maintain stable contact, effectively "feeling" its way through the manipulation without needing a pre-defined kinematic model.

B. Key Technical Components

1. Problem Formulation (MAP Estimation)
The goal is to estimate a grasp $G$ and an interaction direction $d$ given visual observations $V$.

• The authors model the interaction direction as dependent on the grasp point.
• They decompose the problem into estimating a rigid transformation $T$ (independent of the specific grasp point) and mapping it to a direction $d$ on the unit sphere $S^2$.

2. Vision-Based Direction Estimation

• Geometric Priors (Surface Normals): Instead of relying solely on point displacement, the system incorporates surface normals as geometric priors to constrain direction estimation.
• Displacement Flow: The system uses a PointNet++-based network to estimate displacement vectors of points on movable parts between frames.
• Uncertainty Modeling (vMF Distribution): Recognizing that multiple plausible directions may exist for unseen objects, the system models the distribution of interaction directions using von Mises-Fisher (vMF) distributions on the unit sphere. This allows for principled inference under ambiguity.
  • The mean direction $\mu$ is estimated as the Fréchet mean of sampled directions.
  • The concentration parameter $\kappa$ captures the uncertainty.

3. Tactile Control Policy

• Based on prior work (Zhao et al.), the controller uses a GelSight-style tactile sensor.
• It tracks the deformation of markers on the sensor surface to detect contact geometry.
• The controller iteratively updates the end-effector pose ($T_{\Delta}$) to minimize the difference between the current contact state and a reference stable state, maintaining contact stability without tracking kinematic models.

4. Implementation Details

• Detection: A DINOv3-based detector identifies movable and holdable parts, achieving 0.86 mAP.
• Grasp Selection: A sampling-based method selects a collision-free grasp on the "holdable" region (centroid of the mask).
• Dataset: Trained on 385 articulated objects from PartNet-Mobility (SAPIEN simulator) with 39k+ training samples. Real-world validation uses a Kinova Gen3 arm with custom GelSight sensors.

3. Key Contributions

1. Vi-TacMan Framework: A novel vision-touch paradigm where coarse visual cues activate precise tactile control, eliminating the need for explicit kinematic models.
2. Robust Detection Model: A detector achieving 0.86 mAP for identifying movable and holdable parts in complex, multi-component objects.
3. Geometric Priors & Uncertainty Modeling:
   • Integration of surface normals as geometric priors, significantly improving direction estimation.
   • Application of vMF distributions to model directional uncertainty on the unit sphere, enabling robust handling of ambiguous objects.
4. Empirical Validation: Extensive testing on 50,000+ simulated samples and diverse real-world objects, demonstrating reliable cross-category generalization.

4. Experimental Results

• Simulation Studies:
  • Tested on 5,836 samples from unseen object categories (e.g., dishwashers, doors, tables).
  • Performance: Vi-TacMan achieved a mean angle error of 8.13°, significantly outperforming baselines:
    • FlowBot3D: 13.92°
    • Normal-only (learning-free): 12.66°
    • Ours w/o normal: 10.10°
  • Statistical Significance: Improvements over all baselines were statistically significant ($p < 0.0001$).
• Real-World Experiments:
  • Successfully manipulated real-world objects (ovens, cabinets, etc.) using a Kinova Gen3 arm and GelSight sensors.
  • The system established stable contact and completed manipulation tasks by following the estimated direction and refining via tactile feedback.
  • Demonstrated successful sim-to-real transfer without explicit kinematic models.

5. Significance and Future Work

• Paradigm Shift: The work establishes that coarse visual cues are sufficient for reliable manipulation when coupled with tactile feedback. This reduces the reliance on perfect kinematic reconstruction, which is brittle in unstructured environments.
• Interpretability: The hierarchical design separates "intention" (visual direction/grasp) from "execution" (tactile control), offering a communicable intermediate representation that could improve human-robot interaction and safety certification.
• Scalability: The approach is scalable to diverse, unseen objects without requiring retraining for specific kinematic structures.
• Future Directions:
  • Extending to non-rigid objects (e.g., flexible handles).
  • Handling multi-modal interaction directions (e.g., levers that move up or down).
  • Integrating slip detection and dynamic re-grasping to handle grasp failures.

In summary, Vi-TacMan demonstrates that combining the global context of vision with the local precision of touch creates a robust, generalizable solution for articulated object manipulation, overcoming the limitations of purely vision-based or purely tactile approaches.