RoboHitch: Learning Visual Affordance from Disordered Keypoints for Hitch Knots Tying

Technical Summary: RoboHitch – Learning Visual Affordance from Disordered Keypoints for Hitch Knots Tying

Problem Statement
Robotic manipulation of Deformable Linear Objects (DLOs), specifically for tying hitch knots, faces significant hurdles due to complex dynamics and frequent self-occlusions. Existing methods typically rely on precise topological state tracking using ordered keypoints and explicit edge connectivity. This reliance creates fragility; during knot formation, repeated bending and crossings cause tracking drift and topology mismatches, leading to task failure. Furthermore, model-based approaches require explicit state estimation (often via markers) which is difficult to guarantee, while end-to-end learning-based methods often demand vast amounts of real-world data and lack interpretability. A core challenge is the "Point Set Permutation" problem, where identical rope states may be represented by different keypoint sequences due to a lack of consistent correspondence, and the "Topological Ambiguity" problem, where distinct configurations map to identical point sets without explicit connectivity.

Methodology
The authors propose RoboHitch, a framework that learns hitch knot tying policies from human demonstrations using only disordered 3D keypoints and RGB images, eliminating the need for explicit topological ordering. The system operates as a sequential decision-making problem where the policy predicts a 3-tuple action: a grasp index, an in-plane rotation, and a placement location.

The framework consists of three primary modules:

1. Rope State Estimation: Instead of tracking an ordered sequence, the system extracts a disordered set of rope keypoints ($x_{pt}$) from RGB-D point clouds using DBSCAN for spatial clustering and K-means for centroid generation. Local grasping orientations are determined via Singular Value Decomposition (SVD) on local neighborhoods to maximize gripper contact stability.
2. Multimodal Feature Extraction:
   • Dynamic Graph Autoencoder (GAE): A permutation-equivariant GAE processes the disordered keypoints to extract geometric features ($f_{pt}$). By using symmetric message passing and dynamic graph construction based on Euclidean distance, it encodes topological features regardless of input ordering.
   • Convolutional Autoencoder (CAE): A CAE processes the RGB scene image ($x_{rgb}$) to extract visual context features ($f_{rgb}$), capturing global configuration and spatial relationships.
3. Attention-Guided Fusion and Action Prediction:
   • Bidirectional Cross-Attention: To resolve topological ambiguity, the framework employs a bidirectional cross-attention mechanism. This fuses geometric and visual modalities: keypoint features attend to image regions to enrich grasp prediction with visual context, while image features attend to keypoints to incorporate geometric constraints for placement prediction.
   • Affordance Heads: The fused features are decoded hierarchically. A Pick Head classifies the grasp point, a Rotation Head predicts the gripper orientation, and a Place Head generates a pixel-wise placement heatmap. This factorized approach ($P(it, \theta_t, p_t) = P(it)P(\theta_t | it)P(p_t | it, \theta_t)$) simplifies learning and handles action dependencies.

Key Contributions

• Disordered State Representation: The paper presents a method to extract and utilize disordered 3D keypoints from potentially occluded ropes, enabling downstream manipulation without fragile tracking of precise topological order.
• Multimodal Fusion Framework: The introduction of a knotting framework that leverages bidirectional cross-attention to fuse rope geometric features and visual scene features. This facilitates the prediction of pick and place affordances in visually complex environments where explicit topology is unavailable.
• Real-World Validation: The framework is experimentally validated on a single-arm robotic system, demonstrating effective performance in tying hitch knots under self-occlusion conditions.

Results
In real-world experiments, the trained policy achieved an 84% success rate (42/50 trials) in tying hitch knots, outperforming existing model-based (50%) and learning-based (38–66%) single-arm methods.

• Generalization: The method generalized well to unseen backgrounds (4/5 success) but showed reduced performance with larger rope diameters (3/5 success) and failed entirely with polypropylene rope (0/5 success). The authors attribute failures with polypropylene to its high stiffness and elastic restoration, which prevents the rope from maintaining the deformed configurations required for knotting.
• Ablation Studies: Experiments confirmed that geometric keypoints (PT) encode task-relevant features more effectively than visual data alone for grasping, while RGB data is superior for placement. Crucially, the bidirectional cross-attention fusion (Case D) significantly outperformed direct feature concatenation (Case C) and single-modality inputs, proving that effective multimodal fusion is critical for resolving topological ambiguities and achieving task success.

Significance and Claims
The paper claims that RoboHitch bridges the gap between model-based and learning-based approaches by utilizing a learning-based framework trained on human demonstrations but structured around an interpretable, multimodal state representation. By removing the dependency on perfectly tracked topological sequences, the method offers a more robust solution for robotic knot tying in scenarios involving self-occlusion. The authors modestly note that while the approach is effective for flexible materials like nylon, it remains challenged by highly stiff materials and the physical limitations of single-arm manipulation, suggesting future work should focus on dual-arm coordination and robust perception under complex conditions.