Close-Fitting Dressing Assistance Based on State Estimation of Feet and Garments with Semantic-based Visual Attention

Technical Summary: Close-Fitting Dressing Assistance Based on State Estimation of Feet and Garments with Semantic-based Visual Attention

Problem Statement
The paper addresses the challenge of autonomous dressing assistance for close-fitting garments, specifically socks, in the context of an aging population and a shortage of caregivers. While previous research has focused on loose-fitting garments (e.g., shirts, trousers) that allow for significant misalignment, dressing tight garments requires fine force adjustments to manage friction and snagging against the skin while accounting for the garment's deformation. A major hurdle is the variability in human physique (foot size, shape, skin tone, flexibility) and the difficulty of simulating accurate skin-garment friction for reinforcement learning. Existing methods often rely heavily on RGB data, limiting generalizability to unseen backgrounds or objects, and struggle to balance the elastic nature of the fabric with the directional control of pulling forces.

Methodology
The authors propose a multimodal imitation learning framework that integrates semantic object understanding with spatial depth estimation and somatosensory feedback. The system is designed to operate without relying solely on RGB values, thereby improving robustness to background and appearance variations.

1. Semantic and Spatial Perception (Ventral-Dorsal Model):

   • Inspired by the human visual system's dual pathways, the model combines SAM2 (Segment Anything Model 2) for semantic segmentation (ventral stream) and Depth Anything Model (DAM) for depth estimation (dorsal stream).
   • SAM2 generates binary masks for socks and feet, mitigating the influence of color variations and occlusions.
   • DAM infers depth from monocular RGB images. The system fuses these by embedding depth values into the masked regions, creating a "masked depth map" that provides 3D spatial relationships (x, y, z) between the sock and the foot.

2. Attention Mechanisms:

   • Somatosensory Attention: A Selective Kernel Network (SKNet) is employed to dynamically select optimal feature extraction kernels (3×3 and 5×5) based on input scale. This captures local and global changes in tactile feedback, crucial for managing the elasticity and friction of the sock-foot interface.
   • Visual Attention: A Spatial Attention Recurrent Neural Network (SARNN) processes the semantic masks and depth maps. It applies a spatial softmax to generate attention maps, calculating expected 3D keypoints. This allows the system to focus on geometric context rather than just 2D salient regions, aiding in robust localization during occlusion.

3. Control Policy (Hierarchical LSTM):

   • The core policy is a Hierarchical LSTM (specifically EIPL with hierarchical LSTM). It processes six extracted 3D keypoints, joint angles, joint torques, and thumb tactile information.
   • The architecture uses a "union LSTM" to integrate global context from all modalities, which then provides feedback to modality-specific "bottom LSTMs." This structure models both intra-modal and inter-modal dependencies.
   • The network predicts the next state of the sensory-motor sequence, outputting the next joint angles, torques, and a reconstructed image for the next time step.

4. Training and Data Collection:

   • To ensure safety and avoid burdening human subjects during data collection, the model was trained using a mannequin.
   • Data was collected via teleoperation using a PS controller, capturing 12 samples across three foot angles (30°, 40°, 50°).
   • The loss function combines Mean Squared Error (MSE) for joint angles, torques, tactile values, predicted images, and attention points.

Key Contributions

• Multimodal Imitation Learning for Close-Fitting Garments: The paper establishes a method for manipulating close-fitting garments that adapts to individual human differences through proper force interaction, moving beyond the limitations of loose-fitting garment studies.
• Semantic-Based Generalization: By utilizing semantic masks and depth estimation rather than raw RGB data, the model demonstrates the ability to generalize to unseen human feet and background environments.
• Safety-Oriented Training and Evaluation: The study validates a pipeline where a model trained on a mannequin is successfully transferred to real human subjects, addressing the safety and ethical challenges of collecting dressing data from humans.

Experimental Results
The system was evaluated on a humanoid robot (AIREC) with 10 human participants (foot sizes 23.0–26.5 cm) and compared against Action Chunking with Transformer (ACT) and Diffusion Policy (DP).

• Ablation Study: Removing the Hierarchical LSTM or the combination of SAM2 and DAM caused a dramatic drop in success rates (from 100% to 5% and 0%, respectively). Removing DAM or SKNet resulted in moderate degradation, highlighting the critical role of depth and tactile attention.
• Generalization Performance:
  • The proposed model achieved a 84% success rate (42/50) on known backgrounds and 74% (37/50) on unknown backgrounds.
  • In contrast, ACT achieved 66% on known backgrounds and 0% on unknown backgrounds. DP failed to complete the task entirely.
• Force Adaptation: The proposed model maintained consistent tactile forces across different foot sizes (approx. 19N at the toe, 21N at the heel), whereas ACT showed high sensitivity to size variations and inconsistent force application.
• Statistical Significance: A Wilcoxon signed-rank test confirmed a statistically significant difference (p < 0.01) and a large effect size between the proposed model and ACT.

Significance and Claims
The paper claims that the proposed model successfully estimates the state of both the garment and the foot, enabling precise dressing assistance for close-fitting garments. The integration of semantic understanding with depth and somatosensory attention allows the robot to adapt to individual differences in foot size, shape, and flexibility without requiring retraining on human data. The authors emphasize that this approach overcomes the limitations of purely RGB-based visual policies and simulation-based reinforcement learning, which struggle with the complex dynamics of fabric elasticity and skin friction.

Limitations and Future Work
The authors modestly note that while the system handles the insertion and subsequent dressing phases well, the precision of the initial toe-insertion step remains limited and was not rigorously evaluated. Failures in human trials were attributed to morphological variability (e.g., limited ankle flexibility, toenail snagging) and premature arm lifting for larger feet. Future work aims to expand the method to handle dynamic adaptation to unexpected foot movements and to improve online motion replanning based on unintended contact detection.