Technical Summary: A Dataset and Benchmark for Robotic Cloth Unfolding Grasp Selection

Problem Statement

Robotic cloth manipulation, specifically the task of unfolding garments, lacks standardized benchmarks and shared datasets necessary for objective evaluation and comparison of different approaches. While significant progress has been made in rigid object grasping, the deformable nature of cloth introduces unique challenges regarding reproducibility and state estimation. Existing literature often relies on synthetic data or lacks real-world validation, and prior studies frequently report inflated performance metrics due to controlled laboratory settings, multiple recovery attempts, or filtering of failures. There is a critical need for an independent, out-of-the-lab evaluation framework that focuses on the core challenge of selecting appropriate grasp poses for in-air cloth unfolding.

Methodology

The paper introduces the ICRA 2024 Cloth Competition, a head-to-head evaluation designed to address these gaps. The methodology is structured around a standardized benchmark, a large-scale dataset, and a live competition protocol.

1. Benchmark Protocol

The benchmark evaluates algorithms for in-air robotic cloth unfolding. The procedure is streamlined to isolate the grasp selection problem, removing the need for participants to develop full manipulation pipelines (e.g., camera calibration, motion planning). The autonomous procedure consists of:

1. Initialization: The robot autonomously grasps the highest point of a crumpled garment on a table and lifts it.
2. Secondary Grasp: The robot grasps the lowest point of the now-hanging garment.
3. Observation: An RGB-D image of the hanging cloth is captured.
4. Grasp Selection: The participant's algorithm selects a grasp pose for the right arm based on the observation.
5. Execution & Stretching: The robot executes the grasp and stretches the cloth until a tension of 2 N is achieved.
6. Evaluation: A final RGB-D image is captured to measure the degree of unfolding.

2. Hardware and Environment

The competition utilized a dual-arm setup with two UR5e collaborative robots spaced 90 cm apart, equipped with Robotiq 2F-85 grippers. Perception was provided by a ZED 2i RGB-D camera mounted 80 cm above the robots, simulating a head-mounted, ego-centric perspective. The environment was designed to be reproducible, with randomized initial garment configurations to test generalization.

3. Evaluation Metrics

Performance was quantified using Coverage, defined as the ratio of the garment's current projected surface area to its maximum potential surface area (determined by a reference stretch under ideal conditions). Additional metrics included:

• Grasp Success Rate: Whether the second grasp successfully held the cloth.
• Coverage for Successful Grasps: Average coverage considering only trials where the grasp succeeded.
• Execution Time: A recommended maximum of 30 seconds for the grasp selection algorithm.

4. Dataset Construction

The authors compiled a dataset of 679 real-world robotic cloth unfolding attempts across 34 diverse garments (T-shirts and towels). The dataset comprises:

• Training Subset (503 episodes): Collected with human-annotated grasp poses to bootstrap the competition. Annotators used diverse strategies, including semantic keypoint targeting and risk-reward trade-offs.
• Competition Subset (176 episodes): Recorded from the live trials of 11 participating teams, capturing diverse algorithmic strategies and failure modes.
• Data Format: Each episode includes stereo RGB images, depth maps, confidence maps, colored point clouds, robot joint states, and gripper poses, stored in standard formats (PNG, TIFF, PLY, JSON).

Key Contributions

The paper presents three primary contributions:

1. A Novel Benchmark: A standardized, objective benchmark for evaluating grasp selection strategies for in-air robotic cloth unfolding, prioritizing generalization and real-world constraints.
2. The ICRA 2024 Cloth Competition: A unique competition involving 11 diverse teams, providing a rigorous, live evaluation of state-of-the-art methods.
3. A Large-Scale Dataset: A publicly available dataset of 679 unfolding demonstrations, representing one of the largest and most diverse collections of real-world cloth manipulation data, including both human-annotated and algorithm-generated trials.

Results

Eleven teams participated, employing a mix of traditional geometric methods and learning-based techniques (including graspability estimation, semantic keypoint detection, grasp imitation, and coverage prediction).

• Performance: The winning team (AIR-jnu) achieved an average coverage of 0.60 with a grasp success rate of 0.69. The second-place team (Team Ljubljana) achieved 0.57 coverage.
• Method Comparison: Surprisingly, the top-performing methods were traditional, hand-engineered geometric approaches (Intuitive Grasping Determination and Sharp Edge Detection), outperforming many learning-based methods. These methods excelled in both grasp success rates and the coverage of successful grasps.
• Generalization: No significant difference in average coverage was observed between garments seen during training and unseen garments, suggesting that most methods generalized well to new items.
• Failure Analysis: Shirts had a higher grasp success rate (0.70) than towels (0.53), likely due to lower bending stiffness. However, successful grasps on both garment types yielded similar coverage (~0.60). The analysis highlighted a trade-off between grasp success and coverage; some methods prioritized high coverage at the cost of lower success rates.

Significance and Claims

The paper claims that the competition and dataset provide a foundational resource for the robotic cloth manipulation community. Key significances include:

• Real-World Validation: The results reveal a significant discrepancy between the competition's average coverage (~0.60) and values reported in prior work (often >0.8). The authors attribute this to the rigorous, single-attempt, out-of-the-lab nature of the competition, which minimizes biases from multiple recovery attempts or controlled setups. This underscores the necessity of independent, real-world evaluation.
• Re-evaluation of Approaches: The strong performance of traditional geometric methods challenges the assumption that learning-based approaches are inherently superior for this task, suggesting that semantic understanding and robust geometric heuristics remain highly effective.
• Data-Driven Progress: The dataset serves as a critical resource for developing and evaluating grasp selection methods, particularly for learning-based approaches that require high-quality, real-world data to overcome the "sim-to-real" gap.
• Future Benchmarking: The authors hope this work stimulates the creation of further benchmarks and datasets, fostering a cycle of mutual improvement between competitions and data collection to drive progress in data-driven robotic cloth manipulation.

The paper concludes that while robotic cloth unfolding remains an open challenge, the established benchmark and dataset provide a necessary framework for measuring true progress and understanding the trade-offs between grasp reliability and task effectiveness.