Technical Summary: Instant-Fold

Problem Statement

Deformable Object Manipulation (DOM), particularly in tasks like clothing folding, presents significant challenges due to high-dimensional, partially observable states that evolve through long-horizon, topologically complex interactions. Unlike rigid objects, deformable objects admit multiple valid manipulation modes depending on user preference, where differences manifest not only in the final configuration but also in the sequence of intermediate deformations. While language can specify goals, it often fails to precisely and compactly convey the temporal structure, intermediate goals, and subtle spatial conventions required for these tasks. Furthermore, existing representation learning methods struggle to generalize across different manipulation modes because they often interpret varying deformation trajectories as entirely different tasks rather than variations within the same task family. A critical bottleneck is the lack of massive, diverse datasets for DOM comparable to those available for language and vision, making generalization to unseen deformation trajectories difficult.

Methodology: Instant-Fold

The authors propose Instant-Fold, an In-Context Imitation Learning (ICIL) framework designed to infer and execute diverse manipulation modes directly from a single human demonstration without requiring gradient updates at test time. The approach is trained entirely in simulation and transfers zero-shot to the real world. The methodology consists of two primary stages:

1. Deformation-Aware Temporal Contrastive Pretraining

To address the representation challenge, the authors first learn visual representations that are sensitive to topological state changes (e.g., folding, crumpling) while remaining robust to nuisance variations like texture and lighting.

• Representation: Deformable objects are represented as a compact set of geo-semantic tokens, combining 3D positions (derived from RGB-D back-projection and Farthest Point Sampling) with semantic features extracted from a pretrained image encoder (DINOv3).
• Objective: A Temporal Contrastive Pretraining objective is introduced. This self-supervised task leverages dense particle-based simulator geometry to define correspondence targets.
  • Same-Cloth Matching: Uses nearest-particle assignment to ensure tokens corresponding to the same physical cloth location remain close in feature space across frames and trajectories, even under non-rigid deformation.
  • Cross-Cloth Matching: Uses sparse semantic keypoints to establish correspondence across different garment instances, promoting transferability.
  • Loss Function: A soft-weighted supervised contrastive loss is optimized, utilizing dense negatives and Gaussian-weighted positive assignments based on token-to-particle distance.

2. In-Context Policy Learning

The policy is formulated as a conditional trajectory generation problem using a Flow-Matching Transformer.

• Input Context: The model conditions on a single human demonstration, represented as a sequence of "context keyframes." These keyframes are automatically extracted from demonstration trajectories based on end-effector state events (e.g., gripper open/close transitions) to capture temporally distinct deformation phases.
• Architecture:
  • Context Encoder: Maps demonstration keyframes into a structured representation. It employs a shared frozen visual tokenizer and aggregates tokens using a spatial-then-temporal backbone. Key mechanisms include 3D ALiBi (to bias attention toward nearby cloth-robot pairs), learned summary tokens (to distill global intent), and state-event tokens (to preserve sparse grasp/release cues).
  • Flow-Matching Action Decoder: Predicts dual-arm action trajectories (Cartesian positions and gripper openness) by denoising a noisy trajectory. The decoder fuses the current scene with the demonstration context via cross-attention. It is conditioned on the denoising timestep and the current proprioceptive state via AdaLN.
  • Auxiliary Task: An auxiliary keypose prediction branch is included to supervise short-horizon subgoals, aiding long-horizon planning.
• Training: The policy is trained using rectified flow matching on Cartesian positions and binary cross-entropy on gripper openness, entirely within a particle-based simulator (FleX) with extensive domain randomization.

Key Contributions

1. ICIL Formulation for DOM: The paper formulates deformable object manipulation as an In-Context Imitation Learning problem, introducing a flow-matching transformer architecture capable of long-horizon, dual-arm deformable manipulation conditioned on a single demonstration.
2. Temporal Contrastive Pretraining: A novel pretraining method tailored for ICIL that learns topology-consistent, deformation-aware representations. This allows the model to distinguish between different manipulation modes of the same object family rather than treating them as distinct tasks.
3. Zero-Shot Sim-to-Real Transfer: The framework demonstrates successful transfer to real-world dual-arm robots without any real-world data collection, fine-tuning, or additional training.

Experimental Results

The authors evaluate Instant-Fold in simulation and on a real dual-arm robot across various clothing items and folding modes.

• Representation Quality: The temporal contrastive pretraining yields representations that are significantly more temporally consistent and spatially coherent than standard DINOv3 features or previous keypoint-based methods (UniGarmentManip).
• Policy Performance:
  • Demonstration vs. Language: Demonstration-conditioned policies significantly outperform language-conditioned policies in terms of geometric success (C-SR@95) and completion error, even when the language policy is fully pretrained. This highlights the value of demonstrations in conveying geometric structure.
  • Pretraining Impact: Full pretraining on diverse folding modes and random deformations improves generalization to held-out folding modes, with the best model achieving a 58.3% conditional success rate on held-out folds.
  • Ablations: Removing components such as the context encoder, keypose auxiliary loss, or summary tokens leads to substantial performance drops, particularly in generalization to unseen contexts.
  • Scaling: Increasing the diversity of training contexts improves generalization to held-out folds more significantly than it improves performance on seen folds.
• Comparison to Baselines: In simulation, Instant-Fold outperforms prior state-of-the-art methods (ClothFunnels, UniFolding, UniGarmentManip) in both FleX and Isaac Lab environments.
• Real-World Deployment: On a real dual-arm robot, Instant-Fold achieved an average success rate of 60.9% across eight unseen garments (including shirts, shorts, blouses, and jackets) using a single human hand demonstration. This significantly outperforms baselines, which required real-world data collection or fine-tuning and achieved success rates between 9.4% and 29.7%.

Significance and Claims

The paper claims that Instant-Fold establishes a new paradigm for DOM by leveraging In-Context Imitation Learning to capture the diversity of valid manipulation modes. By separating topological reasoning (via contrastive pretraining) from functional reasoning (via flow-matching), the system can infer complex manipulation procedures from a single demonstration. The authors emphasize that their approach eliminates the need for real-world data collection or fine-tuning, addressing a major bottleneck in deploying robot learning for deformable objects. They suggest that ICIL is a promising direction for handling the high-dimensional, continuous state spaces inherent in deformable manipulation, offering a foundation for broader in-context reasoning in this domain.

Limitations

The authors acknowledge several limitations:

• The system assumes reliable object segmentation.
• Experiments start from states within the foldable manifold rather than from highly crumpled configurations.
• The study is currently limited to a single object category (garment tops).
• The context library is manually designed and limited in scale, suggesting a need for future work in automatic context exploration.