Technical Summary: SynthICL

Problem Statement

In-context imitation learning (ICIL) aims to enable robots to learn new tasks from a small number of demonstrations at test time without retraining the policy weights. While promising, training generalizable and scalable ICIL policies remains a significant challenge. Existing approaches face two primary limitations:

1. Data Scarcity and Cost: Methods relying on large-scale real-world data collection are expensive and difficult to scale.
2. Sim-to-Real Gaps in Synthetic Data: Prior synthetic approaches (e.g., Instant Policy) often rely on point-cloud observations. This introduces a substantial sim-to-real gap because real-world point clouds are noisy and incomplete, whereas simulated ones are idealized. Furthermore, point-cloud representations discard rich RGB appearance information, limiting the policy's ability to distinguish objects with similar geometry but different textures.

The central question addressed by this work is whether RGB-only synthetic training can directly produce generalizable in-context policies that transfer effectively to the real world without depth sensing or real-world co-training.

Methodology: SynthICL

SynthICL is a framework that trains ICIL policies entirely from high-fidelity synthetic RGB data. The method consists of three core components:

1. Data Generation Pipeline

The authors implement a pipeline in Isaac Sim to generate "pseudo-demonstrations" (context-task trajectory pairs) without physics simulation, following the concept of Instant Policy but with critical improvements:

• High-Fidelity Rendering: Unlike prior work using PyBullet, Isaac Sim is used to render high-quality RGB images, reducing the visual sim-to-real gap.
• Valid Grasp Sampling: Instead of randomizing grasp poses (which often yields invalid or unnatural configurations), the pipeline pre-computes valid grasp poses using GraspNet for objects from RoboTwin-OD and ShapeNet. Waypoints involving grasping randomly sample from these valid poses.
• Data Scale: The pipeline generates a dataset of 3 million synthetic trajectories, organized into aligned context-task pairs.

2. Model Architecture

The policy is a flow-matching transformer designed to predict action chunks conditioned on visual context.

• Context Encoder: Encodes the demonstration trajectory (multi-view segmented RGB images) using a frozen DINOv3 vision encoder. It employs a decomposed spatial-temporal transformer with separate spatial and temporal attention blocks to efficiently process multi-view, multi-frame data.
• State-Context Encoder: Uses cross-attention to retrieve features from the encoded context that are relevant to the current observation (also encoded via DINOv3).
• Action Decoder: Formulates action prediction as a conditional flow-matching problem. It predicts a velocity field to map noise to the target action chunk, trained to minimize the difference between the predicted flow and the ground-truth flow.

3. Auxiliary Subgoal Prediction

A novel contribution is the introduction of an auxiliary subgoal image prediction head.

• Mechanism: During training, the model predicts the multi-view RGB image of the next subgoal (the next waypoint in the trajectory) alongside the action.
• Objective: This is trained via an MSE reconstruction loss.
• Design: The subgoal head is architecturally separate from the action decoder. It is active during training to encourage the model to learn visually grounded subgoal representations and reason about task progress, but it is disabled at inference time, preserving a simple end-to-end policy.

Key Contributions

1. SynthICL Framework: A scalable framework for training generalizable, RGB-based ICIL policies entirely from high-fidelity synthetic data, eliminating the need for depth sensors, precise camera calibration, and real-world training data.
2. In-Context Subgoal Prediction: A novel training objective that encourages visually grounded reasoning. The model learns to predict intermediate goal images, which significantly improves policy robustness and performance without complicating the inference-time architecture.
3. Empirical Validation: Extensive experiments demonstrating that the method outperforms prior ICIL methods on diverse manipulation tasks in both simulation and the real world.

Experimental Results

Simulation (RLBench)

Evaluated on 12 unseen manipulation tasks with 100 rollouts each:

• SynthICL achieved an average success rate of 75.0%.
• It outperformed Instant Policy (IP) (72.4%), particularly on tasks requiring appearance-based grounding (e.g., distinguishing cups of different colors with identical geometry).
• It outperformed ICRT (60.1%), an autoregressive RGB-based method.
• The variant without Subgoal Prediction (w/o SP) achieved 72.2%, indicating the auxiliary objective provides a measurable boost, especially on precision tasks.

Real-World Evaluation

Evaluated on 16 unseen real-world tasks using a Franka Research 3 robot with 3 RGB cameras:

• SynthICL achieved an average success rate of 79.1% with only one demonstration provided at test time.
• It significantly outperformed IP (72.5%) and ICRT (45.0%).
• The performance gap between the full model and the variant without subgoal prediction was larger in the real world (13.8% difference) than in simulation (2.8%), suggesting the subgoal objective is critical for real-world robustness.
• The method successfully handled tasks requiring texture discrimination and spatial reasoning without depth data.

Ablation and Scaling Studies

• Data Generation: Using Isaac Sim rendering and GraspNet sampling was critical. Pipelines without these modifications achieved success rates of only ~9-11%.
• Data Scaling: Performance improved as the dataset size increased from 3K to 100K trajectories. Interestingly, the subgoal prediction objective only provided benefits when sufficient data (larger than small datasets) was available; on small datasets, the auxiliary loss competed with action learning.
• Architecture: The decomposed spatial-temporal attention mechanism outperformed standard full attention by 11.6%.

Significance and Claims

The paper claims that large-scale RGB-only synthetic training is a viable path toward scalable in-context imitation learning. SynthICL demonstrates that:

1. High-fidelity synthetic data can bridge the sim-to-real gap effectively for ICIL, removing the dependency on costly real-world data collection.
2. RGB observations are superior to point clouds for tasks requiring texture and appearance discrimination.
3. Auxiliary subgoal prediction acts as a powerful regularizer that improves the visual grounding of the policy, leading to better generalization in real-world scenarios.

The authors acknowledge limitations, noting the current reliance on segmented RGB images and difficulties with tasks requiring very high precision or complex physical contacts (e.g., tight insertions). They suggest future work could address these by incorporating more diverse background randomization and physics-based simulation.