RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning

RoboCurate is a novel synthetic data generation framework that improves robot learning by filtering action-annotated videos through simulation-based consistency checks and enhancing visual diversity via image editing, resulting in significant success rate improvements across various robotic manipulation tasks compared to using real data alone.

The Gist

Imagine you want to teach a robot how to cook, fix a car, or tidy a room. The best way to teach a robot is usually to show it thousands of videos of humans doing those tasks perfectly. But here's the problem: filming thousands of hours of real robots is incredibly expensive, slow, and difficult.

So, scientists started using AI to generate fake videos of robots doing these tasks. It's like using a movie director (an AI video generator) to create infinite training clips without needing a real robot on set.

However, there's a catch. These AI-generated movies can be "Hollywood magic" but "physics nightmares." The robot in the video might look like it's grabbing a cup, but in reality, its hand would pass right through the cup like a ghost, or the cup would float away. If you teach a real robot using these fake videos, it will learn bad habits and fail in the real world.

RoboCurate is a new system designed to fix this. Think of it as a super-intelligent film critic and a physics simulator rolled into one.

Here is how it works, broken down into simple steps:

1. The "Movie Director" (Generating Diversity)

First, the system needs lots of different training scenes. Imagine you have one photo of a kitchen table with a red apple.

• The Old Way: Just make a video of that exact scene.
• The RoboCurate Way: It uses a "magic editing tool" to instantly change the scene. It can swap the red apple for a blue mug, change the table from wood to glass, swap the kitchen for a lab, or change the lighting from sunny to dim.
• Why? This creates thousands of unique "movies" so the robot learns to handle any situation, not just the one specific scene it was shown.

2. The "Action Script" (Predicting Moves)

Once the AI generates a video of a robot moving, it needs to know what the robot is actually doing. It uses a special AI (called an Inverse Dynamics Model) to look at the video and guess the robot's movements.

• The Problem: Since the video was made by a "dream machine," the guessed movements might be wrong. The video might show the robot lifting a heavy box, but the AI might guess the robot is just waving its hand.

3. The "Physics Rehearsal" (The Core Innovation)

This is the most important part. Before the robot ever sees the fake video, RoboCurate runs a simulation.

• Imagine you have a video game engine (a simulator) that follows the laws of physics perfectly.
• RoboCurate takes the "guessed moves" from the fake video and runs them in this perfect physics simulator.
• The simulator tries to play out the scene. If the robot in the simulator tries to grab a cup, the simulator checks: Does the hand actually hit the cup? Does the cup fall over? Or does the hand pass through it like a ghost?

4. The "Double-Check" (Filtering)

Now, RoboCurate compares two things:

1. The Fake Video (the dream).
2. The Simulator Video (the physics reality).

It asks a smart question: "Do these two videos move in the same way?"

• If they match: Great! The fake video is physically accurate. The robot's movements make sense. Keep this data.
• If they don't match: The fake video is a "bad movie." The robot might be floating or clipping through objects. Throw this data away.

5. The Result: A Super-Teacher

By using this process, RoboCurate filters out the "bad movies" and keeps only the "blockbuster hits" where the physics are real.

Why does this matter?
The paper tested this on real robots (including a human-like robot named ALLEX).

• Robots trained only on real data (which is rare and expensive) struggled.
• Robots trained on unfiltered fake data got confused and failed.
• Robots trained on RoboCurate's filtered data became much smarter. They improved their success rates by huge margins (sometimes nearly doubling or tripling their performance) because they learned from high-quality, physically accurate examples.

The Analogy in a Nutshell

Imagine you are learning to play tennis.

• Real Data: Watching a pro player practice (expensive, limited).
• Old Synthetic Data: Watching a cartoon where the ball sometimes flies backward or the racket turns into a snake. You learn nothing useful.
• RoboCurate: You watch a cartoon, but a physics professor watches it with you. If the cartoon shows the ball doing something impossible, the professor says, "No, that's wrong, delete it." If the cartoon looks realistic, the professor says, "Yes, that's a good example, keep it!"

RoboCurate essentially builds a massive, high-quality library of "physics-compliant" training videos, allowing robots to learn faster, safer, and more effectively than ever before.

Technical Summary

1. Problem Statement

Robot foundation models (RFMs) require massive, diverse datasets to generalize across tasks and embodiments. However, collecting real-world robot data is costly and labor-intensive. While neural trajectories (synthetic data generated by video generative models annotated via Inverse Dynamics Models, or IDMs) offer a scalable alternative, they suffer from two critical limitations:

1. Action Quality Issues: Video generative models often produce physically implausible videos (e.g., object deformation, overlapping) or fail to follow instructions. When IDMs annotate these flawed videos, the resulting action labels are incorrect, degrading policy performance.
2. Limited Diversity: Existing synthetic data pipelines often rely on a fixed set of initial scenes, leading to a lack of visual and task diversity compared to real-world variations.
3. Inadequate Filtering: Current methods using Vision-Language Models (VLMs) to filter data often rely on coarse judgments of "physical plausibility" or instruction following. They fail to directly verify if the predicted actions are consistent with the visual motion in the video.

2. Methodology: RoboCurate

RoboCurate is a unified framework designed to generate high-quality, diverse synthetic robot data and rigorously filter it. It consists of three main stages:

A. Diversity-Augmented Generation (Stage 1)

To overcome the lack of diversity in synthetic data, RoboCurate employs a controllable visual diversification pipeline:

• Image-to-Image (I2I) Editing: The initial frame of a scene is edited using diffusion models to vary the table appearance, target object identity/appearance, lighting, and background. Crucially, Canny edge maps are used as conditions to preserve the original scene structure and object placement, ensuring the edited image remains a valid starting state for manipulation.
• Video-to-Video (V2V) Transfer: Successful synthetic videos undergo V2V transfer to alter appearance (texture, color) while strictly preserving motion dynamics. This allows the reuse of existing IDM-predicted action labels for the augmented videos.
• Task Instruction Expansion: A VLM generates diverse, plausible task instructions conditioned on the initial scene, covering variations in behavior, target objects, placement, and hand types.

B. Action-Verified Filtering (Stage 2)

This is the core innovation of RoboCurate. Instead of relying on VLMs to judge video quality, RoboCurate verifies the consistency between the generated video and the predicted actions using a simulator.

1. Simulator Rollout: The IDM-predicted action sequence ($a_{IDM}$) is executed in a physics simulator to generate a "rollout video" ($w_{sim}$). Since the simulator enforces physical laws, $w_{sim}$ is guaranteed to be consistent with $a_{IDM}$.
2. Motion Consistency Check: The system compares the generated synthetic video ($w_{gen}$) with the simulator rollout ($w_{sim}$).
3. Attentive Probe: A lightweight classifier (an attentive probe) is trained on top of a frozen pre-trained video encoder (V-JEPA2). It takes pairs of videos as input and predicts whether they exhibit consistent motion patterns and robot geometry.
   • Training: The probe is trained on real-world data pairs (positive pairs: aligned video and simulator rollout; negative pairs: temporally shifted or cross-episode mismatches).
   • Inference: If the probability of motion consistency between $w_{gen}$ and $w_{sim}$ exceeds a threshold, the sample is retained; otherwise, it is discarded.

C. Best-of-N Sampling (Optional)

During the generation phase, RoboCurate can act as a critic. It samples $N$ candidate video-action pairs and selects the one with the highest motion-consistency score, effectively improving the quality of the generated data without discarding the entire batch.

3. Key Contributions

1. Action-Verified Filtering: A novel method that uses simulator rollouts to verify the physical consistency of IDM-predicted actions, moving beyond superficial VLM-based video plausibility checks.
2. Controllable Diversity Pipeline: A systematic approach using I2I and V2V diffusion models to expand scene and appearance diversity while preserving physical constraints and action labels.
3. Unified Framework: Integration of generation, diversification, and rigorous filtering into a single pipeline that produces "curated" neural trajectories.
4. Automatic Supervision: The construction of positive/negative training pairs for the motion-consistency probe using real-world data and simulator rollouts, avoiding the need for noisy human labels.

4. Experimental Results

The authors evaluated RoboCurate using GR00T N1.5 as the base Vision-Language-Action (VLA) model across pre-training and co-finetuning scenarios.

• Pre-training Benchmarks (GR-1 Tabletop & DexMimicGen):

  • On GR-1 Tabletop (300 demos), RoboCurate achieved a +70.1% relative improvement in success rate compared to a real-data-only baseline.
  • On DexMimicGen, it achieved a +16.1% relative improvement.
  • Comparison: A baseline synthetic pipeline (DreamGen) without RoboCurate's filtering/diversity only achieved +26.6% and +4.0% respectively, highlighting the value of the proposed method.

• Real-World Co-finetuning (ALLEX Humanoid):

  • In a challenging real-world setting with a dexterous humanoid robot (ALLEX), RoboCurate yielded a +179.9% relative improvement in success rate over the baseline.
  • Out-of-Distribution (OOD) Generalization: RoboCurate enabled emergent success on novel object pick-and-place tasks (improving from 0.0% to 25.0%) and novel behavior tasks (e.g., pouring a can) where no real data was available for those specific tasks.

• Ablation Studies:

  • Filtering: Action-level filtering outperformed VLM-based physical plausibility benchmarks (DreamGenBench, VideoCon-Physics).
  • Diversity: Performance increased monotonically with higher task and visual diversity.
  • Probe Training: The attentive probe trained on automatically constructed pairs outperformed simple cosine similarity filters and even human-labeled filters (which suffered from noise).

5. Significance

RoboCurate addresses the "garbage in, garbage out" problem in synthetic robot learning. By verifying that the actions generated by IDMs are physically consistent with the visuals via simulator replay, it ensures that the synthetic data used for pre-training is high-quality.

• Scalability: It provides a scalable path to generating diverse, high-fidelity training data without the prohibitive cost of real-world data collection.
• Generalization: The method significantly improves the ability of robot policies to generalize to new objects, tasks, and even different robot embodiments (sim-to-real and cross-embodiment transfer).
• Paradigm Shift: It shifts the focus from merely generating "realistic-looking" videos to generating "action-consistent" trajectories, establishing a new standard for evaluating synthetic robot data.