RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots

This paper introduces RoboCasa365, a large-scale simulation framework featuring 365 everyday tasks across 2,500 diverse kitchen environments and extensive human and synthetic demonstration data, designed to provide a reproducible benchmark for evaluating and advancing generalist robot policies through systematic analysis of task diversity, dataset scale, and environment variation.

The Gist

Imagine you want to teach a robot to be the ultimate housekeeper. You want it to be able to make toast, organize the fridge, wash dishes, and maybe even cook a full dinner, all while navigating a messy kitchen.

The problem? Teaching a robot this way is incredibly hard, expensive, and slow. If you try to teach it in a real kitchen, you might break a lot of dishes, the robot might get stuck, and you'd need thousands of hours of human time to show it what to do. Plus, how do you know if the robot is actually getting smarter, or if it just got lucky with one specific kitchen layout?

Enter RoboCasa365.

Think of RoboCasa365 as a "Massive, Infinite Virtual Kitchen Simulator" designed specifically to train and test these general-purpose robots. It's like a video game for robots, but instead of fighting dragons, the robot is learning to make a sandwich.

Here is a breakdown of what makes this paper special, using some everyday analogies:

1. The "Gym" for Robots (The Environment)

Imagine a gym where a human athlete trains. To get strong, they need to lift different weights, run on different terrains, and face different obstacles.

• The Old Way: Most robot simulators were like a single, tiny room with one chair and one table. The robot learned to push that one chair, but if you put it in a real kitchen with a fridge and a stove, it was lost.
• The RoboCasa365 Way: This framework is like a gym with 2,500 different rooms. Some kitchens are small and cluttered; others are huge and modern. Some have red cabinets, others have wood. The robot gets to practice in thousands of different "versions" of a kitchen so it learns the concept of a kitchen, not just one specific room.

2. The "365 Days of Cooking" (The Tasks)

The name "365" isn't random. It represents 365 different everyday tasks, one for every day of the year.

• The Menu: The robot has to learn everything from simple things like "close the fridge" to complex, multi-step chores like "make a hot dog."
• The Complexity: Making a hot dog isn't just one move. It's a chain reaction: Open the fridge -> Grab the sausage -> Put it on a plate -> Open the mustard -> Squeeze mustard -> Put the bun on the plate.
• The Challenge: The paper tests if the robot can handle these long chains of events without forgetting the first step by the time it gets to the last one.

3. The "Tutor" (The Data)

You can't learn to play piano just by reading a book; you need to watch a master and then practice.

• Human Teachers: The researchers recorded over 600 hours of real humans doing these tasks with a robot arm. This is the "master class."
• The AI Copycats: To get even more practice, they used a clever tool called MimicGen. Think of this as a photocopier for robot movements. It took the human demonstrations and generated 1,600+ hours of new, slightly different variations.
  • Analogy: If a human shows the robot how to pour milk into a cup, MimicGen creates 10,000 new videos showing the robot pouring milk into a cup, but sometimes the cup is on the left, sometimes the right, sometimes the milk is cold, sometimes warm. This teaches the robot to be flexible.

4. The "Report Card" (The Benchmarks)

How do you know if the robot is actually smart? You need a standardized test.

• The Exam: The paper sets up three different types of exams:
  1. Multi-Task Learning: Can the robot learn 300 different tasks at once without getting confused?
  2. Foundation Model Training: Can the robot learn a "general knowledge" base from the massive dataset, and then quickly learn a new, specific task with just a little bit of extra practice? (This is like learning general physics so you can easily learn how to build a specific bridge).
  3. Lifelong Learning: Can the robot learn a new skill today without forgetting how to do the skills it learned yesterday? (This is the hardest part, often called "catastrophic forgetting" in AI).

5. The Results: What Did They Find?

The researchers tested the smartest robot brains (AI models) available today on this new "gym."

• Big Data Works: They found that training on huge, diverse datasets makes robots much better at generalizing.
• Pre-training is Key: Just like a human student who reads a library of books before taking a specific exam, robots that were "pre-trained" on the massive RoboCasa365 data learned new tasks much faster and with less data than robots that started from scratch.
• The Gap: Even with all this data, robots still struggle with very long, complex tasks (like making a full meal) and sometimes forget old skills when learning new ones. But, the paper proves that simulation is the key to getting us there.

The Bottom Line

RoboCasa365 is a massive, open-source playground. It's not just a dataset; it's a complete ecosystem that allows researchers to stop building their own tiny, broken kitchens and start testing their robot brains in a realistic, diverse, and huge virtual world.

It's the difference between teaching a child to swim in a bathtub versus teaching them in a massive, wave-filled ocean. RoboCasa365 is that ocean, and it's helping us figure out how to build robots that can actually live and work in our homes.

Technical Summary

Here is a detailed technical summary of the paper "RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots."

1. Problem Statement

The field of robot learning is moving toward "generalist" robots capable of performing diverse everyday tasks in human environments. However, progress is hindered by two major bottlenecks:

1. Data Scarcity and Diversity: Training robust generalist policies requires vast amounts of diverse interaction data. Existing real-world datasets are limited in scale, task coverage, and environmental variety.
2. Evaluation Limitations: Real-world benchmarking is resource-intensive, noisy, and difficult to reproduce. Existing simulation frameworks often lack the scale (thousands of scenes) and diversity (hundreds of tasks) necessary to systematically evaluate how task diversity, environment variation, and dataset scale affect policy generalization.

2. Methodology: The RoboCasa365 Framework

The authors introduce RoboCasa365, a comprehensive simulation benchmark built upon the existing RoboCasa platform. It is designed to support systematic evaluation across three learning paradigms: multi-task learning, foundation model training, and lifelong learning.

A. Core Components

• Diverse Environments (2,500 Scenes): The framework features 2,500 unique kitchen scenes generated by combining 50 distinct floor plans (modeled from real US homes via Zillow) with 50 distinct styles (fixtures, appliances, textures). This creates a massive distribution of visual and structural variations.
• Task Suite (365 Tasks): The benchmark defines 365 everyday tasks categorized into:
  • 65 Atomic Tasks: Single-skill executions (e.g., opening a drawer, turning a knob).
  • 300 Composite Tasks: Multi-step sequences involving semantic reasoning, long-horizon planning, and memory. These span 60 activity families (e.g., "Cooking," "Cleaning," "Organizing") and include tasks requiring mobile manipulation.
• Large-Scale Datasets (>2,000 Hours):
  • Human Data: 612 hours of teleoperated demonstrations (Franka Panda + mobile base) covering 300 pretraining tasks and 50 target tasks.
  • Synthetic Data: 1,615 hours of data generated using MimicGen, scaling human demonstrations by 100x for 60 atomic tasks.
  • Total: Over 500,000 trajectories.

B. Experimental Setup

The authors evaluated state-of-the-art Vision-Language-Action (VLA) models, including Diffusion Policy, $\pi_0$, $\pi_0.5$, and GR00T N1.5. The experiments focused on four key research questions:

1. Performance of multi-task training on massive datasets.
2. The efficacy of pretraining followed by fine-tuning (Foundation Model paradigm).
3. Capabilities in lifelong learning (sequential task acquisition).
4. The impact of pretraining data composition (task vs. scene diversity, human vs. synthetic data).

3. Key Contributions

1. Scale and Diversity: RoboCasa365 is the first simulation framework to simultaneously offer thousands of unique scenes, hundreds of diverse tasks, and tens of thousands of high-quality demonstrations, surpassing prior benchmarks like Behavior-1K or RoboSuite.
2. Systematic Benchmarking Suite: It provides a standardized protocol for evaluating generalist robots across three distinct learning settings (Multi-task, Foundation, Lifelong), enabling reproducible comparisons.
3. Real-World Validation: The framework includes a "Sim-and-Real" transfer protocol, demonstrating that policies trained on this simulation data can significantly improve performance on real-world robots (DROID Panda arm).

4. Key Results

The authors conducted extensive experiments yielding the following insights:

• Multi-Task Learning:

  • GR00T N1.5 outperformed other baselines ($\pi_0$, $\pi_0.5$, Diffusion Policy), achieving a 20% average success rate across 50 target tasks.
  • Performance dropped significantly for Composite-Unseen tasks (zero-shot), highlighting the difficulty of generalizing to entirely new task sequences without specific training.
  • High-capacity VLA models showed superior generalization compared to simpler diffusion policies.

• Foundation Model Training (Pretraining + Fine-tuning):

  • Data Efficiency: Pretraining on the large-scale dataset followed by fine-tuning on target data yielded a 3x improvement in data efficiency compared to training on target data alone.
  • Generalization: Pretraining significantly boosted performance on Composite-Unseen tasks, suggesting that broad pretraining helps models learn transferable skills for novel tasks.

• Lifelong Learning:

  • The study revealed catastrophic forgetting. As models learned progressively longer-horizon tasks in new phases, their performance on previously learned tasks steadily degraded. This highlights a critical open challenge in continuous robot learning.

• Data Composition Analysis:

  • Task Diversity > Data Scale: Increasing the number of pretraining tasks (from 50 to 300) had a more significant impact on downstream performance than simply increasing the volume of data for fewer tasks.
  • Synthetic Data Quality: While MimicGen provided massive scale, models trained on Human-only data (300 tasks) sometimes outperformed those trained on Human + Synthetic data, suggesting that synthetic data quality varies and requires better curation methods.
  • Scene Diversity: Increasing the number of pretraining scenes from 5 to 2,500 significantly improved zero-shot and fine-tuned performance.

• Real-World Transfer:

  • A model trained on a mix of simulation and real-world data (Sim-and-Real) achieved 79.8% success on real-world tasks, compared to 61.8% for a model trained on real-world data only. This validates the utility of RoboCasa365 for bridging the sim-to-real gap.

5. Significance

RoboCasa365 represents a major step forward in robot learning infrastructure. By providing a large-scale, diverse, and reproducible environment, it allows researchers to:

• Move beyond narrow benchmarks to study true generalization.
• Systematically analyze the trade-offs between data scale, diversity, and model architecture.
• Develop and test lifelong learning strategies that address catastrophic forgetting.
• Accelerate the deployment of generalist robots by providing a reliable simulation testbed that correlates well with real-world performance.

The paper concludes that while generalist policies are improving, challenges remain in long-horizon planning, robustness to physical perturbations (camera pose, joint angles), and retaining knowledge during lifelong learning. RoboCasa365 serves as the foundational tool to address these future challenges.