MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation

The paper introduces MolmoSpaces, a large-scale, open, and simulator-agnostic ecosystem comprising over 230,000 diverse indoor environments and 130,000 annotated objects, designed to enable robust large-scale benchmarking, training, and evaluation of robot navigation and manipulation policies with strong sim-to-real correlation.

The Gist

Imagine you are trying to teach a robot to be the ultimate housekeeper. You want it to be able to walk into any house, find a specific cup, open a fridge, and make a sandwich, even if the house looks nothing like the one it practiced in.

The problem? Real life is messy. There are millions of ways a kitchen can be arranged, thousands of different types of cups, and infinite ways to ask the robot to "get the milk." If you only train a robot in a few perfect, clean test kitchens, it will fail the moment it sees a real, cluttered home.

This paper introduces MolmoSpaces, which is essentially a massive, open-source "video game" universe for training robots.

Here is the breakdown of what they built, using some everyday analogies:

1. The Ultimate Sandbox (The Environments)

Think of existing robot training tools as having a few small, static playrooms. MolmoSpaces is like a giant, procedurally generated theme park with over 230,000 different rooms.

• The Variety: It includes everything from cozy, hand-crafted apartments to wild, computer-generated houses with weird layouts. It even has non-house places like museums, cafes, and offices.
• The Goal: By throwing a robot into 230,000 different scenarios, you force it to learn how to adapt, rather than just memorizing one specific room.

2. The Toy Box (The Objects)

A robot can't learn to pick things up if it only knows how to pick up a red block.

• MolmoSpaces comes with a digital toy box of 130,000 objects.
• These aren't just simple shapes; they are detailed 3D models of real things: mugs, laptops, vases, and even complex items like refrigerators with doors that swing open or drawers that slide out.
• The "Grasp" Secret Sauce: The biggest challenge for robots is knowing how to grab something without dropping it. The authors didn't just give the robot the objects; they pre-calculated 42 million different ways to grab them. It's like having a master chef who has already figured out the perfect grip for every single ingredient in the universe, so the robot can just follow the recipe.

3. The Simulator (The Physics Engine)

Usually, video games look good but physics are fake (e.g., a cup might pass through a table).

• MolmoSpaces is built to be physically realistic. If a robot pushes a heavy box, it slides realistically. If it tries to grab a slippery glass, it might drop it.
• Crucially, this "game" works with the three most popular robot training engines (MuJoCo, Isaac, and ManiSkill). It's like building a game that can be played on a PlayStation, an Xbox, and a PC simultaneously, so no one is left out.

4. The Final Exam (MolmoSpaces-Bench)

How do you know if the robot is actually smart? You give it a test.

• The authors created a standardized exam with 8 different tasks, like "Go find the lamp," "Open the oven," or "Put the apple next to the bowl."
• The "Zero-Shot" Challenge: They tested the robots without letting them study for the specific exam questions. They just threw the robot into a new, unseen room and said, "Do this." This tests if the robot truly understands the world or if it just memorized the answers.

5. The "Real World" Connection

The biggest fear in robotics is the "Sim-to-Real Gap." This is when a robot is a genius in the video game but a total disaster in the real world.

• The paper shows that MolmoSpaces is incredibly accurate. They found a 96% correlation between how well a robot did in the simulation and how well it did in the real world.
• The Analogy: It's like a flight simulator so realistic that if a pilot can land a plane in the simulator, they can almost certainly land it in a real storm.

Why This Matters

Before this, researchers had to build their own tiny, limited test worlds, which made it hard to compare different robots.

• MolmoSpaces is the "Open Source" revolution. It gives everyone the same massive, diverse, and realistic playground.
• It allows researchers to stress-test robots against the "long tail" of weird, rare situations (like a kitchen with a cat on the counter and a broken drawer) that usually break robots.

In short: MolmoSpaces is the ultimate training ground that turns robots from clumsy beginners into adaptable generalists, ready to tackle the messy, unpredictable reality of our actual homes.

Technical Summary

1. Problem Statement

Deploying generalist robots in the real world requires robustness against the "long tail" of everyday situations. Existing robot benchmarks suffer from several critical limitations:

• Scale and Diversity: Most benchmarks rely on a handful of cherry-picked scenes or objects, failing to capture the vast combinatorial space of real-world layouts, object geometries, and task specifications.
• Simulation Fidelity: Many simulators lack realistic physics (e.g., "magic grasps" that bypass contact dynamics) or realistic rendering, leading to poor sim-to-real correlation.
• Task Scope: Existing benchmarks often focus on short-horizon skills in single rooms, neglecting long-horizon, compositional tasks requiring navigation, perception, and manipulation across multi-room environments.
• Evaluation Rigor: Physical evaluation is expensive, slow, and difficult to reproduce, making it impossible to systematically stress-test policies across thousands of scenarios.

The paper argues that to evaluate and improve generalist robot policies, the community needs a large-scale, open, simulator-agnostic ecosystem that supports high-fidelity physics, diverse assets, and rigorous benchmarking.

2. Methodology: The MolmoSpaces Ecosystem

The authors introduce MolmoSpaces, a fully open ecosystem designed to unify diverse scenes, objects, tasks, and tools. It is built upon a modular architecture supporting multiple simulators (MuJoCo, IsaacSim, and ManiSkill).

A. Core Components

1. MolmoSpaces-Scenes (230k+ Environments):
   • Aggregates and processes scenes from AI2-THOR, ProcTHOR, and Holodeck.
   • Includes MSCrafted (120 hand-crafted single-room scenes), MSProc (12k procedural residential houses), MSProcObja (110k scenes with Objaverse objects), and MSMultiType (110k diverse scene types like museums, cafes, and labs generated via LLMs).
   • All scenes are tuned for physical stability (no drifting objects) and navigability.
2. MolmoSpaces-Objects (130k+ Assets):
   • Combines 1.6k THOR objects and 129k curated Objaverse models.
   • Assets include rich semantic metadata (WordNet synsets), physical properties (mass, density), and collision meshes (convex decomposition or primitives).
   • Includes 22 categories of articulated objects (e.g., doors, fridges) with annotated joint parameters.
3. MolmoSpaces-Grasp (42M+ Grasps):
   • Generates 6-DoF grasp poses for 48k interactive objects (rigid and articulated).
   • Uses a pipeline involving antipodal sampling, clustering for diversity, and robustness testing (perturbations for rigid objects; actuation feasibility for articulated ones).
   • Crucially, grasps are validated in-situ within scenes to ensure they are not blocked by surrounding geometry.
4. Simulation Infrastructure:
   • Supports multiple robot embodiments (static Franka FR3, mobile Rainbow RB-Y1, floating grippers).
   • Provides modular experiment composition, allowing flexible combination of scenes, tasks, robots, and camera configurations (RealSense, ZED, GoPro).

B. MolmoSpaces-Bench

The authors construct a benchmark suite comprising 8 base tasks evaluated zero-shot (no fine-tuning on benchmark data):

1. Navigate-to: Find and approach a target object.
2. Pick: Grasp and lift an object.
3. Pick-and-Place: Move an object into/onto a receptacle.
4. Pick-and-Place-Next-To: Place an object adjacent to a receptacle.
5. Pick-and-Place-Color: Similar to (3) but with color-differentiated distractors.
6. Open: Open articulated fixtures (drawers, cabinets).
7. Close: Close articulated fixtures.
8. Open-Door: Mobile manipulation task to open a hinged door.

The benchmark includes controlled variants to test robustness against perturbations (lighting, joint noise, camera occlusion) and an LLM-based system for generating long-horizon tasks.

3. Key Contributions

• Scale and Diversity: The largest open ecosystem of its kind, featuring 230,000+ diverse indoor environments and 130,000+ object assets, significantly exceeding prior works (e.g., RoboCasa, Habitat).
• Simulator Agnosticism: Assets are compatible with MuJoCo, IsaacSim, and ManiSkill, all backed by high-fidelity physics, allowing researchers to choose their preferred simulation backend.
• Comprehensive Grasp Data: The first large-scale dataset providing 42 million annotated grasps validated within complex, cluttered scenes, including specific handling for articulated objects.
• Zero-Shot Benchmarking: A rigorous benchmark suite designed to evaluate generalist policies without task-specific fine-tuning, revealing true generalization capabilities.
• Open Source: All assets, code, and benchmarks are publicly available to accelerate community-driven research.

4. Results and Experiments

The authors evaluated several state-of-the-art policies, including Vision-Language-Action (VLA) models (π-family: π0, π0-FAST, π0.5) and contact-conditioned models (CAP), as well as navigation models (RING, DualVLN).

• Sim-to-Real Correlation: The benchmark demonstrates a strong correlation (Pearson R ≈ 0.96, Spearman ρ ≈ 0.98) between simulation success rates and real-world performance (validated against RoboArena and CAP data). This confirms MolmoSpaces is a reliable proxy for real-world evaluation.
• Policy Performance: Newer models (e.g., π0.5) consistently outperform earlier versions. However, all models show brittleness to distribution shifts.
• Sensitivity Analysis:
  • Prompt Phrasing: Minor changes in instruction wording caused significant success rate drops (up to 14% gap) for earlier models, highlighting language-conditioning limitations.
  • Initial Conditions: Performance degraded when initial joint positions deviated from training distributions.
  • Occlusion: Policies relying heavily on wrist-mounted cameras (like π0.5) failed drastically (success dropped to ~2%) when the wrist camera was occluded, whereas third-person camera occlusion had a lesser impact.
  • Grasp Strategy: Different models exhibited distinct preferences (e.g., π0.5 favored top-down grasps, while CAP favored side grasps), affecting success rates on specific object types (mugs vs. bottles).

5. Significance

MolmoSpaces addresses the critical bottleneck in robot learning: the lack of scalable, diverse, and physically grounded evaluation environments.

• Accelerating Research: By providing a "foundation" for scalable data generation and benchmarking, it allows researchers to move beyond small-scale, hand-crafted evaluations.
• Reliable Simulation: The strong sim-to-real correlation validates the use of high-fidelity simulation for training and testing, reducing the cost and time of physical experimentation.
• Insight into Failure Modes: The ecosystem enables systematic stress-testing, revealing specific failure modes (e.g., sensitivity to camera occlusion or prompt phrasing) that are difficult to uncover with physical trials alone.
• Future-Proofing: The modular design and support for long-horizon tasks position the ecosystem to tackle the next generation of challenges in generalist robot intelligence.

In summary, MolmoSpaces provides the necessary infrastructure to rigorously measure and improve the generalization capabilities of embodied AI, bridging the gap between simulation and real-world deployment.