SAGE: Scalable Agentic 3D Scene Generation for Embodied AI

SAGE is an agentic framework that automatically generates scalable, realistic, and physically valid 3D environments tailored to specific embodied tasks through iterative reasoning and multi-critic evaluation, enabling the training of policies that generalize effectively to unseen scenarios.

The Gist

Imagine you want to teach a robot how to do chores, like picking up a coffee mug from a nightstand and putting it on a desk. To teach the robot, you usually need thousands of real-world examples. But building a real house, filling it with furniture, and having a robot try to move things around is expensive, slow, and dangerous (imagine a robot knocking over a priceless vase!).

This is where SAGE comes in. Think of SAGE as a super-powered, self-correcting virtual architect and interior designer that builds digital rooms specifically for robots to practice in.

Here is how it works, broken down into simple concepts:

1. The "Agent" (The Project Manager)

Instead of a computer following a rigid, pre-written script (like "put a bed here, then a lamp there"), SAGE uses an AI Agent.

• The Analogy: Imagine you hire a very smart, proactive construction manager. You tell them, "I need a bedroom where a robot can learn to move a mug."
• What they do: The manager doesn't just follow orders; they think. They say, "Okay, I'll start with the walls. Now I need a bed. Wait, if I put the bed here, the robot can't reach the nightstand. Let me move the bed." They constantly adjust the plan based on what they see.

2. The "Critics" (The Inspectors)

The manager doesn't work alone. They have two strict inspectors watching their every move:

• The Visual Critic (The Interior Designer): This inspector looks at the room and says, "Hey, that lamp looks weirdly floating in mid-air," or "This room feels empty; we need a rug." They make sure the room looks real and makes sense.
• The Physics Critic (The Safety Engineer): This is the most important one. They run a "gravity test." They ask, "If we drop this pillow on the bed, will it stay there, or will it slide off and crash?" If the room isn't physically stable, the inspector yells, "Fix it!" and the manager has to rearrange the furniture until it passes the test.

3. The "Loop" (The Practice Run)

This is the magic part.

1. The Manager builds a room.
2. The Inspectors check it.
3. If the inspectors find a problem (e.g., a chair is blocking the robot's path, or a bookshelf is falling over), they tell the Manager.
4. The Manager fixes it and builds it again.
5. They repeat this until the room is perfect, realistic, and safe.

This happens automatically and very fast, creating thousands of unique rooms (bedrooms, kitchens, offices) that a robot can practice in.

4. Why is this a Big Deal?

• No More "Fake" Rooms: Old methods often made rooms that looked okay but were physically impossible (like objects floating or colliding). SAGE ensures the rooms are physics-ready. If a robot tries to walk through a wall in SAGE, it will hit the wall, just like in real life.
• Infinite Variety: You can ask for a "messy teenager's bedroom" or a "futuristic cyberpunk office," and SAGE will build it. It doesn't just copy-paste existing rooms; it creates new ones from scratch.
• Scaling Up: Because SAGE can build these rooms instantly, we can train robots on millions of different scenarios. This makes the robots much smarter and better at handling new situations they've never seen before.

The Result: A Robot That Actually Works

The paper shows that robots trained in these SAGE-generated rooms get really good at tasks like "Pick-and-Place" (grabbing a can and putting it on a plate) or "Mobile Manipulation" (driving a robot cart to a table, grabbing a cup, and moving it).

In short: SAGE is like a video game engine that builds its own levels specifically to train robots. It uses an AI team (Manager + Inspectors) to ensure every level is realistic, safe, and diverse, so the robots learn faster and become safer to use in our real homes.

Technical Summary

1. Problem Statement

Embodied AI (robots interacting with physical environments) suffers from a critical data bottleneck. Collecting real-world interaction data is costly, slow, and unsafe. While simulation offers a scalable alternative, existing 3D scene generation methods fail to meet four key desiderata simultaneously:

1. Realism: High-fidelity geometry, appearance, and physics.
2. Diversity: Wide variety of assets, layouts, and styles.
3. Simulation-Readiness: Scenes must be physically stable, collision-free, and directly compatible with simulators (e.g., Isaac Sim).
4. Task-Awareness: Environments must be tailored to specific robot tasks (e.g., pick-and-place, navigation).

Current approaches fall short:

• Rule-based systems (e.g., ProcTHOR) ensure physics but lack diversity and open-vocabulary control.
• Data-driven/Generative models (e.g., DiffuScene) improve realism but often produce physically invalid scenes (floating objects, collisions) and lack simulator validation.
• Static pipelines lack the ability to self-correct or adapt to user intent iteratively.

2. Methodology: The SAGE Framework

SAGE (Scalable Agentic 3D Scene Generation) is an agentic framework that converts open-vocabulary text prompts (and optional images) into simulation-ready 3D environments. It operates under the Model Context Protocol (MCP), allowing an agent to dynamically orchestrate tools and critics.

A. Agent-Driven Generation Loop

The system uses an LLM agent that iteratively refines a scene through a "generate-critique-refine" loop:

1. Generators (Tools): The agent calls specific tools to construct the scene:
   • Scene Initializer: Generates floor plans, walls, and textures (using MatFuse) based on user prompts.
   • Asset Placer: Generates 3D objects (using TRELLIS) from text, estimates physical properties (mass, friction) via Vision-Language Models (VLMs), and places them using depth-first search with collision avoidance.
   • Asset Mover/Remover: Adjusts or deletes objects based on feedback.
2. Critics (Feedback Mechanisms): Two complementary critics evaluate the scene after every modification:
   • Visual Critic: Assesses semantic coherence, spatial layout, and completeness (e.g., "The room lacks a lamp"). It suggests adding or moving objects.
   • Physics Critic (Simulator-in-the-Loop): Loads the scene into Isaac Sim to validate physical stability. It checks for collisions, gravity-induced falls, and stability. If an object falls or collides, the critic rejects the placement and suggests alternatives (e.g., "Move the lamp to the corner" or "Remove the ottoman").

B. Scaling via Augmentation

To train robust policies, SAGE scales single generated scenes into diverse datasets through three levels of augmentation:

1. Object Configuration: Resampling the pose of task-relevant objects.
2. Object Category: Using LLMs to vary geometry and texture (shape, color, material) of objects while keeping the category (e.g., changing a "red mug" to a "blue ceramic mug") and regenerating them via TRELLIS.
3. Scene Layout: Regenerating the background environment (room geometry, non-task objects) while keeping task-relevant objects fixed but repositioned.

C. Action Generation & Policy Learning

• Action Synthesis: For each generated scene, the system automatically synthesizes robot demonstrations using motion planning:
  • Pick-and-Place: Uses M2T2 for grasp pose generation and Curobo for collision-free Inverse Kinematics (IK).
  • Mobile Manipulation: Uses RRT for navigation path planning combined with pick-and-place.
• Policy Training: The resulting large-scale dataset is used to train Diffusion Policies via imitation learning (Robomimic).

3. Key Contributions

1. Agentic Framework: A novel system unifying multiple generators and critics under MCP, enabling adaptive tool use and self-improvement without hard-coded pipelines.
2. Simulator-Validated Stability: The unique integration of a physics critic with Isaac Sim ensures that generated scenes are physically valid (near 100% stability) and directly deployable, a gap left by prior works like SceneWeaver.
3. Open-Vocabulary & Diversity: Supports arbitrary text prompts (e.g., "Cyberpunk game den") and optional image conditioning, generating diverse, stylized, and semantically coherent scenes.
4. SAGE-10k Dataset: A pre-generated dataset of 10,000 scenes across 50 room types and 50 styles, containing 565k unique 3D objects, released to the community.

4. Experimental Results

The authors evaluated SAGE against state-of-the-art baselines (Holodeck, SceneWeaver) and demonstrated its effectiveness in policy learning.

• Scene Generation Quality:

  • Physical Stability: SAGE achieved 99.8% stability and 1.9% collision rate across common room types (Bedroom, Kitchen, Living Room), significantly outperforming baselines (e.g., SceneWeaver had 58.8% stability and 31% collisions).
  • Visual Quality: SAGE scored higher in realism, functionality, and completeness metrics.
  • Open-Vocabulary: Successfully generated complex, stylized scenes (e.g., "Starry-night bedroom," "Rusty restroom") that baselines failed to render coherently.

• Policy Learning & Scaling:

  • Scaling Trends: Policies trained on SAGE data showed clear performance improvements as the number of scenes and demonstrations increased.
  • Generalization: Policies trained on SAGE data generalized significantly better to unseen objects and layouts compared to policies trained on baseline data.
  • Performance: The learned policy achieved success rates approaching those of a "privileged agent" (which has full 3D scene information), demonstrating the high fidelity of the generated data.
  • Cross-Evaluation: SAGE-trained policies outperformed baselines even when tested on scenes generated by the baselines, proving robust generalization.

5. Significance

SAGE represents a paradigm shift in embodied AI data generation. By moving from static, rule-based, or purely generative pipelines to an agentic, self-correcting loop with simulator-in-the-loop validation, it solves the "sim-to-real" gap caused by physically invalid training data.

• Scalability: It enables the infinite generation of diverse, high-quality training data, addressing the data scarcity bottleneck in robotics.
• Safety & Cost: It eliminates the need for expensive and dangerous real-world data collection for initial policy training.
• Future Impact: The framework lays the groundwork for scalable, simulation-driven learning, potentially accelerating the deployment of general-purpose robots in unstructured real-world environments.

In summary, SAGE provides a complete pipeline from text prompt $\to$ physically valid 3D scene $\to$ robot action data $\to$ generalizable policy, demonstrating that high-quality, simulation-validated synthetic data can effectively drive embodied AI learning.