MANSION: Multi-floor lANguage-to-3D Scene generatIOn for loNg-horizon tasks

This paper introduces MANSION, a language-driven framework that generates realistic, multi-floor 3D building environments and the MansionWorld dataset to address the limitations of current single-floor benchmarks in evaluating long-horizon robotic tasks requiring complex spatial reasoning.

The Gist

🏗️ The Big Idea: Building a Digital City for Robots

Imagine you want to teach a robot how to be a helpful assistant. You might start by teaching it to tidy up a single bedroom. That's easy. But in the real world, robots don't just live in one room; they need to work in entire buildings. They need to go from the lobby to the 4th floor, grab a package from a hospital wing, and bring it to a doctor's office.

The problem? Most robot training happens in "digital apartments" that only have one floor. It's like training a pilot in a simulator that only has a single runway. When the robot tries to navigate a real skyscraper with elevators, stairs, and complex layouts, it gets completely lost.

MANSION is a new tool that solves this by automatically generating entire, multi-story buildings for robots to practice in, just by typing a simple sentence.

---

🧠 How It Works: The "Architect + Builder" Team

Think of MANSION not as a single robot, but as a construction crew working together:

1. The Chief Architect (The Brain):
   When you type, "Build me a 3-story hospital," a super-smart AI (a Large Language Model) acts as the Chief Architect. It doesn't just draw a picture; it understands the rules. It knows that elevators need to line up vertically, that stairs need to connect floors, and that a hospital needs an emergency room on the ground floor.

   • Analogy: Imagine a master architect who sketches a blueprint on a napkin, ensuring the stairs on the 2nd floor perfectly match the stairs on the 1st floor.

2. The Geometry Solver (The Math Whiz):
   Once the Architect has the idea, a "Geometry Solver" takes over. This is the part that actually builds the walls and rooms. It uses a special "growth" method. Instead of trying to build the whole building at once (which is too hard), it starts with a central hub and grows rooms one by one, checking constantly that they fit together without crashing into each other.

   • Analogy: It's like playing Tetris, but the pieces are rooms, and the solver is a master player who never lets a piece get stuck in a corner where it doesn't belong.

3. The Interior Designer (The Scene Setter):
   Once the walls are up, the system fills the rooms with furniture. But it's smart about it. It knows that chairs in a classroom should be in neat rows, and that a fridge needs to be accessible. It ensures the robot can actually walk to the objects without getting stuck.

---

🌍 MansionWorld: The Ultimate Robot Playground

The authors didn't just build the tool; they used it to create MansionWorld.

• What is it? A massive library of over 1,000 different buildings.
• What's inside? You can find hospitals, supermarkets, office towers, and schools. Some are small, some are 10 stories tall.
• Why is it special? Unlike other datasets that are just static photos, these are interactive. You can walk through them, open doors, take an elevator, and move objects.

The "Magic Wand" Feature:
Imagine you have a generated office building, but you want to test a robot that needs to find a specific "red stapler" on the 3rd floor. If the building doesn't have a stapler, you don't need to rebuild the whole thing.
MANSION has a "Scene Editing Agent" (a digital handyman). You tell it, "Put a red stapler on the desk," and it instantly modifies the scene to make that task possible. It's like using a magic wand to change the props in a movie set without reshooting the whole scene.

---

🤖 The Reality Check: Robots Are Still Learning

The researchers tested their best current robot brains (AI agents) in these new, complex buildings.

• The Result: The robots struggled. They got confused by the stairs, forgot where they were when they took the elevator, and failed to plan long tasks.
• The Lesson: This proves that current robots are like toddlers who can walk in a straight line but can't navigate a busy mall. MANSION exposes these weaknesses, showing us exactly what needs to be improved for robots to become truly useful in our real, multi-story world.

🚀 Summary in One Sentence

MANSION is a language-powered construction kit that builds realistic, multi-story cities for robots to practice in, revealing that while our robots are getting smarter, they still have a long way to go before they can truly navigate the complex buildings of the real world.

Technical Summary

1. Problem Statement

The field of Embodied AI aims to create agents capable of autonomous reasoning and task execution in complex real-world environments. However, current research faces a critical bottleneck:

• Limited Scope: Existing benchmarks and datasets (e.g., AI2-THOR, Habitat) are largely confined to single-floor, residential, or apartment-scale environments.
• Missing Complexity: Real-world tasks (e.g., package delivery in offices, supply transport in hospitals) are long-horizon and multi-floor, requiring agents to navigate vertical structures (stairs, elevators), manage cross-floor memory, and interact with complex architectural elements.
• Data Scarcity: There is a lack of large-scale, diverse, and interactive 3D simulation environments that span entire buildings. Real-world scanned data is expensive and hard to edit, while existing synthetic generators fail to model vertical consistency or cross-floor connectivity.

2. Methodology: The MANSION Framework

MANSION is a language-driven framework designed to generate building-scale, multi-floor 3D environments from natural language instructions. It employs a hybrid Multimodal Large Language Model (MLLM) + Geometric Solver pipeline.

A. Hierarchical Multi-Agent Pipeline

The generation process is divided into four stages (Fig. 2):

1. Whole Building Planning: A "Chief Designer" agent analyzes the user's natural language prompt to determine global constraints (style, functional zones, target area) and defines the vertical connectivity (stairs/elevators) and cross-floor alignment.
2. Per-Floor Planning: Dedicated agents generate a "bubble diagram" (topological graph) for each floor, specifying room adjacencies, target areas, and connections to vertical cores, ensuring semantic consistency across the building.
3. Floorplan Synthesis (Geometric Solver): This is the core innovation. Instead of letting the LLM directly regress room polygons (which leads to errors), the system uses a verifiable constrained search:
   • Hierarchical Splitting: The solver uses a "cutting" strategy. An MLLM provides coarse "seed" locations for rooms within a parent region.
   • Constrained Growth: A geometric solver (based on Lopes-style constrained growth) iteratively splits the free space. It generates candidate partitions, filters them based on topological constraints (e.g., adjacency requirements), and scores them using an energy-based objective function (penalizing area mismatch, seed distance, and irregular shapes).
   • Vertical Alignment: The solver enforces hard constraints to ensure vertical cores (stairs, elevators) and structural walls align perfectly across floors.
4. Scene Instantiation: The vector floorplans are converted into interactive AI2-THOR scenes. This includes placing walls, doors, and objects. The system uses a "quality-first" approach, enforcing hard reachability constraints and using anchor-based grouping for uniform object distribution.

B. MansionWorld Ecosystem

Built upon MANSION, the authors release MansionWorld, a dataset and ecosystem containing:

• Dataset: Over 1,000 diverse buildings (2–10 stories) covering residential, office, hospital, school, and supermarket domains.
• Cross-Floor Assets: New interactive assets for multi-flight stairwells and functional elevators.
• Skill APIs: High-level atomic skills (e.g., UseStairs, CallElevator, UseElevator) that handle scene transitions and state management seamlessly.
• Task-Semantic Scene Editing Agent: An MLLM-driven agent that modifies static generated buildings to meet specific task preconditions (e.g., "Place a snack on the 2nd-floor table"). It uses a "Check-and-Provision" workflow:
  1. Check: Verify path connectivity and object availability using a static semantic state (JSON).
  2. Provision: If objects are missing, the agent uses tools to place them via an on-demand physics engine (AI2-THOR) and updates the state.

3. Key Contributions

1. MANSION Framework: The first language-driven system to generate vertically aligned, multi-story 3D buildings with provable navigability and structural validity. It decouples high-level semantics (LLM) from low-level geometry (solver).
2. MansionWorld Dataset: A large-scale, reusable ecosystem of 1,000+ buildings with cross-floor connectivity, supporting long-horizon tasks in non-residential domains.
3. Task-Semantic Scene Editing: A novel agent that transforms static environments into dynamic playgrounds by programmatically ensuring task executability (adding missing objects) without regenerating the entire scene.
4. New Benchmark: Established a rigorous testbed for evaluating cross-floor planning, memory, and spatial reasoning, revealing significant gaps in current state-of-the-art (SOTA) agents.

4. Experimental Results

The authors evaluated the system on floorplan generation, object placement, and embodied agent performance.

• Floorplan Generation:
  • On the T2D dataset, MANSION achieved performance comparable to state-of-the-art diffusion models (ChatHouseDiffusion) under manual annotation settings (Micro-IoU ~81.67% vs 82.81%).
  • On the more complex ResPlan-1K dataset (larger, more diverse layouts), MANSION significantly outperformed SOTA methods (Micro-IoU 76.74% vs ~33% for CHD), demonstrating superior generalization to non-residential and complex structures.
• Object Placement:
  • MANSION achieved 100% reachability across all tested room types (bedrooms, classrooms, libraries), whereas other methods (LayoutGPT, Holodeck) had significant unreachable zones.
  • User studies showed MANSION generated scenes with higher realism, diversity, and layout quality, particularly in non-residential settings requiring regular arrangements (e.g., office desks).
• Embodied Agent Benchmarking:
  • SOTA agents (BUMBLE, COME-robot) were tested on single-floor, two-floor, and four-floor tasks.
  • Results: Agents performed reasonably on single-floor tasks but degraded sharply on multi-floor tasks. Success rates dropped to 0% for four-floor tasks.
  • Analysis: Failures were attributed to limitations in long-horizon memory, difficulty in identifying objects in large spaces, and the complexity of managing vertical transitions.

5. Significance and Impact

• Bridging the Gap: MANSION addresses the critical disconnect between current embodied AI benchmarks (single-floor) and real-world requirements (building-scale).
• Scalability: The framework proves that building-scale environments can be generated programmatically with high structural validity, enabling the creation of infinite task variations.
• Research Catalyst: By releasing MansionWorld and demonstrating that current agents fail on these tasks, the paper sets a new standard for future research in spatial reasoning, long-horizon planning, and cross-floor navigation. It highlights the need for new algorithms that can effectively manage memory and planning across multiple levels of a building.