Roomify: Spatially-Grounded Style Transformation for Immersive Virtual Environments

Roomify is a spatially-grounded transformation system that generates themed virtual environments anchored to users' physical rooms by preserving key functional and geometric properties, thereby overcoming the trade-off between full immersion and spatial awareness while enabling personalized design through an interactive authoring tool validated by improved presence and professional workflow value.

The Gist

Imagine you are sitting in your living room, surrounded by your familiar sofa, coffee table, and bookshelf. Now, imagine you put on a VR headset and suddenly, you aren't in your living room anymore—you are inside a Harry Potter castle, a pirate ship, or a futuristic spaceship.

The problem with most VR headsets today is a bit like a "choose your own adventure" book where you have to pick between two bad options:

1. The "Glass Wall" Option: You can see your real room through the headset (like looking through a window), so you know where your coffee table is and won't trip. But it feels like you're just watching a movie on a screen, not being in the story. The magic is broken.
2. The "Total Fantasy" Option: The headset replaces your whole room with a digital world. It feels incredibly magical and immersive, but you can't see your real furniture. You might try to sit on a digital chair that isn't there, or trip over a real table you can't see. It's dangerous and disorienting.

Roomify is the "magic middle ground." It's a new system that lets you keep the safety of your real room while transforming its style into anything you can imagine.

The Magic Recipe: How Roomify Works

Think of Roomify as a digital interior designer that speaks two languages: "Real World" and "Fantasy World." Here is how it works, step-by-step:

1. The "X-Ray" Scan (Seeing the Skeleton)

First, you walk around your room with a VR headset (like a Meta Quest 3) and film a short video. Roomify acts like an X-ray machine. It doesn't just see the walls; it understands the skeleton of your room.

• It knows: "That's a sofa, it's about 2 meters long, and you can sit on it."
• It knows: "That's a coffee table, it's in the middle, and you shouldn't walk through it."
• It creates a digital "scaffold" (a wireframe blueprint) of your real furniture.

2. The "Style Translator" (The Dream Weaver)

Next, you tell the system what you want. You might say, "Make my room look like a cozy, rustic forest cabin," or "Turn this into a pirate ship." You can even upload a picture of a style you like.

The system's AI acts like a creative translator. It takes your "Forest Cabin" idea and looks at the "skeleton" of your real room. It asks:

• How do we turn this real sofa into a "Forest Cabin" sofa?
• Answer: We keep the shape and size (so you can still sit on it), but we change the texture to look like rough wood and moss.
• How do we turn this real fridge into a "Pirate Ship" item?
• Answer: We turn it into a giant wooden barrel (which still holds things, just like a fridge), but we keep it in the exact same spot so you don't walk into it.

3. The "Hybrid" Experience (The Best of Both Worlds)

Finally, you put the headset back on.

• The Magic: Your walls are now covered in magical murals, your floor looks like ancient stone, and your sofa is a throne. You feel like you are in a different world.
• The Safety: Because the system kept the shape and location of your real furniture, you can still walk around without bumping into things. If you reach out to grab your "magical" coffee table, your hand hits the real table. You never lose your sense of where you are.

Why Is This a Big Deal?

The researchers tested this with regular people and professional designers, and here is what they found:

• It feels more real: People felt 63% more "present" (like they were actually there) compared to just looking through the camera (passthrough).
• It's safer than pure VR: People felt much more aware of their surroundings compared to being in a completely fake digital room where they might trip.
• It's a creative tool: Designers loved it because they could instantly see what a client's living room would look like as a "Space Station" or a "Medieval Castle" without building a physical model.

The "Glitch" in the Magic

The paper admits the system isn't perfect yet. Sometimes, the AI gets a little "dreamy" and changes the shape of things too much.

• Example: You have a square table. The AI might turn it into a round table because it thinks that fits the "Pirate" theme better.
• The Fix: Roomify has a special "editor mode" where you can see the invisible wireframe of your furniture. If the AI made a mistake, you can drag the wireframe back to the right shape before you fully dive into the experience.

The Bottom Line

Roomify is like having a magic paintbrush that paints over your real house without erasing the furniture underneath. It lets you live in a fantasy world while keeping your feet firmly planted on your real floor, making VR safer, more immersive, and incredibly fun.

Technical Summary

Here is a detailed technical summary of the paper "Roomify: Spatially-Grounded Style Transformation for Immersive Virtual Environments."

1. Problem Statement

Virtual Reality (VR) systems face a fundamental design trade-off between immersion and spatial awareness:

• Fully Virtual Environments: Offer high immersion but isolate users from their physical surroundings, leading to safety risks (collisions with real furniture) and spatial disorientation.
• Passthrough/Mixed Reality (MR): Maintains spatial awareness by showing the real world but breaks immersion by constantly reminding users they are wearing a headset (the "dream collapsing" effect).
• Existing Solutions: Current approaches like boundary warnings (2D grids) or scene proxies (replacing objects with virtual counterparts) often fail to balance stylistic freedom with geometric consistency. They either restrict transformation to walkable areas, rely on limited asset libraries, or apply surface stylization without preserving the functional semantics of physical objects.

The Core Challenge: How to generate radical, themed virtual environments that are anchored to a user's physical room, preserving the spatial layout and functional semantics of furniture (for safety and navigation) while allowing for complete aesthetic transformation.

2. Methodology

Roomify addresses this challenge through a spatially-grounded generation pipeline and a cross-reality authoring tool.

A. The Generation Pipeline

The system converts a physical room into a themed virtual environment through four stages:

1. Spatial Scene Understanding:

   • Input: A 30–60 second monocular RGB video (captured via Meta Quest 3).
   • Process: Uses SLAM3R for geometric reconstruction and U-ARE-ME for alignment (Manhattan axes/gravity). SpatialLM performs semantic parsing to identify architectural elements (walls, doors) and furniture, outputting a structured JSON scene description with oriented bounding boxes (OBBs).
   • Output: A "spatial scaffold" encoding geometric constraints and semantic roles (e.g., "seating," "storage").

2. Style Extraction and Mapping:

   • Input: Multimodal user prompts (text + reference images).
   • Process: An LLM agent (o4-mini) extracts structured style keywords (category, palette, materials). A mapping agent generates a transformation table categorizing components into: (1) In-scene objects, (2) Boundary elements, and (3) Environmental context.
   • Constraints: The system enforces four objectives: Functional Consistency (e.g., a sofa remains a seat), Style Coherence, Environmental Consistency, and Interaction Safety (collision risk assessment).

3. Multi-Modal Content Generation:

   • In-Scene Objects: Uses a reference-guided 2D-to-3D workflow.
     • Step 1: Select the "best-view" frame from the scan video.
     • Step 2: Generate a stylized 2D image using GPT Image-1 (preserving geometry/orientation).
     • Step 3: Convert the 2D image to a 3D model using Tripo AI (v2.5).
   • Boundaries: Generates seamless PBR textures for walls/floors.
   • Environment: Generates dynamic skyboxes (static + audio) using Blockade Labs and Kling-v2-1.

4. Scene Composition:

   • Registration: Uses an algorithm to align generated 3D assets with the original spatial scaffolds via isotropic scaling and orientation optimization (IoU maximization).
   • Integration: Applies textures with offsets to prevent occlusion conflicts and integrates the skybox for unified lighting.

B. Cross-Reality Authoring Tool

To address registration errors and provide user control, Roomify offers a dual-mode interface:

• MR Mode (Grounding): Users view the physical room overlaid with wireframe scaffolds. They can manipulate bounding boxes (rotate, move, scale), correct semantic labels, and provide multimodal style inputs. This ensures the virtual content aligns with physical reality.
• VR Mode (Immersion): Users transition to a fully immersive view of the transformed environment.
• Feedback Loop: If a user moves furniture in the real world, they can return to MR mode to update the virtual representation, maintaining spatial correspondence.

3. Key Contributions

1. Spatially-Grounded Generation Pipeline: A novel approach that treats physical rooms as "spatial containers." It balances stylistic flexibility with geometric preservation, ensuring that while aesthetics change radically, the functional roles (seating, storage) and gross geometry (footprint, volume) of objects remain consistent for safe navigation.
2. Cross-Reality Authoring Tool: A workflow that leverages MR for precise spatial editing and VR for immersive preview. It allows users to iteratively refine AI-generated content directly within the context of their physical space.
3. Empirical Validation: Two comprehensive user studies demonstrating that Roomify significantly outperforms existing baselines in presence and spatial awareness.

4. Results

The system was evaluated through two studies:

Study 1: Real-World User Experience (N=18)

• Comparison: Roomify vs. Passthrough vs. Fully Virtual (Skybox only).
• Presence: Roomify achieved a 63% improvement in presence over Passthrough and 26% over Fully Virtual baselines.
• Spatial Awareness: Roomify maintained significantly better spatial awareness than Fully Virtual environments (p < 0.05), allowing users to navigate safely without seeing the real world directly.
• Performance: In a "Treasure Hunt" task, Roomify users had the fastest completion times (though not statistically significant) and fewer collision incidents than the Fully Virtual group.
• Usability: System Usability Scale (SUS) score of 78.97 ("Good" to "Excellent").

Study 2: Creative Prototyping (N=8 Design Professionals)

• Comparison: Roomify vs. AI Re-Texturing vs. Text-to-3D Generation.
• Scene Quality: Roomify scored 5.95/7, significantly higher than AI Re-Texturing (3.41) and Text-to-3D (4.50).
• Creativity Support: Roomify scored 6.08/7, praised for its ability to generate novel forms while maintaining semantic consistency (e.g., turning an AC unit into a fireplace that still regulates "temperature").
• Workflow Value: Professionals identified Roomify as a valuable tool for early-stage conceptualization, client visualization, and storyboarding.

5. Significance

• Reconciling Safety and Immersion: Roomify proves that high-fidelity immersion does not require sacrificing spatial awareness. By transforming physical cues into style-consistent virtual elements rather than overlaying non-diegetic warnings, it keeps users "in the story" while preserving their ability to navigate safely.
• Semantic Preservation: The system moves beyond simple texture swapping. It understands the function of objects (e.g., a chair is for sitting), allowing for creative transformations that respect human affordances (e.g., a pirate barrel that still functions as a seat).
• Democratizing Spatial Design: The tool lowers the barrier for creating personalized, room-scale virtual environments, enabling both casual users (for entertainment) and professionals (for design prototyping) to rapidly iterate on spatial concepts grounded in reality.
• Future Directions: The authors acknowledge limitations in geometric hallucinations (e.g., shape distortion) due to current generative models but note that rapid advancements in foundation models (e.g., Gemini 3 Pro) are already mitigating these issues. Future work aims to handle dynamic entities and multi-user scenarios.