FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flow

FlowScene introduces a tri-branch generative model that leverages multimodal graphs and a tight-coupled rectified flow mechanism to simultaneously generate high-fidelity, style-consistent indoor scenes with precise control over object layouts, shapes, and textures.

The Gist

Imagine you are an interior designer, but instead of just sketching on a napkin, you have a magical assistant that can instantly build a fully furnished, 3D room based on your wildest ideas.

This paper introduces FlowScene, a new AI system that acts as that super-powered assistant. Its main goal is to solve a specific problem: How do you get an AI to build a room where every piece of furniture looks like it belongs to the same family, fits perfectly together, and follows your exact instructions?

Here is the breakdown of how FlowScene works, using simple analogies.

1. The Problem: The "Frankenstein" Room

Previous AI tools for making 3D rooms often made two types of mistakes:

• The "Retrieval" Mistake: Imagine asking an AI to build a bedroom. It goes to a giant warehouse of pre-made furniture, grabs a modern leather sofa, a Victorian wooden bed, and a futuristic neon lamp, and dumps them in the room. They fit the names you asked for, but they look terrible together. They lack style consistency.
• The "Blurry" Mistake: Other tools could make things look consistent, but the shapes were often mushy or wrong (like a chair with no legs), and you couldn't control exactly where things went.

2. The Solution: The "Multimodal Graph" (The Blueprint)

FlowScene starts by listening to you. You can talk to it ("I want a big bed next to a small nightstand"), click buttons on a screen, or even upload a photo of a specific chair you like.

The system translates all these messy inputs into a Multimodal Graph.

• Think of this as a "Smart Blueprint."
• Instead of just a list of items, it's a map where every object is a Node (a dot) and every relationship is a Line connecting them.
• The Magic: These nodes aren't just text. They are "multimodal," meaning they can hold text ("red chair"), a picture of a chair, or both. If you don't have a picture for the nightstand, the system knows to look at the "red chair" node to guess what style the nightstand should have.

3. The Engine: "Rectified Flow" (The Fast-Forward Train)

Most AI image generators work like a sculptor chipping away at a block of stone, slowly removing noise to reveal the image. This is slow.

FlowScene uses something called Rectified Flow.

• The Analogy: Imagine you are at a train station. Old AI methods are like a train that stops at 100 tiny stations to get to the destination. Rectified Flow is like a high-speed maglev train that draws a perfectly straight line from "Noise" to "Perfect Room."
• Because the path is straight and direct, FlowScene is much faster than previous methods, generating complex scenes in seconds rather than minutes.

4. The Secret Sauce: The "Three-Branch" Team

FlowScene doesn't just build the room in one go. It uses a team of three specialized workers (branches) that talk to each other constantly:

1. The Architect (Layout Branch): Decides where things go. "The bed goes here, the lamp goes there."
2. The Sculptor (Shape Branch): Decides what the objects look like physically. "The bed needs to be wide, the chair needs four legs."
3. The Painter (Texture Branch): Decides the style and color. "The bed is oak wood, the chair is velvet."

The "InfoExchangeUnit" (The Team Huddle):
This is the most important part. In other systems, these three workers might work in isolation. In FlowScene, they are in a constant huddle.

• While the Sculptor is shaping the bed, it asks the Architect: "Is the bed big enough to fit in this space?"
• While the Painter is choosing the fabric, it asks the Sculptor: "Is this fabric too heavy for this chair shape?"
• If you say "The nightstand must be the same style as the bed," the Painter instantly shares that "style code" with the Sculptor and Architect so they all agree on the look.

This constant communication ensures that if you ask for a "modern" room, the bed, the chair, and the lamp all look modern, not just one of them.

5. Why It Matters

• It's Fast: It generates scenes in a fraction of the time of older tools.
• It's Consistent: It ensures the whole room feels like it was designed by a single human, not a random collection of items.
• It's Flexible: You can give it a text description, a sketch, a photo, or a mix of all three, and it understands.

Summary

Think of FlowScene as a conductor for an orchestra.

• The Multimodal Graph is the sheet music (the instructions).
• The Three Branches are the different sections of the orchestra (strings, brass, percussion).
• The Rectified Flow is the baton that keeps the tempo fast and direct.
• The InfoExchangeUnit ensures that the violins and the drums are playing the same song in the same key, resulting in a beautiful, harmonious 3D room instead of a chaotic noise.

This technology could revolutionize how we design homes, create video game worlds, or even plan robot movements in real-world spaces.

Technical Summary

1. Problem Statement

The paper addresses the challenge of generating high-fidelity, style-consistent 3D indoor scenes from diverse user inputs (text, GUI selections, or mixed modalities). Existing approaches suffer from significant limitations:

• Language-driven retrieval methods: While they can compose plausible scenes from large databases, they lack fine-grained control over individual object geometry and often fail to enforce scene-level style coherence, leading to mismatches in scale, topology, and appearance.
• Graph-based generation methods: These offer better controllability over object relations and structural consistency but typically fail to generate textured scenes in an end-to-end manner. They often produce low-fidelity results or struggle to maintain visual style consistency across objects, limiting their practical utility.

The core goal is to achieve simultaneous high fidelity, precise geometric control, and holistic style consistency across structure and texture, even when some objects lack specific visual or textual cues.

2. Methodology: FlowScene

FlowScene is a tri-branch generative model conditioned on Multimodal Graphs. It decomposes scene generation into three coordinated branches: Layout, Shape, and Texture.

A. Input Representation: Multimodal Graph

The system accepts a scene graph $G_M = (V_M, E)$ where:

• Nodes ($V_M$): Represent objects and can fuse textual embeddings (e.g., from CLIP), visual features (e.g., from DINOv2), or both. Missing modalities are zero-padded.
• Edges ($E$): Encode semantic and spatial relationships (e.g., "left of," "bigger than," "same style as").
• Graph Constructor: Modern LLMs or VLMs parse user inputs (text descriptions or GUI selections) into this multimodal graph structure.

B. Core Mechanism: Multimodal Graph Rectified Flow

The backbone of FlowScene is a tightly coupled Rectified Flow model. Unlike standard diffusion models that process nodes independently or sequentially, FlowScene employs an InfoExchangeUnit (based on a Triplet-Graph Convolutional Network) during the denoising process.

• Mechanism: As the model iteratively denoises data from noise to the target distribution, every node carries a denoising state specific to its branch (bounding boxes for layout, voxelized latents for shape, structured latents for texture).
• Information Exchange: Nodes interact iteratively along graph edges. This allows the model to exchange contextual information (e.g., if one chair is "same style as" another, the texture branch ensures consistency; if a bed is "left of" a wardrobe, the layout branch adjusts positions).
• Advantage: This enables collaborative reasoning across the graph, satisfying both individual object constraints and holistic scene constraints simultaneously. It utilizes Rectified Flow (straight-line paths in latent space) rather than standard diffusion, allowing for faster sampling (fewer steps) and reduced training variance.

C. The Three Generation Branches

1. Layout Branch: Generates 3D bounding boxes (location, size, rotation) for all objects. It uses a LayoutExchangeUnit to ensure spatial constraints (e.g., "behind," "close by") are met.
2. Shape Branch: Generates voxelized object shapes. It uses a ShapeExchangeUnit to ensure geometric consistency (e.g., ensuring two "same style" chairs have similar structural forms). It employs a Shape VQ-VAE to encode sparse voxel grids into compact latent codes.
3. Texture Branch: Generates object textures anchored to the generated geometry. It uses a TextureExchangeUnit to propagate style information. Crucially, for text-only nodes (objects without reference images), the texture is inferred from the exchanged information of other nodes in the graph, ensuring style consistency. It employs a Texture VQ-VAE and a Sparse Transformer.

The final scene is synthesized by populating the generated layout with the textured 3D objects.

3. Key Contributions

1. FlowScene Framework: A novel tri-branch generative model that produces high-fidelity 3D scenes with both geometric and appearance control, conditioned on multimodal graphs.
2. Multimodal Graph Rectified Flow: A core innovation that tightly couples the generation of layout, shape, and texture by exchanging node information during the sampling process. This mechanism ensures fine-grained control over per-object attributes while enforcing scene-level style coherence.
3. Efficiency and Performance: By leveraging Rectified Flow, the model achieves faster generation speeds compared to previous diffusion-based graph methods while significantly outperforming them in realism and consistency.
4. Diverse Input Support: The system supports flexible inputs, including pure text, GUI-based object selection, and mixed inputs with reference images, making it adaptable to various user workflows.

4. Experimental Results

The authors evaluated FlowScene on the SG-FRONT and 3D-FRONT datasets, comparing it against training-free retrieval methods (Holodeck, LayoutVLM) and graph-conditioned generative baselines (CommonScenes, EchoScene, MMGDreamer).

• Scene-Level Realism: FlowScene achieved the lowest Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) across bedroom, living room, and dining room scenarios, indicating a closer match to real-world scene distributions.
• Object-Level Realism: It demonstrated superior performance in Minimum Matching Distance (MMD) and Coverage (COV) metrics, generating more diverse and geometrically accurate object shapes.
• Controllability & Consistency: FlowScene achieved the highest CLIPScore and FPVScore (a metric for vision-text consistency), showing strong adherence to user instructions. It significantly outperformed baselines in Style Consistency (SC), ensuring objects share coherent materials and design themes.
• Human Preference: In a perceptual study with 25 participants, FlowScene received the highest scores for Overall Preference, Visual Quality, and Style Consistency.
• Efficiency: FlowScene is significantly faster than graph-conditioned diffusion baselines. For example, it generates layout and shape in 6.83 seconds (84.93% faster than MMGDreamer) and completes the full textured scene in 37.38 seconds.

5. Significance

FlowScene represents a significant advancement in 3D content creation for industries such as interior design, manufacturing, VR/AR, and robotics.

• Bridging the Gap: It successfully bridges the gap between the controllability of graph-based methods and the high-fidelity, style-consistent generation of generative models.
• Scalability: The use of Rectified Flow allows for efficient, few-step generation, making it viable for real-time or interactive applications.
• Robustness: The ability to infer missing visual details (textures/shapes) for text-only nodes through graph-based information exchange makes the system robust to incomplete user inputs.
• Future Impact: The proposed workflow provides a robust backend for converting diverse, natural user intents into precise, high-quality 3D environments, paving the way for more interactive and personalized digital twin and metaverse applications.