Technical Summary: SceneConductor

Problem Statement

Generating complete, coherent 3D scenes from a single image is a fundamentally challenging task due to the inherent ambiguity of visual evidence. It requires jointly inferring object geometry, spatial layout, inter-object relationships, and environmental context. Existing approaches face significant limitations:

• Holistic Generation: Diffusion-based methods that generate multiple objects and layouts simultaneously often suffer from scalability issues, as memory and computation grow with the number of objects, making them difficult to apply to cluttered real-world scenes.
• Object-Centric Pipelines: Methods that generate objects independently and assemble them rely heavily on layout or pose modules trained on synthetic or domain-restricted datasets. These often fail to generalize to "in-the-wild" images with diverse viewpoints and compositions.
• Agent-Based Systems: While Large Language Model (LLM) and Vision-Language Model (VLM) agents offer planning capabilities, many rely on textual priors rather than image-grounded constraints. Image-conditioned multi-agent systems often employ coarse generator/evaluator roles that operate on the entire scene holistically. This necessitates repeated multi-turn interactions over the full context, increasing latency and limiting the precision of localized corrections.

Methodology: SceneConductor

The authors propose SceneConductor, a multi-agent orchestration framework that decomposes single-image 3D scene generation into three structured, sequential stages. This approach assigns specialized roles to agents, allowing them to operate only on relevant contexts rather than reasoning over the entire scene holistically.

1. Scene Initialization

This stage establishes a coarse 3D scene foundation:

• Mask Refinement: An agent processes segmentation masks (extracted via Grounded-SAM) to resolve redundancies, merge fragmented instances, and split masks covering multiple objects, ensuring a clean one-to-one mapping between masks and physical objects.
• 3D Reconstruction: Refined masks are used to reconstruct object meshes via SAM3D. Identical objects are reconstructed once and replicated to avoid redundancy.
• Layout Prediction: A geometry-aware layout predictor estimates the initial spatial arrangement of objects.

2. Environment Construction

This stage builds the environmental scaffold (room boundaries, surfaces, materials, illumination) to anchor the scene:

• Floor Plan Estimation: Using the initial scene and a predicted point map, the system operates in bird's-eye-view (BEV) space. It extracts valid 3D points, projects them to the horizontal plane, and computes a convex hull to define a coarse stage polygon.
• Scaffold Generation: An agent infers a simplified structure including a floor/support plane and boundary walls. This lightweight scaffold enforces physical support (preventing floating objects) and prevents wall penetrations.
• Contextualization: The agent analyzes the input image to assign wall materials matching the scene's dominant appearance and configures light sources for plausible illumination.

3. Multi-Agent Refinement

The final stage iteratively improves geometric consistency and visual realism through a hybrid of simple and complex operations:

• Planner Agent: Inspects the scene and routes issues to either simple deterministic operations or complex localized revisions.
• Simple Operations: The planner emits a deterministic list for tasks like enforcing physical support, aligning scales/rotations of similar objects, and correcting implausible poses.
• Complex Operations (Specialist Agents): For tightly coupled interactions, the planner isolates a sub-scene of interdependent objects. A specialist agent refines this sub-scene through multi-turn interaction, inspecting rendered images and segmentation to resolve spatial inconsistencies. The refined sub-scenes are then reintegrated into the global scene.

Core Component: Geometry-Aware Layout Predictor

A critical contribution is a layout predictor designed to provide reliable initialization without costly scene-level annotations.

• Training Strategy: Instead of learning a generative distribution from restricted 3D datasets, the model regresses layout parameters (Rotation, Translation, Scale, Floor Rotation) supervised by sparse geometric priors derived from segmentation masks and point maps.
• Architecture: The model encodes image, mask, and point-map features. Object meshes are voxelized and embedded into latent tokens. A transformer with global attention captures inter-object interactions.
• Pose Decomposition: The model predicts per-object poses $(R_i, T_i, S_i)$ in a canonical floor-aligned frame and a shared global floor rotation $F$. This decomposition explicitly separates global floor alignment from per-object pose, reducing degrees of freedom and enforcing a consistent height notion.
• Loss Function: A pointmap-supervised geometry loss combines a one-sided Chamfer distance (from point map to mesh) with a bounding-box alignment term, enabling training on large-scale 2D segmentation datasets without ground-truth 3D poses.

Key Contributions

1. Three-Stage Multi-Agent Framework: A structured pipeline (Initialization, Environment Construction, Refinement) that utilizes specialized agents to improve task coordination and reliability, avoiding the overhead of holistic scene reasoning.
2. Geometry-Aware Layout Predictor: A novel predictor that learns spatial information from real-world images using sparse geometric priors (segmentation and point maps), eliminating the need for scene-level layout annotations and improving generalization to diverse environments.
3. Performance Gains: Demonstrated consistent improvements over existing approaches in geometric accuracy, spatial consistency, and perceptual realism on benchmark datasets.

Experimental Results

The method was evaluated on 3D-FUTURE, ScanNet, and MIT-Indoor-67 datasets.

• Quantitative Metrics: On 3D-FUTURE and ScanNet, SceneConductor achieved the lowest Chamfer Distance (CD) and highest F-Score and IoU-B compared to baselines like 3D-Fixer, SceneGen, and SAM3D. For example, on 3D-FUTURE, it achieved a CD of 0.0089 (vs. 0.0117 for SAM3D) and an F-Score of 0.9261.
• Visual Metrics: Using VLM-based evaluators (Qwen3.5-2B) and CLIP scores, the method outperformed baselines in realism, functionality, layout consistency, and image alignment. On MIT-Indoor-67, it achieved the highest average score (4.2742) compared to SAM3D (4.0179) and VIGA (1.300).
• Ablation Studies: Experiments confirmed that the combination of geometry loss, floor rotation prediction, and training on segmentation data yields the best performance. Specifically, floor rotation stabilizes pose prediction by reducing rotational ambiguity.
• Qualitative Analysis: Visual comparisons show that SceneConductor produces more accurate object rotations and coherent ground-plane structures, particularly in scenes with tilted camera viewpoints where other methods generate unstable layouts.

Significance and Claims

The paper claims that SceneConductor offers a promising direction for scalable and controllable image-to-scene generation. By decomposing the generation process into structured stages and utilizing specialized agents, the framework makes the generation process easier to manage and control compared to holistic approaches.

The authors emphasize that their geometry-aware layout predictor is significant because it enables reliable initialization using only 2D segmentation and point-map data, bypassing the need for expensive scene-level annotations. This allows the system to generalize robustly to diverse real-world environments.

Limitations: The authors acknowledge that the framework relies on multiple foundation models executed sequentially, which introduces inference latency and the risk of error propagation from early stages. Current experiments focus on indoor scenes with texture-level modeling; extending the framework to outdoor/unbounded environments and richer material representations is noted as future work.