Technical Summary: Perceive-then-Plan: Layout-as-Policy for Monocular 3D Scene Layout Estimation

Problem Statement

Recovering structured 3D indoor scene layouts from a single monocular image is a fundamental challenge in computer vision, with applications in robotics, simulation, and content creation. Existing approaches typically formulate this as a direct prediction problem, where models regress object locations, sizes, and orientations in a single forward pass. However, these methods often struggle to enforce global constraints and inter-object relationships. Because 3D indoor scenes are inherently structured and interdependent, small local errors in one-shot predictions can propagate into large global inconsistencies, leading to physically implausible layouts (e.g., floating objects, collisions, or invalid support relations). The core difficulty lies in reconciling ambiguous visual observations with physical and spatial constraints without explicit reward engineering or handcrafted objective functions.

Methodology

The authors propose a Perceive-then-Plan framework that reframes monocular 3D layout estimation as a sequential decision-making problem using Vision-Language Models (VLMs). The pipeline consists of two distinct stages:

1. The Perceiver (Initialization)

The first stage focuses on grounding 3D objects from the input image and 2D detections.

• Architecture: A geometry-enhanced VLM (based on Qwen3-VL) serves as the Perceiver.
• Geometry-Aware Modulation: To bridge the gap between visual features and 3D geometry, the model injects geometric priors from a pretrained geometry encoder (VGGT) into the VLM's visual features via a lightweight modulation module. This allows the model to leverage visual understanding while being guided by geometric cues.
• Representation: Instead of regressing raw 3D vertices, the Perceiver outputs a parameterized representation of 3D boxes (center, scale, and local axes) in a camera coordinate system.
• Canonicalization: The grounded 3D boxes are transformed into a gravity-aligned, grid-based canonical representation. This discretization (rounding positions and orientations to grid indices) prepares the scene state for the subsequent action-based refinement.

2. The LaP Planner (Refinement)

The second stage, termed Layout-as-Policy (LaP), treats the 3D layout as a structured state and refinement as a policy learning problem.

• Action Space: The planner operates on a discrete set of actions: SELECT (choose an object), MOVE (translate), ROTATE_Y (yaw), RESIZE (scale), and STOP. These actions are designed to correct specific failure modes like misalignment, incorrect support, and collisions.
• Training Strategy:
  • Perturbed Data Synthesis: Training data is constructed by taking ground-truth layouts, applying random perturbations (simulating initialization errors), and generating the inverse action sequences required to restore the ground truth.
  • Supervised Fine-Tuning (SFT): The planner is first warm-started with SFT on these perturbed data pairs to learn the instruction-following format and basic corrective behaviors.
  • Direct Preference Optimization (DPO): To learn implicit notions of geometric validity without explicit reward modeling, the authors employ DPO. The model is trained to prefer action sequences that lead to physically coherent layouts over "degraded" sequences (e.g., those with wrong action types, missing steps, or incorrect magnitudes).
• Iterative Process: The refinement is performed iteratively. The planner takes the current scene state and the input image, predicts a sequence of actions, applies them to update the state, and repeats until convergence (signaled by a STOP token or a round limit).

3. Scene Assembly

Post-refinement, a contact scene graph is generated to group objects into relational bundles (e.g., a lamp on a table on the floor). A single-pass gravity simulation is applied to these bundles to resolve minor floating or penetration artifacts caused by the mismatch between abstract 3D boxes and actual 3D assets.

Key Contributions

1. Perceive-then-Plan Pipeline: The authors introduce a novel formulation for monocular 3D layout estimation that decomposes the task into a perception stage (grounding) and a planning stage (iterative refinement), moving away from one-shot prediction.
2. Layout-as-Policy (LaP): They propose casting the layout refinement stage as a policy learning problem where 3D layouts are structured states and refinement operations are discrete actions. This allows for explicit reasoning about constraint satisfaction.
3. Geometry-Enhanced Perceiver: A VLM-based grounding module that integrates visual and geometric features to produce reliable, observation-aligned initial layouts, providing a strong starting point for refinement.
4. Preference-Based Optimization: The use of DPO allows the planner to learn corrective behaviors and understand spatial coherence through relative comparisons of action sequences, avoiding the need for complex reward engineering.

Experimental Results

The method was evaluated on the Hypersim and SUN RGB-D datasets.

• Perceiver Performance: The proposed Perceiver significantly outperforms state-of-the-art open-source and closed-source VLMs (including GPT-4o and Gemini) in 3D grounding metrics. Specifically, it achieves a Reprojection IoU of 0.58 (vs. 0.33 for the next best open-source model) and an Average Depth Error of 0.08m.
• Refinement Performance: Iterative refinement with the LaP Planner yields substantial improvements over the initial Perceiver output.
  • Physical Plausibility: The method reduces the Support Violation Rate (SVR) from 11.63% to 4.85% and the collision count from 3.62 to 1.46.
  • Visual Alignment: The Reprojection IoU improves from 0.54 to 0.65, demonstrating that refinement enhances physical consistency without drifting from visual evidence.
• Comparisons: The approach outperforms one-shot VLM direct refinement and rule-based refinement baselines. Rule-based methods fail to maintain visual alignment, while one-shot VLMs struggle to correct intermediate errors effectively.

Significance and Claims

The paper claims that transforming layout estimation from a one-shot prediction task into an iterative perceive-then-plan process enables better handling of global constraints and complex object interactions. By treating layout refinement as a policy learning problem, the framework naturally supports downstream tasks such as language-guided scene editing and embodied manipulation, as the action-based interface allows for direct modification of the scene state.

The authors emphasize that their approach produces layouts that are more physically coherent and better aligned with visual observations. They position this work as a step toward enabling Vision-Language Models to perform robust 3D reasoning in real-world scenarios, particularly in domains requiring physically valid scene understanding like robotics and simulation. The paper acknowledges limitations, noting its reliance on off-the-shelf 2D detectors and its current focus on structured indoor environments, suggesting future work in multi-view inputs and joint optimization of the perception and planning stages.