How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning

1. Problem Statement

Current Vision-Language Models (VLMs) are evaluated primarily on perceptual realism (visual plausibility) rather than physical validity. While models can generate 3D layouts that look correct, they often fail to adhere to the rigorous structural, procedural, and code-compliance constraints required to actually build an artifact.

• The Gap: Existing benchmarks (e.g., PhysBench, VSI-Bench) test passive comprehension or spatial recall but do not evaluate the ability to synthesize artifacts from scratch under strict engineering constraints.
• The Challenge: Generating a physically realizable structure requires understanding load paths, span limits, connection hierarchies, and building codes (e.g., IRC), which are often invisible in exterior renders. Current models treat the world as a surface rather than a system of constraints.

2. Methodology: The DreamHouse Benchmark

The authors introduce DreamHouse, a benchmark grounded in residential timber-frame construction, a domain with fully codified engineering standards (International Residential Code - IRC) and objectively verifiable correctness.

A. Dataset Construction

• Scale: 26,543 structurally verified timber-frame structures.
• Variety: 13 distinct architectural styles (e.g., A-Frame, Barn, Split-Level, Colonial) ranging from 133 to 1,548 structural members.
• Granularity: All structures meet Level of Detail 350 (LoD 350), the industry standard for fabrication-grade models, specifying member species, cross-sections, and connections.
• Input/Output: Models receive multi-view rendered images of a target structure and must generate Blender Python code to construct the frame.

B. Structural Validation Suite

Instead of relying on expensive physics simulations, the benchmark uses a deterministic, simulation-free validation suite operating directly on the scene graph. It consists of 10 tests across four pillars:

1. IRC Compliance: Load path connectivity, span limits, and member sizing.
2. Physics: Deflection limits (L/360), cantilever ratios, and point load bearing.
3. Geometry: Gap detection, roof coverage, and dual-end restraints.
4. Fabrication: Standard lumber dimensions and assembly ordering.

• Pass Condition: A structure must pass all 10 tests to be considered structurally valid.

C. Evaluation Protocols

The benchmark evaluates models through an iterative agentic loop where the model observes the state, generates code, and receives structured feedback on failures. Three protocols test different levels of scaffolding:

1. Planner-Atomic (TS): One-shot generation of the entire structure.
2. Planner-Reactive (TQ): Self-decomposed sequential generation; failure requires regenerating the entire plan.
3. Planner-Managed (TW): Step-by-step generation with external phase management; failure only resets the current step.

• Input Conditions: Models are tested on Frame (bare timber visible) and Facade (framing occluded by cladding, requiring inference of hidden structure).

D. Metrics

• Structural Validity: Binary pass/fail on all 10 tests.
• Visual Fidelity: Alpha-weighted Mean Squared Error (MSE) comparing rendered outputs to targets.
• Topological Fidelity: Measures member count accuracy, spatial alignment (Hungarian matching), and volumetric overlap.
• Joint Pass Rate: The fraction of structures satisfying both structural validity and visual fidelity.

3. Key Contributions

1. New Evaluation Axis: Establishes Physical Generative Reasoning as a distinct capability orthogonal to visual perception and comprehension.
2. DreamHouse Dataset: A large-scale, verified dataset of 26k+ LoD 350 timber structures with multi-view renderings and construction annotations.
3. Deterministic Validation: A 10-test suite that verifies engineering compliance without physics simulation, enabling fast, interpretable evaluation.
4. Agentic Framework: A protocol-based evaluation showing that scaffold design (how the task is structured) is as critical as model selection.

4. Experimental Results

The authors evaluated three frontier VLMs: GPT-5, Gemini 3 Flash, and Claude Opus 4.5.

• Low Joint Pass Rates: Even the best models achieved a joint pass rate of only 7.1%. Most models fail to satisfy structural and visual constraints simultaneously.
• Orthogonality of Skills: Structural validity and visual fidelity are uncorrelated.
  • GPT-5 excelled at structural validity in one-shot tasks (79.2%) but failed visually.
  • Claude performed well visually but struggled with structural constraints in certain protocols.
  • Gemini showed the highest structural pass rate (78.5%) under the Planner-Managed protocol.
• Protocol Dominance: The choice of generation protocol significantly impacts performance. For example, Gemini's structural pass rate jumped from 45.4% (Atomic) to 78.5% (Managed), overtaking GPT-5. This suggests that stepwise decomposition is crucial for physical reasoning.
• Inference Capability: Under the Facade condition (hidden structure), Gemini and Claude improved their structural performance, suggesting they can infer hidden load paths from appearance. GPT-5's performance dropped, indicating reliance on direct visual mimicry.
• Iterative Refinement: Visual feedback helps improve visual fidelity but does not guarantee structural recovery. Models often fail to correct structural errors even when shown side-by-side comparisons.
• Open-Source Models: Current open-source models (e.g., Qwen3-VL) failed to produce valid code or structures, achieving 0% pass rates, highlighting that this capability is currently frontier-only.

5. Significance and Conclusion

• Beyond Visuals: The paper demonstrates that "looking right" is insufficient for AI in the physical world. True intelligence in construction requires understanding load paths, connectivity, and code compliance.
• Limitations of Current Benchmarks: Standard leaderboards based on visual similarity or static reasoning are poor proxies for physical generative reasoning. A model can look perfect but be structurally unsound (e.g., missing load paths, over-spanned beams).
• Future Direction: The findings suggest that physical generative reasoning is a distinct, underdeveloped capability. Future models need dedicated training on structural grammar and iterative self-correction mechanisms, rather than just scaling up general multimodal pre-training.
• Impact: DreamHouse provides a concrete, verifiable target for the AEC (Architecture, Engineering, Construction) industry and AI research to track progress toward automating design-to-construction pipelines.

In summary, DreamHouse reveals a critical gap in current AI: while models are excellent at perceiving the world, they are currently incapable of constructing it in a physically valid way. The benchmark serves as a rigorous stress test for the next generation of multimodal agents.