Synthetic Homes: An Accessible Multimodal Pipeline for Producing Residential Building Data with Generative AI

This paper introduces a modular, multimodal framework that leverages generative AI to synthesize realistic residential building data from public images and information, thereby overcoming data accessibility and privacy barriers to advance energy modeling and machine learning research.

The Gist

Imagine you are an architect trying to design the perfect, energy-efficient neighborhood. You want to know exactly how much electricity every house will use, where the insulation is weak, and how to upgrade them to save money and help the planet.

The problem? To do this, you need a massive amount of data about real houses: blueprints, photos, material lists, and energy bills. But getting this data is like trying to find a needle in a haystack that is also locked in a vault. It's expensive, hard to find, and often private (nobody wants strangers knowing their home details).

This paper introduces a "Digital Twin Factory" that solves this problem.

Here is how the authors built a machine that creates fake but realistic houses using Artificial Intelligence, so researchers can study them without needing real-world data.

The Recipe: A Four-Step Kitchen

Think of the researchers' pipeline as a high-tech kitchen where they cook up a batch of "Synthetic Homes." They use four main ingredients (or steps):

1. The Scavenger Hunt (Data Collection)
Instead of knocking on doors, the team built a digital robot (a web scraper) that visits public county websites. It grabs basic facts about houses that are already public, like "3 bedrooms," "2,000 square feet," and "built in 1990." It also downloads two photos: a street view of the house and a floor plan.

• Analogy: It's like a detective gathering public records and photos of a neighborhood before the real investigation begins.

2. The Art Critic (Image Processing)
The robot takes those photos and shows them to an AI eye named LLaVA. This AI isn't just looking; it's analyzing. It looks at the roof and says, "That roof looks old and needs tar," or "The windows look modern."

• The Twist: The authors tested two different AI eyes (GPT and LLaVA). They found that GPT was like a distracted tourist who looked at the grass, the trees, and the roof equally. LLaVA, however, was like a focused inspector who ignored the trees and stared intensely at the roof. They chose LLaVA because it actually understood what mattered.

3. The Storyteller (Generating the Blueprint)
Now, the team feeds the "facts" from the scavenger hunt and the "observations" from the Art Critic into a powerful language AI (GPT).

• The AI acts as a translator and a writer. It takes the visual clues and the raw numbers and writes two things:
  1. A GeoJSON file: A digital map and blueprint of the house, including made-up but realistic numbers for how well the walls keep heat in (R-values) and how efficient the air conditioner is.
  2. An Inspection Note: A paragraph written as if a real human inspector walked through the house, noting things like, "The attic insulation looks thin," or "The furnace is from 2010."

4. The Simulator (The Energy Test)
Finally, they take this digital blueprint and feed it into EnergyPlus, a super-complex physics engine used by engineers. This engine runs a simulation: "If this house has these walls and this roof, how much energy will it use in a hot summer?"

• The Result: They get a complete dataset: a house with a photo, a description, a blueprint, and a simulated energy bill.

Why This is a Big Deal

The authors didn't just make up random numbers; they checked their work. They compared their "fake" houses to a massive database of real US homes (called ResStock).

• The Verdict: The fake houses were shockingly similar to the real ones. The "fake" insulation values and energy costs fell right within the normal range of real houses.

The "Magic" of This Approach

Usually, when people use AI to generate data, the AI might "hallucinate" (make up crazy, impossible things). The authors proved that by using a specific type of AI (LLaVA) for the image part and strict rules for the writing part, they avoided these mistakes.

In simple terms:
They built a video game engine for real estate. Instead of playing a game, researchers can now use this engine to generate thousands of realistic, privacy-safe houses. They can test new energy policies, see how solar panels would work on a whole city, or figure out the best way to upgrade old buildings—all without ever needing to invade anyone's privacy or pay for expensive data.

It turns the difficult task of "finding data" into the easy task of "generating data," opening the door for anyone to do better research on how to save energy.

Technical Summary

Here is a detailed technical summary of the paper "Synthetic Homes: An Accessible Multimodal Pipeline for Producing Residential Building Data with Generative AI."

1. Problem Statement

Computational models for Building Energy Modeling (BEM) and Urban Building Energy Modeling (UBEM) are essential for energy policy planning, infrastructure optimization, and climate impact mitigation. However, these models rely on vast amounts of granular data (e.g., material quality, floor plans, HVAC specifications) that are often:

• Inaccessible: Not publicly available or restricted by privacy laws.
• Expensive: High costs associated with acquiring detailed datasets.
• Privacy-concerning: Sensitive information regarding specific residential properties.

Existing methods for generating synthetic data often rely on traditional simulation tools or sensor data, which lack scalability. While Generative AI (GenAI) offers new opportunities, prior work has raised concerns regarding hallucinations (generating factually incorrect data), lack of domain expertise, and poor focus in image processing (e.g., models attending to irrelevant background features rather than the building structure).

2. Methodology

The authors propose a modular, multimodal pipeline that synthesizes realistic residential building data by combining publicly available tabular data, street-view images, and Generative AI. The pipeline consists of four distinct stages:

A. Data Collection (Web Scraping)

• Source: Public county datasets (specifically from a location withheld for review).
• Tools: Selenium WebDriver and ChromeDriver.
• Output: For each property, the scraper extracts metadata (e.g., year built, square footage, number of rooms, heating type) and downloads two images: a street view photograph and a floor plan.

B. Image Processing (Visual Analysis)

• Model: LLaVA (Large Language-and-Vision Assistant).
• Function: Converts visual inputs (street view and floor plans) into textual descriptions.
  • Floor plans define the geometric structure.
  • Street views determine window quality, roof condition, and exterior materials.
• Model Selection & Validation (Occlusion Testing): The authors compared LLaVA against GPT (GPT-4V) using forward occlusion testing.
  • Method: Images were divided into cells; individual cells were masked (blacked out) to measure the semantic change in the model's response.
  • Finding: LLaVA demonstrated superior focus, showing significantly higher sensitivity to roof-related cells (the target of inspection) compared to non-roof cells. GPT exhibited stochastic behavior, reacting equally to irrelevant background elements (trees, grass). This validated LLaVA as the superior choice for extracting relevant building features.

C. Structured Data Generation (GeoJSON & Inspection Notes)

• Model: GPT (OpenAI).
• Input: The textual descriptions from LLaVA combined with the scraped tabular data.
• Task:
  1. Generate a GeoJSON file containing the building footprint (geometry), scraped metadata, and five estimated performance parameters (HVAC heating/cooling COP, wall/roof R-values, air change rate).
  2. Generate a home inspection note summarizing energy-related observations (insulation, HVAC age, visible upgrades).
• Constraint: Strict prompts ensure valid JSON output and prevent hallucinations by grounding the generation in provided data.

D. Simulation (EnergyPlus)

• Process: The generated GeoJSON is converted into an Input Definition File (IDF) for EnergyPlus, a standard building energy simulation engine.
• Mechanism: A template IDF is populated with variables from the GeoJSON (geometry, R-values, COPs). The ExpandObjects utility expands these templates into full systems, and EnergyPlus runs the simulation to produce energy consumption results.
• Output: A unified JSONL dataset containing the original data, inspection notes, and simulation results.

3. Key Contributions

1. Novel Multimodal Pipeline: A flexible framework that integrates web scraping, vision-language models (LLaVA), text-generation models (GPT), and physics-based simulation (EnergyPlus) to create end-to-end synthetic building datasets.
2. Empirical Validation of Visual Focus: The paper provides rigorous evidence (via occlusion testing) that LLaVA is better suited than GPT for architectural inspection tasks due to its early-fusion architecture, which allows for better attention to specific building features (like roofs) compared to late-fusion models.
3. Cost-Effective Scalability: The pipeline generates data at a negligible cost (approx. $0.0014 per home for API costs, excluding compute), enabling the creation of large-scale datasets for regions where data is scarce.
4. Realism Verification: The authors statistically validate the synthetic data against ResStock (a trusted US residential dataset), demonstrating high overlap in key energy parameters.

4. Results

The authors evaluated the pipeline on 258 synthetic homes in Climate Zone 4A.

• Realism Metrics: The synthetic data was compared against the 10–90% range of ResStock data.
  • Wall R-Value: 93.41% overlap.
  • Roof R-Value: 100.0% overlap.
  • Heating COP: 100.0% overlap.
  • Cooling COP: 66.28% overlap.
• Statistical Distribution: The mean and median values of the synthetic variables closely matched the interquartile ranges (IQR) of the real-world ResStock data.
• Model Performance: The occlusion tests confirmed that LLaVA focuses on relevant image features (roof) with a mean difference ~20% higher than non-relevant areas, whereas GPT showed no such distinction.

5. Significance and Future Directions

• Democratizing Energy Research: By removing barriers related to cost, availability, and privacy, this framework allows researchers to train and validate ML models for energy efficiency without relying on restricted datasets.
• Mitigating AI Hallucinations: The study demonstrates that combining GenAI with physics-based simulation and strict prompting can produce reliable, domain-specific data, addressing concerns about the unreliability of LLMs in scientific contexts.
• Future Applications:
  • Training ML algorithms to predict energy efficiency retrofits.
  • Scaling to commercial buildings or different geographic regions.
  • Using the generated "labeled" homes (with simulation results) to train models that recommend specific energy upgrades for real-world properties.

In conclusion, "Synthetic Homes" presents a robust, low-cost methodology for generating high-fidelity, multimodal building data, bridging the gap between generative AI capabilities and the rigorous requirements of urban energy modeling.