💻 computer science

BEMEval-Doc2Schema: Benchmarking Large Language Models for Structured Data Extraction in Building Energy Modeling

This paper introduces BEMEval-Doc2Schema, the first community-driven benchmark framework featuring the Key-Value Overlap Rate (KVOR) metric to systematically evaluate and compare large language models' performance in extracting structured data from building documentation for automated energy modeling.

Original authors: Yiyuan Jia, Xiaoqin Fu, Liang Zhang

Published 2026-02-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yiyuan Jia, Xiaoqin Fu, Liang Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching Robots to Read Blueprints

Imagine you are a master architect. You have a pile of messy, handwritten notes, old PDFs, and scattered Excel sheets describing a house. Your job is to turn that mess into a perfect, digital 3D model that a computer can use to calculate exactly how much energy the house will use.

Right now, humans have to do this translation manually. It's slow, boring, and prone to mistakes.

Enter Large Language Models (LLMs) like the "smart brains" behind ChatGPT or Gemini. The researchers in this paper asked: "Can we teach these AI brains to read those messy notes and automatically build the perfect digital model for us?"

The answer is: Yes, they can start to do it, but they aren't perfect yet. And to figure out how good they are, the authors built a "driving test" for AI.

1. The Problem: The "Lost in Translation" Issue

Building energy modeling is like trying to translate a novel written in a dialect of "Engineer" into a computer language.

The Input: Unstructured documents (PDFs, spreadsheets, drawings).
The Output: A strict, structured format (like a digital filing cabinet) that simulation software can read.

Currently, there is no standard way to grade how well an AI does this translation. It's like giving a student a test without an answer key. The researchers realized they needed a standardized "driving test" to see if the AI is actually safe to drive on the road of construction.

2. The Solution: "BEMEval" (The AI Driving School)

The authors created a framework called BEMEval. Think of this as a Driving School for AI.

The First Test (BEMEval-Doc2Schema): The first test in the school is called "Doc2Schema." It's a simple task: Take a messy document and turn it into a neat, structured list.
The Test Track: They used three famous "practice cars" (real building data sets):
1. L100: A standard, basic house (like a Toyota Corolla).
2. NZERTF: A super-high-tech, net-zero energy house (like a Tesla).
3. iUnit: A tiny, modular apartment (like a smart camper).
The Answer Keys: They used two different "rulebooks" (schemas) to see how the AI organized the data:
- HPXML: A very detailed, complex rulebook (like a 500-page manual for a luxury car).
- EPC: A simpler, flatter rulebook (like a quick checklist).

3. The Grading System: The "KVOR" Score

How do you grade an AI? You can't just ask, "Did you get it right?" because the AI might get the right number but put it in the wrong box.

The authors invented a metric called KVOR (Key-Value Overlap Rate).

The Analogy: Imagine a game of "Simon Says." The teacher (the ground truth) says, "Put the Red ball in the Left box."
If the AI puts the Red ball in the Left box, it gets a point.
If it puts the Red ball in the Right box, or the Blue ball in the Left box, it gets zero points.
The KVOR score is simply the percentage of balls the AI put in the correct boxes. A score of 1.0 is perfect; 0.0 is a total disaster.

4. The Race: Who Won?

The researchers pitted two AI giants against each other: GPT-5 (from OpenAI) and Gemini 2.5 (from Google). They tested them in two ways:

Zero-Shot: "Here is the task. Go!" (No help).
Few-Shot: "Here is the task, and here are three examples of how to do it perfectly. Now you try."

The Results:

Gemini 2.5 won. It consistently got higher scores than GPT-5. It was better at understanding the rules and keeping the data organized.
The "Few-Shot" Boost: When the AI was given examples to learn from, both models got much better. It's like giving a student a practice quiz before the real exam.
The Difficulty Factor: The AI struggled more with the complex "HPXML" rulebook (the luxury car manual) than the simple "EPC" checklist. The more complex the rules, the more the AI got confused.
The "Hallucination" Trap: Sometimes, the AI would guess a number that wasn't in the text just to fill a blank. The test penalized this heavily.

5. The Twist: The AI is Cheating (Sort Of)

The researchers noticed something funny. When asked to extract data, the AI often tried to write a computer program to do the job for itself, rather than just reading the text and writing the answer.

The Metaphor: Imagine you ask a robot to "Sort these 100 socks by color." Instead of picking them up and sorting them, the robot starts writing code to build a sock-sorting machine.
While clever, this shows the AI is trying to find a shortcut rather than truly understanding the meaning of the words.

6. Why This Matters

This paper isn't just about one test; it's about building a community standard.

Before this, everyone was making up their own tests. Now, we have a shared "driving school" where we can all see which AI is actually ready for the road.
It shows us that AI is getting good at simple tasks but still needs help (like examples) for complex ones.
It suggests that maybe we need to design our "rulebooks" (schemas) to be simpler and more "AI-friendly" to help the robots do their jobs better.

The Bottom Line

The paper says: "We built a scoreboard to measure how well AI can turn messy building documents into clean data. The AI is getting better, especially when we give it examples, but it still makes mistakes on complex tasks. We need to keep testing to make sure it's safe to use in real construction projects."

It's the first step toward a future where you can upload a PDF of your house plans, and an AI instantly builds the energy model for you, saving engineers hours of tedious work.

1. Problem Statement

Building Energy Modeling (BEM) is critical for decarbonization, yet the process of converting unstructured building documentation (e.g., PDFs, audit reports, drawings) into structured, simulation-ready data remains a labor-intensive, manual bottleneck.

The Gap: While Large Language Models (LLMs) show promise in automating this extraction, their application in BEM lacks rigorous evaluation. Current research is fragmented, lacking:
- Publicly available, task-specific datasets.
- Standardized performance metrics.
- Reproducible benchmarks to compare different models and prompting strategies.
The Challenge: Engineering applications require high precision and repeatability. LLMs are prone to "hallucinations" and semantic errors, making it difficult to trust them for critical tasks like mapping building parameters to standardized schemas (e.g., HPXML, EPC) without a systematic way to quantify accuracy.

2. Methodology

The authors introduce BEMEval, a modular benchmark framework designed to evaluate foundation models across distinct BEM tasks. The paper focuses on its inaugural benchmark: BEMEval-Doc2Schema.

A. Framework Components

Task: Converting unstructured residential building descriptions into structured, schema-compliant data (HPXML and EPC formats).
Datasets (Inputs): Three representative residential building cases were selected to ensure diversity:
1. HERS L100: A standardized base building for testing simulation software credibility.
2. NIST NZERTF: A highly instrumented, net-zero energy single-family home with complex HVAC and renewable systems.
3. NREL iUnit: A modular, factory-built micro-apartment representing multifamily typologies.
Ground Truth (Outputs): Structured datasets in two distinct schemas:
- HPXML: A hierarchical, industry-standard XML format for residential energy auditing and retrofit (complex, deep hierarchy).
- EPC Schema: A flat, key-value TOML format based on normative calculation standards (simpler, reduced hierarchy).
Evaluation Metric: Key–Value Overlap Rate (KVOR).
- Formula: $KVOR = \frac{\text{Number of correctly matched key-value pairs}}{\text{Total number of keys in reference schema}}$
- This metric penalizes both missing keys and incorrect value assignments, providing a quantitative measure of structural and semantic accuracy.

B. Experimental Setup

Models Evaluated:
- GPT-5 (OpenAI, 2025 release): Focused on enhanced reasoning and structured data generation.
- Gemini 2.5 (Google DeepMind, 2025 release): Integrated multimodal understanding and symbolic reasoning.
Prompting Strategies:
- Zero-Shot: Task instructions and schema documentation only.
- Few-Shot: Task instructions plus $N$ examples of input-output pairs to guide mapping behavior.

3. Key Results

The study evaluated the models across the three datasets and two schemas.

Model Performance:
- Gemini 2.5 consistently outperformed GPT-5 across all datasets and prompting strategies.
- Best Result: Gemini 2.5 achieved a KVOR of 0.851 on the iUnit (EPC) dataset using zero-shot prompting.
- Worst Result: GPT-5 achieved a KVOR of 0.000 on the L100 (HPXML) dataset using zero-shot prompting.
Impact of Prompting:
- Few-shot prompting significantly improved accuracy for both models. For example, Gemini's performance on L100 (HPXML) doubled from 0.040 (zero-shot) to 0.082 (few-shot).
Schema Complexity:
- EPC (Flat Schema) yielded significantly higher KVOR scores than HPXML (Hierarchical Schema).
- The deep hierarchy and complex cross-references in HPXML posed greater challenges, leading to lower extraction accuracy.
Building Complexity:
- The NZERTF case (highly complex systems) yielded the lowest scores among HPXML cases, indicating that system interdependencies and high parameter density degrade model performance.

4. Key Contributions

BEMEval Framework: The proposal of the first open, community-driven benchmark framework specifically for evaluating LLMs in Building Energy Modeling.
BEMEval-Doc2Schema: The implementation of the first specific benchmark for structured data extraction, featuring:
- Curated datasets from real-world test facilities (L100, NZERTF, iUnit).
- Ground truth references in industry-standard schemas (HPXML, EPC).
- A novel, interpretable metric (KVOR) for quantifying extraction accuracy.
Open Source Ecosystem: All resources (datasets, schema definitions, evaluation scripts, and documentation) are publicly available on GitHub to ensure reproducibility and encourage community extension.
Empirical Insights: The study provides the first quantitative evidence that:
- Schema design (flat vs. hierarchical) drastically impacts LLM performance.
- Few-shot prompting is essential for domain-specific tasks.
- Current general-purpose LLMs struggle with complex, hierarchical engineering schemas without specialized fine-tuning.

5. Significance and Future Work

Significance: This work bridges the gap between AI research and building simulation practice. By establishing a standardized "ImageNet for BEM," it enables fair comparison of tools, tracks progress, and identifies failure modes (e.g., hallucinations in complex schemas). It suggests that "AI-friendly" schema design (simpler, flatter structures) may be necessary to accelerate automation in the near term.
Limitations:
- The dataset size is currently small (3 buildings).
- Models were not domain-fine-tuned (only general-purpose LLMs were used).
- The KVOR metric treats all keys equally, ignoring the varying impact of parameters on simulation outcomes.
Future Directions:
- Expanding the dataset to include diverse building types, vintages, and commercial structures.
- Developing domain-specific fine-tuned models.
- Introducing weighted metrics that prioritize critical simulation parameters.
- Integrating robustness testing (e.g., measuring hallucination rates and cost-latency tradeoffs).

In conclusion, BEMEval-Doc2Schema establishes a foundational step toward reliable, automated BEM workflows, demonstrating that while LLMs are capable of structured data extraction, their success is heavily dependent on schema complexity and the use of few-shot guidance.