Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements

Technical Summary: Ishigaki-IDS-Bench

Problem Statement
While Large Language Models (LLMs) are increasingly capable of generating structured outputs like JSON, SQL, and code, public resources for evaluating generation that must simultaneously satisfy industry-standard XML schemas and strict domain vocabulary constraints remain limited. In the Architecture, Engineering, and Construction (AEC) domain, Building Information Modeling (BIM) relies on the Industry Foundation Classes (IFC) standard for data representation. The Information Delivery Specification (IDS) is an XML-based standard used to define machine-readable information requirements for these IFC models.

Generating IDS from natural language requirements or tabular checklists is a complex constrained structured generation task. It requires not only syntactic validity but also:

1. Standard Conformance: Adherence to the IDS schema and standard.
2. Vocabulary Mapping: Correctly mapping natural language terms (e.g., "walls") to specific IFC standard classes.
3. Constraint Expression: Accurately defining value constraints (e.g., fire ratings) within the IDS checking units.
4. Semantic Correspondence: Ensuring the generated XML reflects the intent of the input document without ambiguity.

Current evaluation methods for general-purpose structured generation do not adequately address these domain-specific layers of complexity, nor do they provide a framework for analyzing failures in formal validity versus content accuracy.

Methodology
The paper introduces Ishigaki-IDS-Bench, a benchmark designed to evaluate the ability of LLMs to generate IDS XML from BIM information requirements.

• Dataset Construction: The benchmark consists of 166 examples derived from 83 practical scenarios, expanded into both Japanese and English. These examples were authored and verified by BIM/IDS experts. The dataset includes input contexts (CSV or natural language), target IFC versions, and corresponding "gold" IDS files.

  • The construction process involved brainstorming use cases, manual creation of inputs and gold outputs, expert review for semantic correspondence, and formal validation using the IDSAuditTool.
  • Metadata is provided for input format, language, turn setting (single vs. multi-turn), IFC version, and construction domain.
  • The dataset focuses on three core facets: entity (IFC element types), attribute (basic IFC fields), and property (additional information with property sets, base names, values, data types, and cardinality).

• Two-Stage Evaluation Protocol:

  1. Formal Validity (Stage 1): Uses IDSAuditTool to assess whether the generated output is processable and conforms to standards. It reports three metrics:
     • Processability: Whether the audit reached a completed state (ok, structure-error, or content-error).
     • Structure: Conformance to XML structure and IDS XSD schema.
     • Content: Conformance to the IDS standard and IFC vocabulary constraints.
  2. Content Agreement (Stage 2): A facet scorer compares the generated IDS against the gold IDS. It normalizes namespaces, whitespace, and capitalization, then performs exact matching on comparison blocks (applicability/requirements, facet types, and constraints). This stage calculates Precision, Recall, and Macro F1 scores, capturing failures where the XML is valid but semantically incorrect relative to the input.

• Baseline Experiments: The authors evaluated 10 LLMs (including closed proprietary models like GPT-5.5 and open-weight models like Qwen3 and Kimi) in a zero-shot setting. The prompts included task instructions and input requirements but no few-shot examples or gold IDS references.

Key Contributions

1. Ishigaki-IDS-Bench Dataset: A high-quality, expert-verified benchmark containing 166 examples with gold IDS files and multifaceted metadata, addressing the lack of public resources for domain-standard XML generation in AEC.
2. Two-Stage Evaluation Framework: A protocol that combines formal validity checks (via IDSAuditTool) with facet-level content agreement scoring, enabling a nuanced analysis of where generation fails (syntax vs. semantics).
3. Zero-Shot Baselines and Analysis: Comprehensive evaluation of 10 LLMs, providing a diagnostic view of current model capabilities and failure modes in generating IDS.

Results
The evaluation of 10 LLMs revealed significant challenges in stable IDS generation:

• Performance Gap: The best-performing model, GPT-5.5, achieved a 65.6% Macro F1 for content agreement but only a 27.7% pass rate for the Content audit (conformance to IDS standard and IFC vocabularies).
• Processability vs. Content: Many models exhibited high Processability rates (e.g., 99.4% for Kimi K2.6) but low Content pass rates, indicating they can generate syntactically processable XML but struggle to adhere to domain-specific constraints.
• Model Scaling: Among open-weight Qwen models, increasing parameter size improved F1 scores (from 20.2% for 8B to 26.8% for 397B) but did not significantly improve Content pass rates (remaining between 10.2% and 19.3%), suggesting scale alone is insufficient for mastering domain standards.
• Input and Setting Effects:
  • Multi-turn settings (updating existing IDS) yielded significantly higher performance (84.8% F1 for GPT-5.5) compared to single-turn generation from scratch (58.6% F1).
  • CSV inputs resulted in higher F1 scores than natural language inputs, likely due to clearer mapping of requirements to facets.
• Failure Modes: Common errors included formal violations of XSD/IFC constraints, incorrect property set names, and ambiguity in mapping natural language targets to value constraints.

Significance
The paper positions Ishigaki-IDS-Bench as a diagnostic resource rather than a large-scale training corpus. Its significance lies in:

• Highlighting that current LLMs can express partial information requirements as IDS but struggle to stably generate outputs that satisfy the rigorous constraints of the IDS standard and IFC vocabulary.
• Providing a framework for comparative evaluation and failure analysis that can guide the development of constrained structured generation methods conforming to domain standards.
• Offering a template for evaluating other domain-specific structured generation tasks (e.g., medical HL7/FHIR, financial XBRL) that require standard conformance and external validation.

The authors explicitly state that the benchmark is released under the CC BY 4.0 license on GitHub and Hugging Face, with evaluation scripts and reproducibility documentation provided. They note limitations, such as the exclusion of certain facets (classification, material) and the reliance on zero-shot evaluation, framing future work around extending these scopes and exploring few-shot or fine-tuned approaches.