Engineering Reasoning and Instruction (ERI) Benchmark: A Large Taxonomy-driven Dataset for Foundation Models and Agents

1. Problem Statement

Current Large Language Model (LLM) benchmarks are insufficient for evaluating engineering capabilities due to several critical gaps:

• Lack of Domain Specificity: General benchmarks (e.g., MMLU, BIG-bench) focus on broad knowledge or multiple-choice formats, failing to capture the rigorous, constraint-aware reasoning required in engineering.
• Absence of Structural Rigor: Existing datasets often lack explicit output contracts (e.g., unit consistency, verification steps) and do not distinguish between "plausible narration" and "genuine reasoning."
• Inadequate Coverage: Engineering workflows require a controlled cross-product of domains (fields), task intents (what the model must do), and difficulty tiers (rigor levels). Existing benchmarks often aggregate scores, masking systematic weaknesses in specific subdomains or task types.
• Circularity Risks: Many synthetic benchmarks suffer from "circularity," where the same model generates and evaluates data, leading to inflated performance metrics and unverified hallucinations.

2. Methodology

The authors developed the ERI Benchmark, a taxonomy-driven instruction dataset designed to train and evaluate engineering-capable LLMs and agents.

A. Dataset Construction & Taxonomy

• Scale: 57,750 instruction–response records across 1,155 unique cells.
• Taxonomy Dimensions:
  1. Fields (9): Civil, Mechanical, Electrical, Chemical, Environmental, Aerospace, Materials, Fire Science, and Industrial Engineering.
  2. Subdomains (55): Specific topics within each field (e.g., Structural Analysis, Thermodynamics, Fluid Mechanics).
  3. Intent Types (7): Definition, Explanation, Calculation, Comparison, Design/Synthesis, Troubleshooting, and Code/Standard.
  4. Difficulty Tiers (3): Undergraduate, Graduate, and Professional.
• Generation Strategy: Records were generated using a seed-based approach where each cell (Field × Subdomain × Intent × Difficulty) contains 50 samples. The generation was constrained by a strict JSON schema to ensure machine-checkable outputs.
• Validation Protocol: To address circularity and hallucination risks, the authors employed a Convergent Validation Protocol:
  • Cross-Provider Independence: The generator (GPT-5 Mini), the evaluated models, and the judge panel (Claude Haiku 4.5, GPT-4.1 Mini, Mistral Small 3) come from distinct providers with no shared training pipelines.
  • Multi-Judge Averaging: Scores are averaged across three judges to dilute individual biases.
  • Hallucination Bounding: By analyzing where frontier models disagree with reference answers, the authors empirically bounded hallucination risk to 1.7%.

B. Evaluation Framework

• Scoring: A rubric-based assessment layer scores models on a 1–5 scale based on engineering behaviors (e.g., stated assumptions, verification steps, constraint satisfaction).
• Automatic Checks: The harness flags empty outputs, refusal patterns, unit inconsistencies, and constraint violations.
• Human-in-the-Loop: A small, stratified subset is used for human adjudication to calibrate the rubric and resolve disagreements.

3. Key Contributions

1. First Multi-Field Engineering Benchmark: ERI is the first large-scale dataset to systematically cover nine engineering fields with a controlled cross-product of intents and difficulty levels.
2. Intent-Conditioned Evaluation: It isolates specific engineering work types (e.g., distinguishing between "explaining a concept" vs. "troubleshooting a failure"), allowing for granular diagnosis of model weaknesses.
3. Robust Validation Methodology: The paper introduces a rigorous protocol for synthetic benchmarks that uses cross-provider triangulation and convergent validity analysis to empirically bound hallucination risks, setting a new standard for dataset reliability.
4. Open-Source Infrastructure: The release includes the dataset (JSONL), taxonomy specifications, validation scripts, and an evaluation harness to enable reproducible comparisons and regression testing.

4. Key Results

The benchmark was tested on seven LLMs, revealing a distinct three-tier performance structure:

• Frontier Models (Top Tier):

  • Models: GPT-5, Claude Sonnet 4, DeepSeek V3.1.
  • Performance: Achieved mean scores > 4.30 (on a 5-point scale).
  • Robustness: GPT-5 showed minimal performance degradation (only 1.4% drop) between undergraduate and graduate-level questions. They demonstrated high reliability across all fields and intents.
  • Statistical Note: The performance gap between Claude Sonnet 4 and DeepSeek V3.1 was negligible (Cohen's d = 0.02), suggesting competitive performance is no longer architecture-specific.

• Mid-Tier Models:

  • Models: Mistral 7B, Llama 3.3 70B.
  • Performance: Scores around 3.75–3.81.
  • Insight: Parameter count alone does not guarantee performance; Llama 3.3 70B (10x larger than Mistral 7B) performed similarly, suggesting domain-specific reasoning requires more than just scale.

• Lower-Tier Models:

  • Models: Qwen 2.5 7B, Llama 3.1 8B.
  • Performance: Scores < 3.30 with failure rates (scores ≤ 2) exceeding 10% (vs. <1% for frontier models).
  • Degradation: These models showed steep performance drops (approx. 10% loss) as difficulty increased from undergraduate to graduate levels, indicating an inability to handle complex constraints.

• Task-Specific Findings:

  • Easiest: Definition (DEF) tasks.
  • Hardest: Design (DES) and Troubleshooting (TRB) tasks.
  • Domain Sensitivity: Weaker models struggled disproportionately in specialized fields like Aerospace and Fire Science compared to Industrial Engineering.

5. Significance

• For Research: Provides a standardized instrument to diagnose systematic errors, test model versions, and study the effects of instruction tuning and routing strategies in engineering contexts.
• For Industry: Offers a risk-mitigation tool. By identifying failure rates in specific intents (e.g., troubleshooting) and difficulty tiers, practitioners can make informed decisions about model deployment and the necessity of human-in-the-loop safeguards.
• For Education: Serves as a resource to teach and assess engineering reasoning norms, such as stating assumptions and verifying results.
• Methodological Impact: The convergent validation protocol offers a blueprint for creating reliable synthetic benchmarks in other domains where human-verified ground truth is expensive or impractical to obtain at scale.

Conclusion: The ERI benchmark demonstrates that while frontier LLMs are approaching expert-level performance in engineering reasoning, smaller models remain unsuitable for safety-critical applications without significant safeguards. The dataset and its validation framework provide a critical step toward trustworthy AI integration in engineering workflows.