Technical Summary: DiagnosticIQ

Problem Statement

Industrial asset maintenance relies on engineer-authored symbolic rules that trigger based on sensor conditions to prompt corrective actions. While automated monitoring systems effectively detect anomalies by evaluating these rules against sensor data, a critical bottleneck exists in the rule-to-action step: translating a triggered symbolic condition into a specific, asset-appropriate maintenance action. This translation requires deep, asset-specific knowledge of failure modes typically gained through years of hands-on experience, which often exceeds the expertise of the operators receiving the alerts.

The paper posits that Large Language Models (LLMs) could serve as decision support for this step. However, systematic evaluation has been hindered by a lack of realistic, standardized benchmarks that distinguish between genuine reasoning about industrial conditions and surface-level pattern matching. Existing benchmarks often focus on fault recognition or sensor-failure association rather than the downstream decision of what action to take once a fault is detected.

Methodology

The DiagnosticIQ Benchmark

The authors introduce DiagnosticIQ, a benchmark suite comprising 6,690 expert-validated multiple-choice questions (MCQs) derived from 118 expert-authored rule-action pairs across 16 heterogeneous industrial asset types (e.g., Air Handling Units, Chillers, Boilers).

Dataset Construction Pipeline:

1. Input: Expert-curated rule documents containing logical conditions (temporal predicates, sensor thresholds), maintenance actions, and metadata.
2. Normalization: Rules are converted into Disjunctive Normal Form (DNF). This expands a single rule into multiple logically valid trigger scenarios (conjunctions), ensuring coverage of distinct operational states rather than a single abstract example.
3. Question Generation: Two complementary question types are generated:
   • Selection: Identifying the correct maintenance action given a triggered condition.
   • Elimination: Identifying an action that is not supported by the condition.
4. Distractor Sampling: Distractors are not random; they are sampled using embedding-based similarity from rules of the same asset family but with different logical conditions. This ensures distractors are "operationally adjacent" (plausible but incorrect) rather than semantically distant.
5. Context: Questions include asset descriptions to anchor the reasoning in specific configurations (e.g., horizontal vs. vertical pumps).

Variants for Targeted Evaluation

To probe specific failure modes, the authors constructed five variants:

• DiagnosticIQ Pro: Expands options from 4 to 10 to test discrimination among plausible distractors.
• DiagnosticIQ Pert: Perturbs surface forms (ordering, wording) to test robustness against formatting biases.
• DiagnosticIQ Verbose: Paraphrases symbolic conditions into natural language to measure the symbolic-to-linguistic gap.
• DiagnosticIQ Aug: Inverts logical operators (e.g., < to >) to test if models rely on rule templates rather than evaluating logical content.
• DiagnosticIQ Rationale: Collects chain-of-thought explanations for expert validation.

Experimental Setup

The study benchmarks 29 LLMs (spanning 9 families, including frontier closed-source and open-weight models) and 4 embedding baselines. Evaluation includes zero-shot prompting, fine-tuning experiments (SFT and GRPO), and a human evaluation with 9 engineering practitioners to validate benchmark difficulty.

Key Results

1. Frontier Performance and Brittleness

• Top Performance: The leading model, claude-opus-4-6, achieves 73.59% Macro accuracy on the base DiagnosticIQ. The top three models (claude-opus-4-6, gpt-5.4, gemini-3.1-pro-pre) cluster within one Macro point of each other.
• Brittleness under Stress: Performance drops significantly under stress:
  • DiagnosticIQ Pro: The leader drops 13.8 points (to 59.81%) when options expand to 10.
  • DiagnosticIQ Pert: Top models show only ~48% consistency when question phrasing is perturbed.
  • DiagnosticIQ Aug (Logical Inversion): When logical operators are inverted, the leader selects the original (now incorrect) answer 63.4% of the time, indicating a reliance on rule-shape recognition rather than logical evaluation.
  • DiagnosticIQ Verbose: Paraphrasing symbolic conditions into natural language causes an average accuracy drop of 4.2 points, revealing a dependency on symbolic structure as reasoning scaffolding.

2. Comparison with General Benchmarks

While frontier models have advanced significantly on general-science benchmarks like GPQA Diamond (advancing ~19 points), their progress on DiagnosticIQ is stalling (advancing only ~8 points). The gap between general reasoning and industrial maintenance reasoning has widened, suggesting that post-training optimizations for general science do not cleanly transfer to industrial rule evaluation.

3. Failure Modes and Patterns

• Pattern Matching vs. Reasoning: Analysis of the DiagnosticIQ Aug variant reveals that models treat structured industrial rules as templates to recognize rather than conditions to evaluate.
• Distractor Sensitivity: Under distractor expansion, many models (including reasoning-tuned ones like o1) fall back to lexical similarity. The strongest embedding baseline (all-mpnet-base-v2) outperforms 14 of the 29 LLMs on DiagnosticIQ Pro.
• Difficulty Predictors: Logistic regression shows that special function counts (arithmetic/comparison operations) are the strongest predictor of failure, reducing correctness odds by 47% per operation. Counterintuitively, higher logical complexity (more clauses) slightly helps accuracy, likely by providing structural scaffolding.

4. Human Evaluation

A panel of 9 engineering practitioners achieved a mean accuracy of 45.0% on a subset of 40 questions. Notably, practitioners with self-reported experience in the relevant asset classes (AHU, Chillers) did not significantly outperform the general mean, confirming that the benchmark requires specialist diagnostic knowledge beyond operational familiarity. Frontier LLMs outperformed the human mean on several questions where no human practitioner could identify the correct answer.

5. Operational Impact

Despite accuracy rankings, the model with the highest estimated operational impact (based on energy savings from correct recommendations) was gemini-3.1-pro-pre, not the accuracy leader claude-opus-4-6. This suggests that for deployment, models that are consistently correct on high-impact rules may be preferable to those with marginally higher aggregate accuracy.

Significance and Claims

The paper claims that DiagnosticIQ is the first benchmark to simultaneously cover Industrial-asset Specificity, Option Distractors, Condition Understanding, and Action Recommendation.

The primary significance of the work is the revelation of a "brittleness arc" in current frontier LLMs. While models can achieve high accuracy on standard industrial maintenance tasks, they exhibit significant fragility when:

1. Distractor spaces expand.
2. Logical content is inverted.
3. Symbolic structures are paraphrased into natural language.

The authors conclude that the deployment bottleneck for LLMs in industrial maintenance is not a lack of capability, but a lack of calibration and robustness. Current models treat structured rules as templates to recognize rather than conditions to evaluate. The benchmark provides a necessary tool to distinguish between genuine reasoning and surface pattern matching, highlighting that gains on general-knowledge benchmarks do not guarantee readiness for high-stakes operational deployment where consistency across phrasings and logical variations is non-negotiable.