Technical Summary: ADMITBench – A Safety-Governed Reference Framework

1. Problem Statement

In industrial operations, the safety of a system is not determined solely by the correctness of a diagnosis, but by the admissibility of the proposed action. Existing evaluations of Large Language Models (LLMs) in industrial settings often focus on fault classification, sensor explanation, or general task completion. However, a model may correctly identify a fault (e.g., a cooling loss in a reactor) yet recommend an intervention that violates operating procedures, exceeds delegated authority, relies on untrusted sensor data, or leads to unacceptable physical consequences.

Current benchmarks fail to address the "diagnosis-to-action gap." They do not sufficiently evaluate whether a recommendation is permissible under specific plant constraints, safety instrumented functions, and time-dependent process consequences. ADMITBench addresses this by shifting the unit of evaluation from the natural language answer to the proposed action record, assessing whether that action is admissible within a versioned, safety-governed context.

2. Methodology

ADMITBench operates as a reference framework that evaluates industrial LLM advisories against a versioned plant profile (referred to as a "cartridge"). The methodology is built on a non-compensatory evaluation hierarchy, meaning a failure in any critical safety gate renders the entire action ineligible, regardless of performance in other areas.

2.1 Core Components

• Action Record: The unit of evaluation is a structured record containing the proposed action, asserted diagnosis, cited evidence, authority scope, confidence levels, and recovery paths. This replaces free-text answers to ensure auditability.
• Plant Profiles (Cartridges): Plant-specific logic (safety cases, procedures, system graphs, and consequence models) is encapsulated in versioned cartridges. The common evaluation logic remains separate from plant-specific content to ensure portability.
• Authority Model: The framework distinguishes between three deployment configurations:
  • A0 (Advisory): The model recommends verification or escalation but cannot issue direct commands.
  • A1 (Supervised): The model proposes bounded actions for human approval.
  • A2 (Constrained Autonomous): The model proposes actions within a strict grammar and operating envelope.
  • Note: Authority can narrow due to degraded evidence but cannot expand based on model claims.

2.2 The Evaluation Hierarchy (T0–T6)

The framework employs a tiered system where T0–T4 are hard gates (non-compensatory), T5 is for utility ranking, and T6 is for audit.

• T0 (Record Integrity): Checks structural completeness, type validity, and successful compilation against the safety case graph.
• T1 (Evidence & State Validity): Verifies that the decision is based on available, temporally valid, and consistent evidence.
• T2 (Hazard & Causal Understanding): Ensures the record addresses the decision-relevant hazard sufficiently to justify the action or appropriately escalates if understanding is insufficient.
• T3 (Action Admissibility): Validates authority, procedure conformance, preconditions, reversibility, and recovery obligations.
• T4 (Physical Consequence Verification): Uses dynamic or rule-based checks to verify the action avoids unacceptable physical consequences over a defined verification horizon.
• T5 (Utility Ranking): Only applied to records that pass T0–T4. Ranks eligible actions based on efficiency, disruption, or preference.
• T6 (Traceability): Records source linkage and audit evidence for reproducibility.

2.3 Eligibility and Scoring

A record that fails any hard gate (T0–T4) receives an aggregate score of None (ineligible) and is excluded from utility ranking. Only eligible records are assigned a weighted aggregate score based on subscores from T1–T5. This design ensures that "safe" but "useless" actions are not penalized, while "useful" but "unsafe" actions are strictly rejected.

3. Key Contributions

The release 0.1.0 of ADMITBench provides the following specific contributions:

1. A Reference Framework: A structured approach to evaluating industrial LLM advisories at the action level rather than the diagnosis level.
2. Common Action-Record Format: A schema for parsing model outputs into structured records containing diagnosis, evidence, authority, and recovery data.
3. Non-Compensatory Evaluation Hierarchy: A strict gating mechanism (T0–T4) that prevents unsafe actions from being ranked highly due to other strengths.
4. Implemented Plant Profiles: Two reference cartridges are released:
   • cstr_alpha: A jacket-cooled continuous stirred-tank reactor (CSTR) profile for evaluating evidence, hazard understanding, and physical consequences.
   • column_alpha: A distillation column profile that includes a specific case (D03) testing authority mismatch and escalation requirements.
5. Public Implementation: A repository containing the evaluator logic, the two cartridges, and the reporting tools.

4. Results and Limitations

The paper does not present broad comparative results across multiple models or claim general industrial validity. Instead, it demonstrates the feasibility of the evaluation format:

• The framework successfully processes action records from the two released profiles (cstr_alpha and column_alpha).
• It correctly identifies and separates diagnosis accuracy from action admissibility.
• It successfully enforces the "hold," "verify," and "escalate" actions as valid responses when evidence is degraded.

Explicit Limitations and Non-Claims:

• Not a Safety Certificate: The framework is a technical evaluation tool, not a functional safety certification, site acceptance test, or deployment authorization.
• Limited Generality: The two released profiles are reference settings, not complete representations of industrial operations. Broad industrial generality is not established.
• No Human-Factor Validation: The framework does not prove human-factors adequacy (e.g., operator trust, automation bias) in real control rooms.
• No Autonomous Deployment: Release 0.1.0 is for research and technical evaluation only; it does not authorize physical execution of autonomous controllers.

5. Significance and Claims

The paper positions ADMITBench as a reference framework to make action-level evaluation explicit, inspectable, and replayable. Its primary significance lies in:

• Shifting the Focus: Moving the industry conversation from "Can the model diagnose?" to "Is the proposed action admissible?"
• Safety Governance: Implementing a safety-governed evaluation contract where eligibility is determined by explicit, non-compensatory checks derived from versioned plant profiles.
• Reproducibility: Providing a structure where every gate verdict is replayable against a fixed action record, evaluator version, and plant profile.

The authors explicitly state that the current release is a public reference implementation for technical and research evaluation. Future work includes portability validation using the Tennessee Eastman process, the introduction of adversarial splits, and the development of a registry for independently reviewed cartridges. The paper concludes that while the framework provides a rigorous method for evaluating the "diagnosis-to-action gap," it remains a tool for engineering review and assurance, not a replacement for accountable human judgment or formal safety certification.