Technical Summary: DIAGFLOWBENCH

Problem Statement

Large Language Models (LLMs) are increasingly deployed as maintenance advisory systems, grounded in procedural documentation (e.g., flowcharts) to constrain outputs to approved steps and prevent hallucination. While existing benchmarks evaluate a model's ability to follow these procedures on clean, mappable inputs, they fail to address a critical real-world dynamic: operators frequently deviate from the documented path. In practice, operators report novel symptoms, raise questions mid-inspection, or use domain-specific shorthand that does not map cleanly to the procedure's edges.

Current evaluation paradigms treat "off-procedure" recognition as a separate, single-turn abstention task. However, in a live diagnostic dialogue, a model must simultaneously advance through compliant turns and recognize non-compliant inputs within the same conversation. The paper argues that existing benchmarks do not adequately test a grounded model's ability to handle these "off-procedure" inputs, particularly the risk that a model might select a real but contextually inappropriate step rather than fabricating a fact.

Methodology

The authors introduce DIAGFLOWBENCH, a benchmark designed to evaluate flowchart-grounded models under both cooperative and off-procedure conditions.

Dataset Construction

• Source: 50 anonymized industrial diagnostic flowcharts from a consumer manufacturer, covering fault classes such as conveyor belt tracking and robotic placement.
• Structure: The graphs contain between 7 and 60 nodes (mean 31) with decision branching factors of 2–4.
• Conversations: These flowcharts were rendered into 1,676 multi-turn conversations.
  • Cooperative: 838 conversations where the operator strictly follows the procedure.
  • Mixed: 838 conversations where off-procedure turns are injected at non-terminal positions.
• Injection Categories: Off-procedure inputs are stratified into three types based on their relationship to the procedure:
  1. Coverage Gap: A symptom the procedure has no branch for.
  2. Undocumented Malfunction: A fault outside any branch condition at the current node.
  3. Unrelated Question: A query entirely outside the diagnostic task.
• Noise Perturbation: A robustness pass introduces typos, telegraphic phrasing, and abbreviations characteristic of maintenance work-order text.

Evaluation Protocol

The study evaluates ten commercial and open-weight models (including Gemini, GPT-4o, Llama 4, Qwen3, Mistral, and Nemotron variants). The evaluation defines specific failure modes for off-procedure turns:

• Correct Abstention (CA): The model recognizes the input is out-of-scope and declines/redirects.
• Fabrication (FA): The model returns a step absent from the procedure (standard hallucination).
• Forced Mapping (FM): The model returns a genuine node from the procedure graph, but no edge into that node is entailed by the operator's utterance. This is the core failure mode investigated.

Metrics include Step Accuracy (SA) and Termination Recognition (TR) for on-procedure turns, and the rates of CA, FA, and FM for off-procedure turns. A language-model judge (Claude Haiku) classifies off-procedure responses, validated against human annotators.

Key Contributions

1. DIAGFLOWBENCH Dataset: A dataset of 50 industrial flowcharts converted into 1,676 multi-turn conversations that explicitly contrast compliant dialogues with off-procedure counterparts.
2. Comprehensive Evaluation: An assessment of ten diverse models across commercial and open-weight categories, analyzing performance across different architectures and scales.
3. Identification of "Forced Mapping": The paper identifies and defines a specific failure mode where models forcefully assign an off-procedure query to an inapplicable step. Unlike fabrication, these steps are structurally valid and grounded in the manual, making them difficult to detect via standard similarity-based grounding checks.

Results

• On-Procedure Competence: Models generally perform well on compliant inputs, with Step Accuracy (SA) ranging from 70.1% to 85.0%. However, Termination Recognition (TR) is significantly more volatile (38.8% to 98.5%).
• Off-Procedure Competence:
  • Fabrication (FA) is low: Grounding successfully prevents models from inventing steps (FA rates between 2.2% and 8.6%).
  • Forced Mapping (FM) is dominant: FM rates are substantially higher and more volatile (15.7% to 67.4%), emerging as the primary failure mode.
  • Abstention is inconsistent: Correct Abstention (CA) rates vary widely and are not reliably predicted by model scale, architecture, or access tier. Notably, older models (Llama 3.3 70B) sometimes outperform newer generations (Llama 4) in off-procedure handling.
• Recovery: A single FM or FA error causes a catastrophic collapse in subsequent performance. The recovery rate (correctly identifying the next step after a failure) drops to between 1.0% and 9.1%, compared to the 70–85% baseline for clean conversations.
• Drivers of FM: Forced mapping is strongly governed by lexical similarity between the off-procedure query and the procedure text. Models are more likely to force-map "Coverage Gap" inputs (which share vocabulary with the manual) than unrelated questions.

Significance and Claims

The paper claims that the constraint mechanisms designed to prevent hallucination (grounding models to a closed set of procedural steps) may inadvertently create a "blind spot." By forcing models to select from a valid set of nodes, the system may camouflage contextual errors as valid steps.

The authors argue that:

1. Reliability cannot be inferred from low FA rates: A model with low fabrication rates can still be highly unreliable due to high forced mapping rates.
2. Standard guardrails are insufficient: Because FM steps are traceable to the manual and structurally valid, they evade standard hallucination detection.
3. Evaluation must measure failure safety: Evaluating advisory systems requires measuring not just compliance, but how safely a model fails when reality diverges from the manual.
4. Model selection is not a silver bullet: Neither model scale nor reasoning-oriented architecture definitively predicts off-procedure reliability; in some cases, reasoning models exhibit an unusually low tendency to abstain.

The study concludes that robust deployment requires external verification layers capable of evaluating contextual entailment, rather than relying solely on generative models to distinguish between ambiguous and genuinely off-procedure inputs.