Evaluating the Use of LLM-based Multi-Agent Systems for Internal Information Sharing and Access Within Manufacturing

Technical Summary: Evaluating LLM-based Multi-Agent Systems for Internal Information Sharing in Manufacturing

Problem Statement
Internal information sharing within manufacturing enterprises is currently hindered by data silos, heterogeneous digital systems, and poor data quality, particularly across the IT/OT divide defined by the ISA-95 functional hierarchy. While Large Language Models (LLMs) are proposed as solutions for integrating operational technology (OT) and information technology (IT) data, there is a significant lack of empirical evidence comparing their performance against unassisted human users on representative manufacturing tasks. Furthermore, existing research often neglects lower ISA-95 levels (L1–L2), lacks quantitative human benchmarking, and fails to test system resilience under variable data quality conditions common in real-world manufacturing (e.g., scanning errors, truncated identifiers).

Methodology
The study utilized the VOOM Smart Factory Demonstrator, a 3D-printing production line, to generate heterogeneous data spanning ISA-95 Levels 1 to 3. The dataset included location tracking, relationship tracking, packing lists (PDFs), machine logs, and Authorised Release Certificates (PDFs).

A modular, multi-agent LLM system was developed using the LangChain framework, featuring four specialized agents:

1. Master Agent: Orchestrates workflows and decomposes queries.
2. Data Retrieval Agent: Interfaces with raw data sources via a domain-specific tool suite.
3. Reconciliation Agent: Cross-validates information from disparate sources.
4. Synthesis Agent: Generates human-readable reports.

The system was benchmarked against nine human participants (engineering students with technical literacy but no prior factory experience) across three task complexities:

• E-Tasks (Easy): 3-step information retrieval.
• M-Tasks (Medium): 4-step traceability.
• H-Tasks (Hard): 6-step complex verification.

Performance was evaluated under four data quality conditions:

• Q0: Baseline (uncorrupted).
• Q1: Whitespace injection in identifiers.
• Q2: Truncated identifiers.
• Q3: Severed relational links (one-directional removal).

Six LLMs from three providers (Anthropic, DeepSeek, OpenAI) were tested, with each task executed three times to account for stochastic variation.

Key Results

• Overall Performance: The top-performing LLMs significantly outperformed the average human participant in both accuracy and speed. The mean accuracy across all six LLMs was 80.2%, compared to 62.2% for humans. The best model, Claude Sonnet-4, achieved 94.1% accuracy, while the best human participant achieved 100%. However, smaller models (e.g., GPT-4o Mini at 50.7%) fell below the human average.
• Speed: LLMs were substantially faster. The pooled median completion time for LLMs was 14 seconds, compared to 152 seconds for humans.
• Task Complexity: Contrary to the experimental design assumption, task difficulty did not correlate linearly with the number of steps. Humans and LLMs performed better on complex 6-step H-tasks than on simpler 3-step E-tasks, suggesting that interface fragmentation and search ambiguity, rather than procedural length, drive error rates.
• Data Quality Resilience:
  • Robust: The system was resilient to whitespace injection (Q1) and severed relational links (Q3), often maintaining near-perfect accuracy where humans struggled.
  • Vulnerable: Performance degraded sharply with truncated identifiers (Q2), causing accuracy drops of up to 42.2 percentage points for some models. This highlights a vulnerability in deterministic lookups when character data is missing.
• Prompt Sensitivity: A sub-study revealed that increasing prompt verbosity (adding context and step-by-step scaffolding) significantly reduced accuracy for top models, with some dropping from ~94% to ~22% accuracy under "Long" prompt conditions.

Key Contributions

1. Empirical Benchmarking: This study provides one of the first quantitative comparisons of integrated multi-agent LLM systems against unassisted human operators in a manufacturing context, moving beyond qualitative surveys or isolated NLP metrics.
2. Cross-Level Evaluation: It addresses the literature gap regarding lower ISA-95 levels (L1–L2), demonstrating the feasibility of retrieving and synthesizing data across the IT/OT boundary.
3. Data Quality Stress Testing: The research empirically quantifies how specific data corruption types (whitespace, truncation, link severance) impact automated retrieval, revealing that LLMs are robust to some noise but brittle to identifier truncation.
4. Architectural Validation: The study validates a multi-agent architecture with adaptive retrieval and reconciliation, showing its efficacy in handling cross-document synthesis tasks that traditional RAG systems struggle with.

Significance and Claims
The paper claims that multi-agent LLM systems have strong potential to serve as a decision-support layer for manufacturing information access, offering substantial gains in speed and average accuracy over manual processes. However, the authors emphasize that this advantage is conditional:

• Model Selection is Critical: Not all LLMs are suitable; smaller or less capable models can underperform humans.
• Human Oversight Required: The best human participants outperformed all models, indicating that LLMs should augment, not replace, skilled operators.
• Design Constraints: Successful deployment requires careful prompt design (avoiding excessive verbosity), rigorous data validation (specifically for identifier integrity), and human oversight to manage edge cases.

The study concludes that while these systems can effectively bridge information silos in bounded environments, their viability in uncontrolled, real-world manufacturing settings requires further validation with larger, more experienced participant groups and live industrial data.