Technical Summary: FactoryLLM

Problem Statement

In modern smart factories, production systems operate as interconnected networks where faults in one component often cascade across multiple machines and software layers. Diagnosing these cross-system failures is challenging not due to a lack of information, but because critical technical knowledge is fragmented across hundreds of pages of disparate manuals, each using different terminologies and system abstractions.

Existing solutions face significant limitations:

• Keyword-based search fails when operator queries do not match the specific terminology of technical manuals.
• Rule-based expert systems require extensive manual engineering and struggle to adapt as equipment evolves.
• Knowledge graphs are costly to construct and maintain at scale.
• Current LLM/RAG implementations in manufacturing are typically restricted to single-machine or single-document scenarios, lacking the ability to reason across heterogeneous documentation from interacting devices (e.g., hardware and its fleet management software).
• Data Privacy: Many industrial applications rely on proprietary cloud models, creating barriers for organizations unwilling to share sensitive operational data.

Methodology: FactoryLLM Architecture

To address these gaps, the authors propose FactoryLLM, a safe, open-source AI playground designed to evaluate Retrieval-Augmented Generation (RAG) models in a cross-machine context. The system is built as a six-layer, configuration-driven pipeline:

1. Configuration Layer: Allows operators to define the underlying LLM provider (supporting local, open-source, or cloud models), prompting strategies (Input-Output, Chain-of-Thought, Tree-of-Thought, Graph-of-Thought), and RAG strategies (vector-based or graph-based). All settings are persistently stored to ensure experimental reproducibility.
2. Knowledge Ingestion Layer: Handles the upload and processing of technical documents (PDF, DOCX, TXT). Documents are segmented into chunks and indexed into session-specific databases. The system supports both ChromaDB for vector-based retrieval and NebulaGraph for graph-based retrieval, enabling the joint indexing of documentation from multiple machines (e.g., an Autonomous Intelligent Vehicle and its Mobile Planner software).
3. Retrieval Layer: Upon receiving a query, this layer executes the selected RAG strategy. In vector mode, it performs similarity searches; in graph mode, it explores relationships between concepts. It retrieves top-k relevant document sections from the unified index of multiple machines.
4. Reasoning Layer: Processes the retrieved context using the configured LLM and prompting strategy to synthesize a grounded response.
5. Interaction Layer: A chat-based interface where users submit queries and view responses, serving as the bridge between the user and the pipeline.
6. Evaluation Layer: The system employs a dual-evaluation setup to assess performance:
   • RAGAS Framework: Measures metrics such as faithfulness, answer relevance, context precision, and context recall.
   • LLM-as-a-Judge: Utilizes NVIDIA's metrics to provide additional assessment of the generation quality.

Key Contributions

The paper makes three primary contributions:

1. FactoryLLM Platform: The introduction of an open-source, safe LLM playground specifically designed to support reasoning across multiple heterogeneous documentation sources spanning different machines and processes.
2. Cross-Machine Case Study: A demonstration of the platform's efficacy using a realistic smart manufacturing scenario involving an Autonomous Intelligent Vehicle (AIV) and its Mobile Planner fleet management software. This setup tests the system's ability to diagnose faults that manifest across both hardware and software layers.
3. Systematic Evaluation: A comprehensive assessment of multiple LLMs and prompting strategies using standardized metrics (RAGAS and LLM-as-a-Judge) to evaluate cross-machine reasoning performance, addressing the lack of standardized evaluation in industrial maintenance contexts.

Results

The authors evaluated three LLMs across 30 maintenance queries derived from approximately 600 pages of cross-machine documentation (AIV and Mobile Planner manuals).

• Groundedness: Every model evaluated achieved a groundedness score above 0.88, indicating that the generated responses were strongly supported by the retrieved documentation.
• Effectiveness: The results suggest that FactoryLLM is effective in performing cross-machine document reasoning, successfully retrieving and synthesizing information from disparate sources to explain fault scenarios.
• Privacy and Safety: The system successfully demonstrated the ability to run local or open-source LLMs without requiring the sharing of sensitive industrial data, providing a controlled environment for experimentation.

Significance and Claims

The paper positions FactoryLLM as a critical step toward solving the "expertise gap" in smart manufacturing, where the retirement of experienced technicians leaves fewer operators capable of diagnosing complex, multi-system faults.

• Bridging the Gap: FactoryLLM fills three specific research gaps: the lack of cross-machine reasoning in current RAG systems, the prevalence of proprietary (non-reproducible) implementations, and the absence of standardized evaluation metrics for manufacturing maintenance tasks.
• Reproducibility and Transparency: By being open-source and supporting local execution, the platform allows the community to test and compare different models and strategies without data privacy concerns.
• Baseline for Research: The authors claim FactoryLLM provides a repeatable baseline for future research into cross-machine maintenance reasoning, moving beyond single-document or single-machine evaluations to address the interconnected reality of modern factory floors.

The full code and documentation are publicly available to facilitate community testing with specific manufacturing scenarios.