Technical Summary: ManuRAG

Problem Statement

The transition to Industry 5.0 requires intelligent Question Answering (QA) systems capable of processing the complex, multi-modal data inherent in manufacturing. Manufacturing documents contain diverse data types—including text, images, formulas, tables, CAD models, and sensor data—that are often interlinked. Conventional Retrieval-Augmented Generation (RAG) systems typically struggle with this complexity, often relying solely on text or failing to capture the full context of multi-modal information. This limitation leads to inefficiencies in production workflows and hinders effective decision-making, as engineers cannot seamlessly retrieve integrated insights from mixed-format documents (e.g., a query requiring both a textual specification and a corresponding technical diagram). Furthermore, existing Large Language Models (LLMs) are prone to hallucinations and lack the specialized grounding required for the highly technical and domain-specific nature of manufacturing data.

Methodology

To address these challenges, the authors propose ManuRAG, a Multi-modal Retrieval-Augmented Generation (MMRAG) framework specifically designed for Manufacturing Question Answering (MQA). The framework operates through four primary stages:

1. Multi-modal Data Extraction:

   • Utilizing the pdf-extract-kit framework, the system processes manufacturing PDFs to separate content into distinct components: text, images, tables, and formulas.
   • Layout Detection: Employs LayoutLMv3 to identify and filter document structures (e.g., titles, page numbers) and locate content regions.
   • Formula Extraction: Uses YOLOv8 to detect and isolate inline and block formulas (e.g., Young's modulus, Fick's law) to preserve mathematical integrity.
   • Table and Image Extraction: Converts tables into structured LaTeX formats using a model trained on PubTabNet and extracts figures.
   • OCR: Applies Paddle-OCR to digitize text from non-selectable regions.
   • All extracted elements are assigned Universally Unique Identifiers (UUIDs) to maintain consistent referencing within the text.

2. Indexing:

   • The system creates vector databases for both textual and visual data.
   • Text Vector Database: Text chunks (including formulas and image references) are embedded using an LLM-based text embedding model (EmbedText).
   • Image Vector Database: Visual data (figures, tables) is encoded using a multi-modal embedding model (EmbedMM, e.g., CLIP) to enable cross-modal search (text-to-image and image-to-text).
   • The framework explores different indexing strategies, including maintaining separate databases for text and images versus converting visual data into descriptive text before embedding.

3. Retrieval:

   • The system calculates cosine similarity between the query embedding and stored data embeddings.
   • It retrieves top-k relevant text chunks and, depending on the variation, retrieves associated images either via direct image embedding similarity or by following references embedded in the retrieved text.

4. Answer Generation:

   • Retrieved context (text, images, and references) is combined with a domain-specific prompt and the user query.
   • The system utilizes either a standard LLM (for text-only or text-converted contexts) or a Multi-modal LLM (MLLM) to generate the final answer.

ManuRAG Variations

The paper evaluates four specific variations to determine the optimal alignment strategy:

• ManuRAG1: Embeds text and images into separate vector databases; uses an MLLM for generation.
• ManuRAG2: Eliminates the image vector database; relies solely on text embeddings and an LLM.
• ManuRAG3: Similar to ManuRAG1 but prioritizes retrieving images specifically referenced within the retrieved text chunks.
• ManuRAG4: A novel approach where visual data referenced in text is transformed into descriptive text using an MLLM before embedding. This aligned text is then indexed in the text vector database, and answers are generated using an LLM.

Key Contributions

1. Domain-Specific Framework: Introduction of a comprehensive multi-modal QA system tailored to the manufacturing domain, designed to identify operational bottlenecks and support data-driven decision-making.
2. Benchmark Dataset: Creation and release of a dataset comprising 1,515 QA pairs derived from Fundamentals of Modern Manufacturing, covering mathematical questions (MathQ), multiple-choice questions (MCQ), and review questions (RQ).
3. Innovative Indexing Strategy: Proposal of a method to align text and images by converting visual data into descriptive text prior to embedding (ManuRAG4), enhancing the interpretation of visual information within a text-centric retrieval pipeline.
4. Comparative Evaluation: Systematic evaluation of traditional RAG, high-quality text RAG, and various MMRAG configurations, offering insights into the efficacy of different multi-modal integration strategies.

Experimental Results

The authors evaluated ManuRAG against baselines including GPT-4O (standard and Chain-of-Thought), standard RAG, and high-quality text RAG (RAGhq) across the three dataset categories.

• Overall Performance: ManuRAG4 consistently outperformed all other methods across all datasets.
  • MathQ: Achieved the highest Factual Correctness (62.03%), significantly outperforming GPT-4O (49.91%) and standard RAG (59.51%).
  • MCQ: Achieved the highest scores in Factual Correctness (87.61%), Semantic Similarity (96.82%), and ROUGE (86.77%).
  • RQ: Maintained the highest Factual Correctness (45.75%) despite the text-heavy nature of the questions.
• Context Retrieval: ManuRAG4 demonstrated superior Context Precision for MathQ (93.53%) and MCQ (86.84%), indicating that converting visual data to text before embedding improved the relevance of retrieved context for complex queries.
• Variation Analysis:
  • ManuRAG1 and ManuRAG3 showed limitations in handling mathematical reasoning, suggesting that direct image retrieval without text conversion may not suffice for complex formulaic queries.
  • ManuRAG2 (text-only) performed well but generally lagged behind ManuRAG4, highlighting the value of integrating visual information.

Significance and Claims

The paper claims that ManuRAG establishes a versatile tool for domain-specific QA, demonstrating that integrating multi-modal data through specialized extraction and alignment strategies significantly enhances answer accuracy, reliability, and interpretability in manufacturing. The authors assert that ManuRAG4's design, which aligns visual and textual data before embedding, effectively bridges the gap between diverse data formats and complex manufacturing queries.

While the primary focus is manufacturing, the authors note that the framework's adaptable design positions it as a potential solution for other data-intensive domains such as law, healthcare, and finance. The study highlights that while multi-modal RAG has seen applications in healthcare, its potential in manufacturing—specifically for handling intricate schematics and technical diagrams—remains underexplored and is effectively addressed by this work.

The authors acknowledge limitations, noting that future work could involve stronger multi-modal embedding models (e.g., Jina-CLIP), advanced retrieval designs, and the inclusion of additional data types such as CAD models, videos, and time-series sensor data.