🤖 AI

A Multimodal Manufacturing Safety Chatbot: Knowledge Base Design, Benchmark Development, and Evaluation of Multiple RAG Approaches

This paper presents the design, implementation, and systematic evaluation of a multimodal, RAG-powered manufacturing safety chatbot, introducing a domain-specific benchmark and a methodology to optimize accuracy, latency, and cost for Industry 5.0 safety training.

Original authors: Ryan Singh, Austin Hamilton, Amanda White, Michael Wise, Ibrahim Yousif, Arthur Carvalho, Zhe Shan, Reza Abrisham Baf, Mohammad Mayyas, Lora A. Cavuoto, Fadel M. Megahed

Published 2026-02-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ryan Singh, Austin Hamilton, Amanda White, Michael Wise, Ibrahim Yousif, Arthur Carvalho, Zhe Shan, Reza Abrisham Baf, Mohammad Mayyas, Lora A. Cavuoto, Fadel M. Megahed

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "Safety Sidekick": Making Machines Talk Sense to Workers

Imagine you’ve just started a new job in a massive, high-tech factory. You’re standing in front of a giant, complex machine—maybe a robotic arm or a heavy-duty lathe—and you realize you aren't quite sure which button to press to clear a jam without getting your hand caught.

In the old days, you’d have to stop everything, find a thick, dusty paper manual, or hunt down a supervisor. In the "Industry 5.0" era, researchers want to give you a "Safety Sidekick"—a smart, conversational chatbot that lives on your phone or tablet, can "see" what you’re looking at, and can answer your questions instantly.

This paper describes how a team of researchers built, tested, and perfected this digital sidekick.

1. The Problem: The "Manual" Headache

Manufacturing is dangerous. Even with modern tech, people get hurt every day. The researchers identified three big problems with current training:

It’s Static: Learning happens once a year in a classroom, but the factory floor changes every minute.
It’s Hard to Access: Finding a specific safety rule in a 500-page manual while a machine is humming is like trying to find a specific needle in a haystack while wearing oven mitts.
It’s Not Personal: Every machine is slightly different, and every worker has different needs.

2. The Solution: The "Smart Librarian" (RAG)

The researchers didn't just build a generic AI (like a standard ChatGPT) because generic AIs sometimes "hallucinate"—they make things up confidently. In a factory, a "confident lie" can be fatal.

Instead, they used a technique called RAG (Retrieval-Augmented Generation).

The Analogy:
Think of a standard AI like a student taking an exam relying only on their memory. They might remember things correctly, or they might misremember a detail.
RAG is like giving that student an "Open Book Exam." Before the student answers your question, they quickly flip through a library of only official, verified manuals (OSHA laws, machine manuals, etc.), find the exact right page, and then summarize that specific info for you. This ensures the "Sidekick" isn't guessing; it's quoting the experts.

3. The Experiment: The "Ultimate Taste Test"

The researchers didn't just build it; they put it through a grueling "stress test." They created a massive test called a Benchmark.

They picked three different types of machines—ranging from an old-school manual mill to a high-tech collaborative robot (a "cobot")—and wrote dozens of tricky questions. They then tested 24 different "brain configurations" for the chatbot to see which one worked best.

They were looking for the "Goldilocks Zone" across three categories:

Accuracy: Does it give the right safety advice? (The most important!)
Latency: How fast does it answer? (If it takes 2 minutes, the worker has already moved on.)
Cost: How much does it cost to run? (If it's too expensive, small businesses won't use it.)

4. The Results: Finding the Winner

After running thousands of tests, they found that the "brain" of the chatbot matters immensely.

They discovered that using "Keyword Search" (looking for exact technical terms) combined with a powerful but efficient AI model was the winner. It achieved an 86.66% accuracy rate, cost only a fraction of a penny per question, and answered in about 10 seconds.

The Metaphor:
It’s like finding the perfect personal assistant. You don't want an assistant who is a genius but takes an hour to reply (too slow), nor do you want one who is lightning-fast but constantly gives you the wrong directions (too dangerous). You found the one who is smart, quick, and affordable.

5. Why This Matters

This isn't just about a cool app; it's about a new way of working. By creating an open-source tool and a standardized test (the benchmark), the researchers have given the entire manufacturing world a blueprint.

In the future, every worker could have a multimodal assistant—one that can hear your voice, read your text, and see through your camera—to ensure that everyone goes home safe at the end of the shift.

Technical Summary: A Multimodal Manufacturing Safety Chatbot

1. Problem Statement

Modern manufacturing is transitioning from Industry 4.0 (automation-focused) to Industry 5.0, which emphasizes human-centricity, resilience, and safety. Despite technological advances, workplace injuries remain high, costing the U.S. economy billions annually. Traditional safety training is often static (classroom or annual lectures) and fails to provide real-time, context-aware guidance when operators encounter specific equipment or unexpected hazards. There is a critical need for an adaptive, accessible, and accurate system that can provide immediate safety instructions in a dynamic shop-floor environment.

2. Methodology

The researchers employed a Design Science Research (DSR) methodology, following a six-stage process to develop and evaluate a technical artifact.

Design Requirements: The system was designed to satisfy three core constraints: High Accuracy (to prevent fatal errors), Low Latency (to ensure responsiveness), and Low Cost (to ensure adoption by small/medium manufacturers).
Artifact Development (RAG Framework): The authors developed a Retrieval-Augmented Generation (RAG) chatbot.
- Knowledge Base: A curated corpus of OSHA regulations, NIOSH safety alerts, and Original Equipment Manufacturer (OEM) manuals for three machines (Bridgeport Mill, Haas CNC Lathe, and UR5e Cobot).
- Retrieval Strategies: They implemented and tested six distinct retrieval mechanisms: OpenAI Keyword search, OpenAI Semantic search, local BM25 (lexical), and three Graph-based RAG variants (Graph Eager, Graph MMR, and Vanilla similarity).
- Multimodality: The chatbot supports text, image (visual context), and speech-to-speech interactions.
Experimental Design: A full-factorial design was used to test 24 different RAG configurations. The variables included the retrieval approach, the LLM model (GPT-5-mini vs. GPT-5-nano), and the retrieval depth (top- $k$ = 3 vs. 7).
Evaluation Pipeline:
1. Benchmark Development: A domain-specific dataset of expert-validated Q&A pairs (Easy, Moderate, and Challenging) was created.
2. Automated Evaluation: Used an "LLM-as-a-judge" to measure accuracy, alongside metrics for end-to-end latency and token-based cost.
3. Human Evaluation: A blind side-by-side comparison by 11 industry experts and researchers to validate the top-performing configurations.

3. Key Contributions

The Artifact: An open-source, multimodal, domain-grounded safety training chatbot.
The Benchmark: A validated, expert-vetted Q&A dataset specifically designed for evaluating AI in manufacturing safety.
The Methodology: A systematic, reproducible framework for designing, testing, and selecting RAG configurations in high-stakes industrial environments.

4. Results

The study revealed significant trade-offs between the three design requirements:

Accuracy: The larger model (GPT-5-mini) significantly outperformed the smaller model (GPT-5-nano). Among retrieval methods, OpenAI Keyword and Semantic searches were the most accurate, significantly outperforming Graph-based and Vanilla methods.
Latency: The smaller model was faster, but the larger model's accuracy was deemed more critical. Increasing retrieval depth ( $k=7$ ) increased latency slightly but did not significantly improve accuracy.
Cost: While the mini model was more expensive per query, the absolute cost remained extremely low (approx. $0.005 per query).
The Winning Configuration: The GPT-5-mini + OpenAI Keyword + top- $k$ =7 pipeline was selected for deployment. It achieved:
- Accuracy: 86.66%
- Average Cost: $0.005 per query
- Average Latency: 10.04 seconds (end-to-end)

5. Significance

This research provides a blueprint for deploying AI in safety-critical industrial settings. By demonstrating that RAG can ground LLMs in authoritative regulatory and technical documentation, the authors show how to mitigate "hallucinations" that could lead to physical harm. The work bridges the gap between general-purpose AI and the specialized, high-reliability needs of Industry 5.0, offering a scalable way to modernize worker training and hazard prevention.