Technical Summary: Industrial-Instruction

Problem Statement

Industrial technical reports constitute a high-value knowledge repository for maintenance, troubleshooting, and product engineering, yet they remain largely inaccessible to standard Large Language Models (LLMs). These documents suffer from heterogeneous structures, integrating dense prose, complex specifications, and tabular data, which complicates indexing and reasoning. While general-purpose LLMs have advanced, they lack the specialized depth required for industrial domains, often failing to provide the precise, actionable guidance needed by engineers. Furthermore, a critical gap exists in the availability of public instruction-tuning and benchmark datasets derived from real-world industrial documentation, particularly those that account for tabular data and the specific challenges of retrieval-augmented generation (RAG) in noisy environments. Existing benchmarks, such as FailureSensorIQ, highlight that even leading models struggle with industrial reasoning, achieving low accuracy on tasks requiring the mapping of failure modes to sensor data.

Methodology

The authors propose Industrial-Instruction, an end-to-end framework designed to construct instruction-tuning and benchmark datasets directly from industrial technical reports. The methodology is demonstrated through a case study using 906 publicly available Panasonic documents (7,525 pages).

1. Layout-Aware Extraction and Preprocessing

To handle the complex structure of PDFs, the pipeline employs Dots.OCR, a vision-language model, to extract text, tables, and layout information. Unlike rule-based methods that often fragment content, Dots.OCR converts documents into Markdown, preserving the semantic relationship between text and tables. The pipeline filters out images to focus on text-based reasoning, removes duplicates and empty pages, and utilizes EmbeddingGemma (a 300M parameter model) with FAISS for efficient semantic indexing and retrieval.

2. Synthetic Data Generation via Five Scenarios

The core innovation lies in the generation of question-answer (QA) pairs grounded in five realistic query-document relationships, designed to train models on robustness against retrieval noise and multi-step evidence integration:

• r0 (Useless Document): The retrieved document is related but provides no useful information, training the model to avoid hallucination.
• r1 (Single-Document Support): A single document offers clues but not an explicit answer.
• r2 (Multi-Document Support): Multiple documents provide scattered clues requiring integration.
• r3 (Single-Document Answer): A single document contains the complete, explicit answer.
• r4 (Multi-Document Answer): Multiple documents collectively provide the answer, requiring multi-hop reasoning.

3. Dual-Model Generation Pipeline

To investigate the impact of the generator model on downstream performance, the authors executed the pipeline twice:

• Open-Weight Version: Using Qwen3-30B-A3B-Instruct locally.
• Frontier Version: Using the closed, API-based Claude-Opus-4.6.

Both runs utilized identical prompts, retrieval pipelines, and scenario definitions. The generated samples underwent rigorous rule-based filtering to ensure valid JSON structures, complete option descriptions for multiple-choice questions, and the absence of embedded answers in the question text.

Key Contributions

1. Two Industrial QA Datasets: The release of the "Panasonic Dataset" in two parallel versions (Qwen-generated and Claude-generated). Each dataset contains approximately 13,600 high-quality QA pairs with associated source documents and a held-out benchmark split, specifically targeting small LLMs (<10B parameters).
2. End-to-End Construction Pipeline: A reproducible pipeline that adapts multi-scenario RAG paradigms to heterogeneous industrial PDFs, combining layout-aware extraction, semantic indexing, and automated quality filtering.
3. Comparative Study of Data Generators: A direct comparison between open-weight and frontier models as data generators, analyzing raw data quality, downstream fine-tuning gains, cost, and general-knowledge retention.

Experimental Results

The study evaluated small open LLMs (specifically Qwen-4B-Instruct) on the Panasonic benchmark and the FailureSensorIQ benchmark.

• Performance Gains: Full fine-tuning on the Industrial-Instruction dataset significantly improved performance. On the Panasonic benchmark, Set-Match Accuracy increased from 28.5% to 42.0% (Qwen-generated) and from 40.9% to 56.4% (Claude-generated). F1 scores rose from 46.6% to 63.5% and 58.55% to 72.66%, respectively.
• General Knowledge Retention: Fine-tuning on the Claude-Opus-4.6 generated data resulted in negligible catastrophic forgetting (MMLU accuracy dropped by only 0.05 points), whereas the Qwen-generated data caused a measurable drop of 1.26 points, particularly in the Humanities category.
• Cost vs. Quality: The Claude-generated dataset required roughly $330 in API costs compared to $3.2 for the local Qwen run. While the Claude dataset was "cleaner" (0.5% filtering rate vs. 43% for Qwen) and yielded larger downstream gains, the authors note the improvement was not strictly proportional to the two-orders-of-magnitude cost difference.
• Robustness Limitations: A critical finding was that all fine-tuned models (regardless of the generator or dataset) achieved 0% accuracy on the perturbed (rephrased) variants of the FailureSensorIQ benchmark. This suggests a fundamental brittleness in the Qwen-4B backbone regarding question rephrasing that the current dataset construction did not address.

Significance and Claims

The paper claims to offer a practical, reproducible path toward scalable industrial benchmarks derived from real-world documentation. It demonstrates that:

• Small language models (<10B parameters) can be effectively trained on industrial data to achieve substantial performance gains, making them viable for resource-constrained industrial environments.
• Open-weight models are a highly cost-effective alternative to frontier models for generating synthetic training data, provided rigorous filtering is applied.
• The specific structure of the dataset (modeling five distinct retrieval scenarios) is crucial for training models to handle the noise and complexity of real industrial RAG systems.

The authors conclude that while their framework successfully bridges the gap between general LLMs and industrial knowledge, the inability of current models to handle perturbed questions remains a significant limitation for future research, suggesting a need for adversarial or paraphrased variants in future dataset iterations.