💻 computer science

A Retrieval-Augmented Generation Method for Industrial Documents based on Reinforcement Learning

This paper proposes RLo-RAG, a Reinforcement Learning-based Multi-Round Retrieval-Augmented Generation model enhanced with LoRA, which utilizes a dynamic reward function to iteratively retrieve high-relevance document chunks and significantly improves retrieval precision and generation quality for industrial documents compared to traditional RAG methods.

Original authors: Ling WeiQing, Yongqi Zhi

Published 2026-09-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ling WeiQing, Yongqi Zhi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern industrial world, factories and engineering firms generate mountains of complex documents. These are not simple manuals but dense collections of technical reports, equipment logs, and test records that hold the key to solving critical problems. For decades, finding specific information within these vast archives has been a slow, manual task, often relying on an employee's personal memory or expertise. Recently, powerful computer programs known as large language models have offered a new way forward. These models can read and understand human language with remarkable skill, acting like a super-intelligent assistant. However, when asked about specific industrial details, these assistants often struggle because they were trained on general internet data, not on the specialized, fragmented records of a factory floor. To fix this, researchers developed a method called retrieval-augmented generation, which works by having the computer look up relevant facts from a private database before answering a question. Yet, even this improved method often fails when a question requires connecting dots across multiple documents, leading to incomplete or confused answers.

A team of researchers at Tongji University has addressed this specific challenge by creating a new system designed to handle the messy, broken-up nature of industrial knowledge. They call their approach RLo-RAG, a method that combines the power of large language models with a learning technique known as reinforcement learning. In simple terms, reinforcement learning is a way of teaching a computer program by letting it try different actions and rewarding it when it makes a good choice, much like training a dog with treats for correct behavior. In this system, the computer acts as a curious agent that does not just search for an answer once. Instead, it asks a series of questions, learns from the results of each search, and refines its next question to fill in the missing pieces of information. This process continues until the computer feels it has gathered enough evidence to construct a complete and accurate answer.

The researchers found that industrial documents often suffer from a problem called semantic fragmentation. This means that the information needed to answer a single question is scattered across different sections, tables, and even separate files. A traditional search might find a paragraph describing a test procedure but miss the specific numbers located in a table on a different page. The new system solves this by using a dynamic reward system. As the computer searches, it receives feedback based on three factors: how relevant the found information is, whether it is repeating information it has already seen, and how quickly it is finding the answer. If the computer finds a highly relevant document, it gets a reward. If it wastes time looking at the same information twice, it receives a penalty. Crucially, the system adjusts these rewards over time. In the early stages of learning, it focuses heavily on finding high-quality information, but as it gets better, it learns to balance finding good information with finishing the task efficiently.

To test this idea, the researchers built a prototype system and trained it using a technique called LoRA, which allows them to teach the computer specific skills without needing to rebuild its entire brain from scratch. They tested the system on two well-known datasets designed to challenge computers with questions that require connecting multiple facts. The results showed that their new method significantly outperformed existing approaches. While older methods often missed key details or got stuck in loops of searching the same information, the new system successfully retrieved the correct pieces of information in 88.9 percent of cases on one of the test sets. It also generated answers that were more accurate and better organized than those produced by other advanced models. The researchers noted that the system's ability to decide when to stop searching and when to keep looking was a key factor in its success, preventing it from wasting time while ensuring it did not leave out important facts.

The team also compared their method against other ways of using reinforcement learning to see if their specific approach was the best choice. They found that their strategy, which carefully balances the computer's learning process, was more stable and effective than other popular methods. This stability is vital in industrial settings where a wrong answer could lead to costly mistakes. By using a smooth, step-by-step adjustment of its learning goals, the system avoided the confusion that can happen when a computer tries to learn too many things at once. The researchers validated these findings by running their experiments multiple times to ensure the results were consistent and not just a lucky fluke. They also demonstrated the system's capability with a real-world example involving civil aircraft manufacturing, showing how it could piece together information about electrical testing requirements from text, tables, and diagrams to provide a complete answer.

While the system shows great promise, the researchers acknowledge that there is still work to be done. Currently, the system relies on text descriptions to link to images and tables, meaning it understands that a paragraph mentions a specific chart, but it does not deeply analyze the visual content of that chart itself. Future improvements could allow the system to "see" and understand the images and diagrams directly, rather than just reading the text that points to them. Despite this limitation, the study demonstrates a significant step forward in making artificial intelligence useful for complex, real-world industrial tasks. By teaching computers to think more like human experts—iterating, checking their work, and knowing when they have enough information—the researchers have created a tool that could help factories and engineers unlock the full value of their vast document archives.

Technical Summary: RLo-RAG for Industrial Documents

1. Problem Statement

The paper addresses the critical challenge of knowledge fragmentation and incomplete retrieval in complex industrial documents (e.g., equipment manuals, technical reports, test records). While Large Language Models (LLMs) combined with Retrieval-Augmented Generation (RAG) offer a pathway for industrial knowledge reuse, existing methods face significant limitations:

Semantic Fragmentation: Industrial documents contain nested references, cross-document citations, and scattered quantitative data (tables, figures) that traditional single-hop retrieval often misses.
Static Retrieval: Existing multi-hop RAG methods (e.g., fixed-iteration strategies or rigid path selection) struggle to parse dynamic semantic associations across sections.
Hallucination and Accuracy: Standard LLMs suffer from hallucinations in vertical domains, and current RAG systems lack the adaptability to iteratively refine queries to gather all necessary evidence before generating an answer.
Efficiency vs. Quality Trade-off: Current approaches often fail to balance retrieval completeness with computational efficiency, leading to redundant retrievals or premature termination.

2. Methodology: RLo-RAG

The authors propose RLo-RAG (Reinforcement Learning with LoRA-enhanced Multi-Round Retrieval Augmented Generation), a framework designed to optimize the retrieval process through a Markov Decision Process (MDP).

Core Architecture

The system consists of three main components:

Query Retriever (Querier): A policy network fine-tuned via Reinforcement Learning (RL) to generate iterative search queries based on the current state.
Retriever: A standard cosine-similarity based retriever that fetches top-k chunks from the knowledge base.
Generator: An LLM that synthesizes the final answer once the retrieval process concludes.

Key Technical Components

A. Multi-Round Retrieval as an MDP

The retrieval process is formalized as a sequential interaction between an agent and the environment:

State Space ( $S$ ): Composed of the original user query ( $q_0$ ), historical retrieval information ( $HRI$ including past queries and retrieved chunks), and a set of unretrieved but relevant paragraphs ( $SP$ ).
Action Space: The generation of a refined query statement ( $q_i$ ) by the Querier.
Transition: The retriever fetches new chunks ( $p_i$ ) based on $q_i$ , updating the $HRI$ and removing retrieved items from $SP$ . The process continues until a "Process Calculator" determines the information is sufficient or the maximum rounds are reached.

B. Reinforcement Learning with LoRA

To avoid the high cost of full-parameter fine-tuning, the authors employ Low-Rank Adaptation (LoRA) to fine-tune the base LLM (ChatGLM3-6B) for the query generation task.

Training Strategy: The model is trained using the PPO (Proximal Policy Optimization) algorithm.
Objective: The policy network learns to maximize cumulative rewards by identifying information gaps and generating queries that bridge them, rather than relying on heuristic prompting.

C. Dynamic Reward Function

A novel, multi-dimensional reward function guides the RL agent, balancing quality and efficiency through adaptive weighting:

Similarity Reward ( $R_s$ ): Encourages retrieving chunks highly similar to the most relevant unretrieved content.
Redundancy Penalty ( $R_r$ ): Penalizes the retrieval of content similar to previously retrieved chunks (using Jaccard distance) to prevent loops.
Efficiency Reward ( $R_e$ ): Applies a time-decay factor to encourage completing tasks in fewer rounds.

Dynamic Weighting: The weights ( $\omega_s, \omega_r, \omega_e$ ) are adjusted via a curriculum learning strategy. Early training emphasizes quality (similarity), while later training shifts focus toward efficiency, ensuring the model learns a robust retrieval path.

D. Industrial Implementation

For real-world application, the paper proposes a layout-aware semantic chunking strategy. Instead of fixed-length truncation, the system uses document layout analysis to preserve context (sections, tables, figures) and constructs a Knowledge Graph (Neo4j) to link text, tables, and figures via explicit reference relationships (e.g., "as shown in Figure 3").

3. Key Contributions

Formalization of Multi-Round Retrieval: The paper is the first to formalize the complex multi-round retrieval problem in industrial documents as an MDP, moving beyond heuristic rules to a learned policy network.
RL-Driven Query Optimization: It introduces a reinforcement learning-based query generator (fine-tuned with LoRA) that dynamically refines search queries to maximize information entropy gain, outperforming static or prompt-based multi-hop methods.
Dynamic Reward Mechanism: The design of a composite reward function with a curriculum-based dynamic weighting mechanism effectively solves the exploration-exploitation dilemma, balancing retrieval precision with computational efficiency.
System Validation: The authors validate the approach through both public multi-hop reasoning datasets (HotPotQA, HoVer) and a prototype system applied to civil aircraft manufacturing documents, demonstrating improved cross-modal and cross-document reasoning.

4. Experimental Results

The proposed RLo-RAG was evaluated against single-round baselines (RAG, DPR) and advanced multi-round baselines (LeReT, Self-RAG, CoT-RAG) on HotPotQA and HoVer datasets.

Answer Generation Performance: RLo-RAG achieved the highest scores across all metrics. On HotPotQA, it reached an F1 score of 80.2% and EM of 66.9%, outperforming the next best model (CoT-RAG) by 1.8 percentage points in F1 and 1.4 in EM.
Retrieval Precision: In terms of retrieving the correct chunks (EM), RLo-RAG achieved 88.9% on HotPotQA, significantly surpassing CoT-RAG (79.2%) and Self-RAG (77.8%).
Ablation Studies:
- RL vs. Heuristic: Removing the RL module (replacing it with heuristic prompting) resulted in an 8.4-point drop in F1, confirming the necessity of the learned policy.
- RL Strategy: PPO outperformed A2C, DPO, and GRPO, attributed to its ability to value intermediate states and maintain policy stability via clipping.
- Weighting Strategy: The "smooth cooperative dynamic strategy" for reward weights yielded superior performance and stability compared to abrupt or single-dimension adjustments.

5. Significance and Claims

The paper claims that RLo-RAG provides a robust solution for the semantic fragmentation inherent in industrial knowledge management. By integrating reinforcement learning with parameter-efficient fine-tuning, the method enables LLMs to perform adaptive, multi-hop reasoning that traditional RAG systems cannot achieve.

The authors assert that their approach:

Enhances Semantic Integrity: By iteratively retrieving and aggregating scattered information (text, tables, figures), the system generates answers with higher accuracy and completeness.
Optimizes Efficiency: The dynamic reward mechanism prevents unnecessary retrieval rounds, making the system viable for real-time industrial applications.
Scalability: The use of LoRA allows for efficient adaptation to specific industrial domains without the computational cost of full model retraining.

The paper concludes that while the method shows significant promise, future work should focus on deeper multimodal fusion (beyond text-referencing) and handling non-standard query formulations through domain-specific term dictionaries.