💻 computer science

Corrective Retrieval-Augmented Generation Adaptation in Closed Domains for Development of Intelligent Human Resource Management Systems

This paper presents a Design Science Research study on a closed-domain Smart HR system for SMEs that demonstrates a Corrective RAG adaptation with same-chapter hard-negative training significantly improves answer faithfulness over standard RAG, despite a trade-off in overall accuracy.

Original authors: Devi Willieam Anggara, Marcellino Joestiawan, Mohd Shafry Mohd Rahim, Ibrahim Salim Sulaiman Al Farsi, Fetty Fitriyanti Lubis, Dion Tanjung

Published 2026-09-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Devi Willieam Anggara, Marcellino Joestiawan, Mohd Shafry Mohd Rahim, Ibrahim Salim Sulaiman Al Farsi, Fetty Fitriyanti Lubis, Dion Tanjung

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern workplace, the ability to make sound decisions about people is the difference between a thriving company and one that struggles to survive. For small and medium-sized businesses, which form the backbone of many economies, this task is often made harder by a lack of formal systems. Managers frequently rely on memory or informal advice rather than a trusted, written guide when hiring, evaluating performance, or planning careers. To help, researchers have turned to a type of artificial intelligence that can read vast amounts of text and answer questions based on what it finds. This technology, known as retrieval-augmented generation, works by searching a library of documents to find relevant information before an AI model writes a response. This approach is designed to stop the AI from making things up, a common problem where the computer sounds confident but is actually inventing facts. However, standard versions of this technology have a flaw: they often grab documents that look related but do not actually answer the specific question, leading to answers that are vague or misleading.

A team of researchers from Indonesia and Malaysia set out to fix this problem for the specific needs of human resource management. They built a smart system designed to help small businesses manage their employees by answering questions strictly from a single, authoritative book: the 16th edition of Armstrong's Handbook of Human Resource Management Practice. This book is a standard reference for the field, containing established rules and best practices. The researchers wanted to know if they could make the AI smarter by adding two specific improvements. The first was a "correction" step that checks if the information found is actually useful before using it. The second was a way to organize the book's pages with extra labels, like tags or categories, to help the search find the right pages more easily. They tested these ideas by creating a system that could handle three critical areas of human resources: hiring new staff, managing employee performance, and planning career growth.

The researchers constructed a digital library from the handbook, breaking it down into small, manageable sections. They then built a multi-layered system to process questions. One layer of the system was designed to add descriptive tags to the text, such as identifying whether a section was about a procedure, a concept, or a comparison. Another layer acted as a strict gatekeeper. When the system retrieved a piece of text to answer a question, this gatekeeper evaluated whether the text truly answered the query or if it was just vaguely related. If the text was judged to be incorrect or unclear, the system did not simply guess; it went back to the library, rephrased the question, and searched again within the same trusted book. This ensured that every answer remained grounded in the official source, never drifting into unverified internet searches.

To see if these changes actually worked, the team ran a series of rigorous tests. They created 270 specific questions covering the three human resource domains, ranging from how to structure a job interview to what criteria should be used for a promotion. They then compared three different versions of the system. The first was a basic version that simply searched and answered without any checks. The second version used the extra tags to organize the search but still lacked the gatekeeper. The third version used both the correction mechanism and the gatekeeper. The results showed a clear winner. The system with the correction mechanism and the gatekeeper produced answers that were significantly more faithful to the source material. In other words, the answers it gave could be traced directly back to the specific sentences in the book that supported them. The system successfully corrected its own mistakes about 28 percent of the time, re-searching and refining its path to find the right information.

However, the study also revealed a surprising limitation. The version that used only the extra tags to organize the search, without the correction gatekeeper, did not improve the quality of the answers. In fact, it performed no better than the basic system and, in some cases, was slightly worse. This finding suggests that simply organizing the information better is not enough if the system cannot verify that the retrieved information is actually the right answer. The researchers also noticed a trade-off. While the corrected system was more faithful to the source, it sometimes scored slightly lower on a general measure of how "complete" an answer seemed. This happened because the system was so careful to stick only to what was explicitly in the book that it occasionally left out helpful context that a human might have added from general knowledge. For a small business, however, the researchers argue that sticking strictly to the verified source is more important than adding extra details that might be wrong.

The study concludes that for specialized fields like human resources, where accuracy is critical, the ability to check and correct the search results is far more valuable than simply organizing the data better. The researchers demonstrated that by building a system that refuses to guess and instead re-evaluates its findings, they could create a tool that small businesses could trust to make decisions about hiring and promotions. The work proves that the most effective way to build a reliable AI assistant for a specific domain is not just to feed it more data, but to give it a mechanism to question its own sources. This approach ensures that the advice given to a business owner is not just a confident-sounding guess, but a verified fact drawn from a trusted guide.

Technical Summary: Corrective Retrieval-Augmented Generation Adaptation in Closed Domains for Intelligent HR Management Systems

Problem Statement

Small and Medium Enterprises (SMEs) in Indonesia face significant challenges in Human Resource Management (HRM), particularly in recruitment, performance management, and career development. These challenges stem from informal work patterns, a lack of standardized criteria, and limited access to professional HR decision-support systems. While traditional Human Resource Information Systems (HRIS) automate transactional tasks, they fail to provide the conceptual knowledge and adaptive decision support required for complex HR queries.

Retrieval-Augmented Generation (RAG) offers a potential solution by grounding Large Language Model (LLM) outputs in external knowledge, thereby reducing hallucinations. However, standard "Basic RAG" architectures exhibit two critical limitations in the HR domain:

Lack of Quality Verification: Basic RAG feeds retrieved documents directly into the generator without verifying their relevance, risking the propagation of low-quality or ambiguous context into critical HR decisions.
Shallow Vector Representations: Standard vector embeddings capture only surface-level semantics, often failing to handle comparative or predictive queries effectively, which are common in HR contexts (e.g., comparing competency frameworks or assessing promotion readiness).

Furthermore, existing advanced RAG frameworks like Corrective RAG (CRAG) are designed for open-domain settings, relying on web search as a fallback when retrieval fails. This is unsuitable for closed-domain SME applications where answers must be strictly auditable and confined to a specific, validated knowledge source (e.g., Armstrong's Handbook of Human Resource Management Practice).

Methodology

The research employs the Design Science Research Methodology (DSRM) to develop and evaluate a multi-agent Smart HR system. The system is designed as a closed-domain Question Answering (QA) system grounded exclusively in Armstrong's Handbook (16th edition).

System Architecture

The system utilizes a two-tier multi-agent pattern:

Domain Orchestrator: Classifies user queries into three domains (Recruitment, Performance Management, Career Development) and manages conversational context.
CRAG Sub-workflow: A specialized retrieval tool invoked by the orchestrator to perform retrieval, quality evaluation, and correction.

Technical Components

The study implements two primary technical layers:

Structured RAG (SRAG) Layer (Indexing Time): Enriches text chunks with LLM-generated metadata (topics, chunk type, knowledge-graph triples, and semantic tags) before embedding. Unlike the original SRAG design for financial data, this adaptation removes "sentiment" metadata and re-maps "chunk types" to HR-specific categories (conceptual, procedural, comparative, evaluative, and example).
Closed-Domain CRAG Adaptation (Inference Time):
- Retrieval Evaluator: A fine-tuned flan-t5-large model acts as a retrieval evaluator. It is trained using a same-chapter hard-negative strategy, where chunks from the same chapter as a positive example but irrelevant to the specific query are used as negative training data. This forces the model to learn query-level relevance rather than just chapter-level topic similarity.
- Corrective Mechanism: The evaluator scores retrieved chunks. If a chunk is judged INCORRECT (low relevance), the system does not fall back to the open web. Instead, it reformats the query with a fixed HRM-context prefix and re-retrieves from the same curated Qdrant knowledge base. If AMBIGUOUS, it re-retrieves with a different prefix and merges results.

Experimental Design

An ablation study was conducted using 270 synthetic queries (90 per domain) to isolate the contributions of the proposed components. Three variants were tested:

Basic RAG: Baseline with plain indexing and no evaluator.
SRAG-only: Uses metadata-enriched indexing but no retrieval evaluator.
Closed-Domain CRAG: Uses plain indexing but includes the T5 evaluator and the re-retrieval mechanism.

The three variants were run against the same 270-query dataset, producing 810 total inferences for the analysis. Evaluation metrics included RAGAS (Faithfulness, Answer Relevancy, Context Precision, Context Recall) and LLM-as-Judge (Accuracy, Completeness, Hallucination). Statistical significance was determined using the Wilcoxon signed-rank test.

Key Contributions

The paper presents two specific technical contributions realized as artifacts:

C1: Closed-Domain CRAG Adaptation: A modification of the standard CRAG architecture that replaces the open-web fallback with a re-retrieval mechanism against the same curated knowledge base. This ensures that all answers remain auditable and confined to the standardized HRM source.
C2: Same-Chapter Hard-Negative Evaluator: A retrieval evaluator trained specifically to distinguish between chunks that share a topic (chapter) and chunks that actually answer the query. This is a prerequisite for the corrective mechanism to function reliably in a closed domain where nearly all retrieved content is topically relevant.

Results

The ablation study yielded the following findings:

Hypothesis 1 (H1) Confirmed: The Closed-Domain CRAG adaptation (C1+C2) achieved significantly higher Faithfulness than Basic RAG ( $p < 0.001$ , Cohen's $d = 0.457$ , medium effect). This improvement was consistent across all three HR domains. The mean Corrective Action Rate (CAR) was 0.28, indicating the evaluator successfully triggered re-retrieval for roughly 28% of queries.
Hypothesis 2 (H2) Rejected: SRAG-only did not significantly improve Faithfulness over Basic RAG ( $p = 0.388$ , $d = 0.071$ ). In some domains, SRAG-only performed marginally worse. This suggests that metadata enrichment alone, without a relevance-evaluation layer, does not improve answer quality in this closed-domain context and may even increase hallucination risk by providing more varied but unverified context.
Faithfulness vs. Accuracy Trade-off: Both augmented variants (CRAG and SRAG) showed significantly lower LLM-as-Judge Accuracy compared to Basic RAG. The authors attribute this to the definition of the metrics: Faithfulness measures traceability to retrieved context, while Accuracy measures holistic factual correctness. CRAG's "Knowledge Refinement" trims context to only what is verified, increasing traceability but potentially reducing the breadth of information available to the generator, thus lowering the holistic accuracy score.

Significance and Claims

The paper claims to provide an auditable closed-domain CRAG artifact and empirical evidence regarding the unequal contributions of different RAG components to answer quality.

Primary Driver of Quality: The study concludes that the corrective mechanism (C1) is the primary contributor to improved Faithfulness, while the hard-negative evaluator (C2) is the enabling component that allows the corrective mechanism to operate reliably.
Limitation of Metadata Alone: The research demonstrates that in a closed-domain system with a curated knowledge base, investing in metadata enrichment (SRAG) without a relevance-evaluation layer is insufficient to improve answer quality.
Architectural Guidance: The findings offer guidance for system architects: for domains requiring high auditability and strict adherence to a source (like HRM in SMEs), a closed-domain CRAG adaptation is superior to Basic RAG or SRAG-only, despite the trade-off in holistic accuracy scores.

The authors maintain a modest stance, acknowledging limitations such as the use of synthetic queries, the lack of validation by human HR practitioners, and the fact that the interaction between SRAG and CRAG (e.g., whether SRAG strengthens the evaluator) remains untested. Future work is proposed to explore hybrid configurations and larger sample sizes.