🤖 AI

HIRA: A Human-in-the-Loop Retrieval-Augmented Cascade for Document Classification in Regulated Industries

HIRA is a training-free, on-premises retrieval-augmented cascade for regulated document classification that combines multi-modal retrieval with a locally hosted LLM verifier and human-in-the-loop feedback to achieve high accuracy while minimizing human review and eliminating the need for costly model retraining.

Original authors: Shangxuan Tian, Yanhui Chen, Carlos Queiroz

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shangxuan Tian, Yanhui Chen, Carlos Queiroz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the high-stakes worlds of banking, healthcare, and law, documents are the primary currency of trust. Every loan application, medical record, and legal contract must be sorted into the right category to ensure compliance, safety, and efficiency. For decades, the standard approach to this sorting problem has been to teach a computer by showing it thousands of labeled examples, a process known as training. However, this method hits a wall in regulated industries. The rules of the game are strict: sensitive documents cannot leave the secure environment where they are created, meaning they cannot be sent to the cloud to be processed by powerful, remote artificial intelligence. Furthermore, the specific categories needed for a bank's internal files are often unique and change frequently, making it impossible to gather the massive, pre-existing datasets required to train a new model from scratch. When a bank tries to update its sorting system, the process is so slow and expensive that it often becomes impractical, leaving institutions stuck with outdated tools or forced to rely on expensive human workers to read every single file.

Researchers Shangxuan Tian, Yanhui Chen, and Carlos Queiroz have proposed a different way forward, a system called HIRA, designed specifically to navigate these tight constraints. Instead of trying to retrain a complex model every time a new document type appears or a rule changes, HIRA acts more like a highly organized librarian who learns by doing. The system operates in two stages. First, it scans a new document and immediately searches its internal memory for similar examples it has seen before. It looks at the text, the layout, and even the visual shape of the page to find the best matches. If the system finds a clear match, it sorts the document instantly. If the document is confusing or looks too much like several different categories, the system does not guess; instead, it passes the file to a second, more careful stage. This stage uses a powerful, locally hosted artificial intelligence to review the document alongside the best examples the system found, asking it to make a final decision. If even this second stage is unsure, the document is sent to a human expert for correction.

The true innovation lies in what happens after that human correction. In traditional systems, a human fixing a mistake is just a one-time fix. In HIRA, every correction becomes a permanent lesson. When a human corrects a document, the system saves that example in its memory with a special weight, making it more likely to be recognized next time a similar document arrives. It also updates a map of confusion, learning exactly which categories tend to get mixed up and what specific words or visual features distinguish them. This allows the system to improve its accuracy over time without ever needing to retrain its core software or update its underlying mathematical weights. The system gets smarter simply by remembering its mistakes and the specific details that resolved them.

The researchers tested this approach on two very different sets of data. The first was a public benchmark of 2,737 documents ranging from letters to scientific reports. On this set, the system started with a small number of examples and processed a stream of new documents. By asking humans to correct only about a quarter of the incoming files, the system's accuracy rose to match the performance of a hypothetical "oracle" that had seen every single document with its correct label from the very beginning. This was a significant leap, outperforming a standard artificial intelligence that tried to guess without any help by a wide margin, while using far fewer computer resources. The system managed to send only about 40% of the documents to the second, more expensive AI stage, and even fewer to the human reviewers, proving that it could filter out the easy cases efficiently.

The second test was a real-world challenge involving a private collection of 30,233 trade-finance documents from a major bank. This environment was much harder, with 80 different categories, many of which were nearly identical to one another, such as different types of invoices or certificates. The data was heavily skewed, with some categories having thousands of examples and others having very few. Despite these difficulties, the system processed the entire stream of documents and asked human workers to correct only 1,945 of them, which is just 6.4% of the total. This small amount of human effort was enough to boost the system's accuracy from a modest level to a highly reliable one. The researchers found that as the system learned from these corrections, the number of documents needing human review dropped sharply, settling at a low rate as the system's memory grew richer.

This work demonstrates that in regulated industries, the path to high accuracy does not necessarily require massive datasets or constant software retraining. By combining a smart search mechanism with a local artificial intelligence and a feedback loop that learns from human corrections, it is possible to build a system that adapts to new documents and changing rules on its own. The system respects the strict requirement that data never leaves the secure environment, yet it achieves performance levels that rival systems trained on vast amounts of data. The results suggest that for industries where data privacy and compliance are paramount, a system that learns selectively and remembers its corrections is a practical and powerful alternative to the traditional methods of training artificial intelligence.

Technical Summary: HIRA – A Human-in-the-Loop Retrieval-Augmented Cascade

Problem Setting
The paper addresses document classification in regulated industries (e.g., finance, healthcare, legal), where standard supervised learning and cloud-based Large Language Model (LLM) approaches face four critical constraints:

Cold Start & Drift: Taxonomies are bespoke and data is confidential, preventing the creation of large labeled training sets. Furthermore, document templates drift over time, requiring frequent model updates.
Data Residency: Regulatory requirements (e.g., bank secrecy) forbid documents from leaving the on-premises environment, ruling out cloud-hosted LLM APIs.
Compliance-Bound Retraining: Retraining and redeploying models triggers lengthy governance reviews, making weight-based adaptation impractical for the cadence of data drift.
Auditability & Cost: Decisions must be explainable via precedents, and human review capacity is a fixed, expensive bottleneck. The goal is to maximize accuracy per human label rather than raw accuracy.

Methodology: The HIRA System
HIRA (Human-in-the-Loop Retrieval-Augmented Cascade) is a training-free, on-premises pipeline designed to operate without updating model weights. It functions as a two-stage cascade with a feedback loop:

Tier 1: Multi-Modal Retrieval with RRF Fusion
The system processes a document through three parallel retrievers over a current index:
1. BM25 over raw OCR text.
2. Dense sentence embeddings (MiniLM) over OCR.
3. Image-level representations (SigLIP2).
  These ranked lists are fused using Weighted Reciprocal Rank Fusion (RRF). Per-class scores are aggregated, and if the top prediction exceeds a calibrated threshold ( $\tau_{t1}$ ), the document is classified immediately.
Tier 2: LLM Contrastive Verification
Documents failing the Tier-1 gate are escalated to a locally hosted LLM verifier (DeepSeek-R1-Distill-Qwen-32B). The verifier receives:
- The document's OCR text.
- Top- $k$ retrieved exemplars.
- Label descriptions.
- Distinguishing terms: Specific tokens mined from past misclassifications of high-confusion pairs (identified via a confusion graph).
  The LLM outputs a label, an ordinal confidence bucket, and a justification. If confidence exceeds a second threshold ( $\tau_{t2}$ ), the label is accepted; otherwise, the document enters the Human-in-the-Loop (HITL) queue.
EDRM: Evolving Document Relevance Memory
This is the system's adaptive memory, updated by every human correction without changing model weights:
1. Margin-Weighted Exemplar Insertion: Corrections are inserted into the retrieval index with weights based on the "margin" of the initial misclassification. Harder errors (negative margins) receive higher weights. Entries decay over time to handle concept drift.
2. Dirichlet-Smoothed Confusion Graph: A matrix tracks confusion between predicted and true classes. High-confusion pairs trigger the extraction of distinguishing terms, which are injected into future Tier-2 prompts to help the LLM differentiate similar categories.

Key Contributions

Cold-Start Adaptation: HIRA achieves high performance with minimal initial annotation (e.g., 10 documents per class) and adapts to long-tail distributions without retraining.
Training-Free RAG Cascade: The system outperforms LLM zero-shot baselines by combining retrieval with a local verifier, avoiding the need for fine-tuning.
EDRM Memory Mechanism: A dual-signal memory system that updates both the retrieval index (via margin weighting) and the verification prompt (via confusion graphs), enabling continuous improvement solely through data updates.

Experimental Results
The authors evaluated HIRA on two datasets: a private Financial-80 corpus (30,233 documents, 80 categories) and the corrected Tobacco-3482 benchmark (2,086 pool documents, 10 categories).

Financial-80 (Production):
- HIRA processed the full stream, requesting human correction for only 6.4% (1,945 documents).
- Macro-F1 improved from 0.6218 (LLM zero-shot baseline) to 0.8548.
- The human review queue rate dropped from ~18.8% to 6.2% as the index matured.
Tobacco-3482 (Benchmark):
- HIRA achieved a Macro-F1 of 0.9423, which is 17.4 percentage points higher than the locally hosted LLM zero-shot baseline (0.7683).
- The system invoked the LLM verifier for only ~40% of documents, reducing LLM calls by 60% compared to zero-shot.
- With 518 human corrections (24.8% of the pool), HIRA matched the performance of a "fully labelled pool oracle" (where all documents are indexed with ground truth).
- In a comparison with a fine-tuned DiT model using the same annotation budget (618 labels), HIRA outperformed the supervised model (0.9423 vs. 0.9193) without requiring GPU-intensive training or governance reviews.

Significance and Claims
The paper claims that HIRA provides a practical alternative to repeated model retraining for document classification in regulated deployments. By leveraging selective human feedback and retrieval-memory adaptation, the system:

Reduces Operational Costs: It significantly lowers the volume of documents requiring human review and reduces LLM inference costs by routing only ambiguous cases to the verifier.
Ensures Compliance: It operates entirely on-premises, keeping data within the deployment environment and avoiding the governance overhead associated with model weight updates.
Adapts to Drift: It handles long-tail categories and near-synonym confusion through the EDRM mechanism, continuously improving its retrieval and verification capabilities as corrections are accumulated.

The authors conclude that this combination of multimodal retrieval, local verification, and selective HITL is sufficient to deploy robust classification systems in regulated industries without the need for traditional supervised fine-tuning cycles.

Technical Summary: HIRA – A Human-in-the-Loop Retrieval-Augmented Cascade

More like this