🤖 AI

RAPT: Retrieval-Augmented Post-hoc Thresholding for Multi-Label Classification

This paper introduces RAPT, a model-agnostic, retrieval-augmented post-hoc thresholding wrapper that dynamically adapts label selection thresholds for multi-label classification by leveraging similar historical cases, thereby significantly outperforming static baselines and few-shot LLMs in both accuracy and computational efficiency.

Original authors: Lasal Jayawardena, Nirmalie Wiratunga, Ikechukwu Nkisi-Orji, Darren Nicol

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lasal Jayawardena, Nirmalie Wiratunga, Ikechukwu Nkisi-Orji, Darren Nicol

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian in a massive, chaotic library. Your job is to look at a new book and decide which categories it belongs to—like "History," "Cooking," or "Science."

In the old way of doing things, you’d use a rigid rule: "If the book looks 50% like a history book, tag it as History." But this is problematic. Sometimes a book is 60% history but also 40% cooking (a cookbook about the Roman Empire). A rigid 50% rule might miss the cooking tag entirely, or worse, tag a simple recipe book as "History" because it mentions ancient ingredients.

This paper introduces RAPT, which stands for Retrieval-Augmented Post-hoc Thresholding. Think of RAPT not as a new librarian, but as a smart assistant who stands next to your existing librarian and whispers advice before they make their final decision.

The Problem: The "One-Size-Fits-All" Trap

Most AI systems that sort documents use a global threshold (like that 50% rule). This is brittle.

Noise: If a document is scanned poorly (OCR noise), the AI gets confused.
Imbalance: Some categories are rare; others are common. A single rule doesn’t work for both.
Context: A document about "Waste" might be a legal form in one context and a casual email in another. A fixed rule can’t tell the difference.

The Solution: RAPT’s "Neighborly Advice"

RAPT uses a technique called Case-Based Reasoning. Here’s how it works, step-by-step:

The First Guess: The main AI model (the "librarian") looks at a new document and gives it scores for every possible label. It also creates a "digital fingerprint" (embedding) of the document.
Finding Neighbors: RAPT looks at the library’s history. It finds the 10 most similar past documents (neighbors) based on that digital fingerprint.
Learning from Mistakes: RAPT looks at how those neighbors were actually labeled in the past. It asks: "Did the AI over-tag or under-tag those similar documents? What was the right number of tags for that type of document?"
Adjusting the Threshold: Instead of using the rigid 50% rule, RAPT adjusts the cutoff for this specific document based on its neighbors.
- Example: If the 10 most similar documents usually had 3 tags, RAPT tells the AI, "Hey, don’t just pick the top 1. Look for the top 3."
- Example: If the similar documents were hard to classify, RAPT might say, "Be stricter. Only pick labels with very high confidence."

Why It’s Better: The "Smart Wrapper"

RAPT is a wrapper. You don’t need to retrain your expensive AI model. You just plug RAPT on top of whatever AI you already have. It works with:

Metric Learners: AI models that are good at finding similarities.
Transformers: Large language models (like BERT) that understand text deeply.

The Results: Fast, Cheap, and Accurate

The researchers tested RAPT on industrial documents (from a waste management company) and six public datasets (news, legal, medical, etc.).

Accuracy: RAPT consistently beat the old "fixed rule" methods. In the industrial test, it achieved an 87% Macro-F1 score (a measure of accuracy across all categories, especially the rare ones).
Efficiency: This is the big win. Compared to using a massive Large Language Model (LLM) like Qwen to do the same job:
- RAPT was 115 times faster.
- RAPT used 13.5 times less memory.
- RAPT was 2 times more accurate than the LLM in this specific task.

The Creative Analogy: The Thermostat

Think of your AI classifier like a thermostat in a house.

Old Way (Global Threshold): You set the thermostat to exactly 70°F. But in winter, 70°F feels cold. In summer, 70°F feels hot. The house is uncomfortable.
RAPT Way (Dynamic Thresholding): RAPT is like a smart sensor that checks what the weather was like in similar houses nearby. If the neighbors are all heating up because it’s a cold snap, RAPT adjusts your thermostat to 72°F. If it’s a mild day, it keeps it at 68°F. It adapts the "comfort level" (threshold) based on local context, not a rigid global rule.

In Summary

RAPT is a lightweight, smart layer that sits on top of existing AI classifiers. It looks at similar past examples to decide how strict or loose the AI should be when tagging a new document. It makes the AI more accurate without needing to retrain it, and it does so much faster and cheaper than using giant, expensive language models.

Technical Summary: RAPT for Multi-Label Classification

Problem Statement
Industrial multi-label document understanding pipelines rely on scoring candidate labels and applying thresholds to generate a final label set for each document. This selection step is critical, as errors propagate to downstream information extraction and verification tasks. However, global score thresholds are brittle in practice due to OCR noise, label imbalance, instance-dependent label cardinality, and asymmetric error costs. As document formats evolve, maintaining static thresholds becomes difficult. The core challenge is converting per-label activation scores into a variable-size label set when the semantics of a given score value differ across regions of the instance space and across labels.

Methodology
The authors propose RAPT (Retrieval-Augmented Post-hoc Thresholding), a model-agnostic wrapper that applies case-based reasoning (CBR) to adapt label set selection post-hoc, without retraining the underlying classifier. RAPT requires only two outputs from any backbone predictor: document representations (embeddings) for similarity search and per-label confidence scores.

The methodology follows a standard CBR cycle:

Case Base Construction: A case base is constructed from the training set. Each case consists of an instance $x$ , its embedding $z$ , the model’s predicted score vector $\hat{Y}$ , and the ground truth label vector $Y$ . Thresholding evidence is derived automatically from the discrepancy between $\hat{Y}$ and $Y$ , requiring no additional annotation.
Case Retrieval: For a query document $q$ , the system retrieves the $k$ most similar cases from the case base using cosine similarity on embeddings. The similarity scores serve as importance weights.
Local Neighborhood Adaptation: The wrapper adapts the query’s score vector by transferring corrective evidence from retrieved neighbors. For each neighbor $i$ $i$ , a residual-based correction is calculated:
- Model Adaptation ( $\Delta_{1,i}$ ): Aligns the neighbor’s score profile with the query’s own model output ( $\hat{Y}_q - \hat{Y}_i$ ).
- Residual Correction ( $\Delta_{2,i}$ ): Captures the model’s error on the neighbor ( $Y_i - \hat{Y}_i$ ) and transfers this error pattern to the query.
  The final adapted score vector $Y_q$ is a similarity-weighted aggregation of these corrected scores.
Dynamic Thresholding: The adapted scores are converted to binary labels using one of two neighborhood-driven strategies:
- AvgCount: Estimates the expected number of active labels ( $k^*$ ) by computing a similarity-weighted average of the neighbors’ label cardinalities. The threshold is set to yield approximately $k^*$ positive labels.
- Rank Calibration: Sets the threshold based on the weighted average of the minimum confidence scores assigned to positive labels in the retrieved neighborhood, reflecting local decision boundaries.

Key Contributions

A framework for constructing a case base from training data that infers threshold values from ground truth labels and model scores, eliminating the need for extra data or human annotation.
Neighborhood-driven dynamic thresholding strategies (AvgCount and Rank Calibration) that utilize residual error information from retrieved cases to adapt the multi-label score vector prior to thresholding.
Evaluation demonstrating consistent performance gains over global and label-wise static thresholding baselines across an industrial dataset and six public benchmarks, with comparisons against few-shot Large Language Models (LLMs).

Results
The evaluation involved an industrial dataset from WM Nicol (transportation and waste management) and six public benchmarks (Reuters-21578, EUR-Lex, Event Classification, MIMIC-III, OHSUMED, and AmazonCat-13K). Backbones included episodic metric learners (Matching Networks, Prototypical Networks, Relation Networks, SNAIL) and fine-tuned transformers (BERT, DeBERTa).

Industrial Performance: On the WM Nicol dataset, RAPT achieved its best performance with metric learners, specifically Prototypical Networks combined with RAPT# (AvgCount), reaching a 0.87 Macro-F1. This represented a 10.8% relative gain over a fixed 0.5 threshold and an 11.1% gain over a globally optimized threshold. Fine-tuned transformers averaged 0.775 Macro-F1.
Comparison with LLMs: RAPT configurations achieved approximately 2× the Macro-F1 of the best few-shot LLM baseline (Qwen3-32B, K=5). Furthermore, RAPT required at least 115× less inference time (38–236 ms vs. 27–48 s per document) and 13.5× less GPU memory (≤2.23 GB vs. >30 GB).
Generalization: Across 175 model–dataset combinations on public benchmarks, RAPT outperformed the best static threshold baseline in 130 cases (74.3%), with an average improvement of +0.093 Macro-F1. Metric learners showed win rates of 75–100% across all datasets.
Failure Modes: Analysis of industrial misclassifications identified two primary failure modes: co-occurrence-driven false positives (where spurious labels frequently co-occur with true labels in the neighborhood) and format–content ambiguity (where embeddings prioritize superficial formatting over semantic content).

Significance
The paper positions RAPT as a deployment-oriented solution for industrial document workflows. Its significance lies in its ability to improve label set selection robustness without retraining classifiers or altering their architecture. By leveraging case-based reasoning, RAPT adapts to local instance characteristics, offering a lightweight, efficient alternative to static thresholding and computationally expensive LLM-based approaches. The authors highlight its suitability for high-throughput environments where document formats and label distributions evolve over time.