💬 NLP

ConceptRM: The Quest to Mitigate Alert Fatigue through Consensus-Based Purity-Driven Data Cleaning for Reflection Modelling

This paper introduces ConceptRM, a novel method that leverages co-teaching and consensus-based analysis on perturbed datasets with minimal expert annotations to effectively identify reliable negative samples from noisy production data, thereby significantly improving reflection models' ability to mitigate alert fatigue.

Original authors: Yongda Yu, Lei Zhang, Xinxin Guo, Minghui Yu, Zhengqi Zhuang, Guoping Rong, Haifeng Shen, Zhengfeng Li, Boge Wang, Guoan Zhang, Bangyu Xiang, Xiaobin Xu

Published 2026-02-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yongda Yu, Lei Zhang, Xinxin Guo, Minghui Yu, Zhengqi Zhuang, Guoping Rong, Haifeng Shen, Zhengfeng Li, Boge Wang, Guoan Zhang, Bangyu Xiang, Xiaobin Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Crying Wolf" of Code Review

Imagine you are a software developer. You have a helpful robot assistant (an AI) that reviews your code every time you make a change. Its job is to find bugs and suggest improvements.

At first, this is great. But soon, the robot starts screaming "ALERT!" at everything.

"You used a comma here!" (Trivial)
"This variable name is slightly long!" (Nitpick)
"I think this code might break the universe!" (Hallucination/Fake)

You get 100 alerts a day, but 90 of them are nonsense. You start ignoring the robot. This is called "Alert Fatigue." It's like a fire alarm that goes off every time you toast bread. Eventually, when a real fire starts, you don't even look up because you're tired of the noise.

The Old Solution: The "Human Filter" (And Why It Failed)

To fix this, companies tried to train a second AI (a "Reflection Model") to act as a filter. This filter's job is to read the robot's alerts and decide: "Is this real? Or is it garbage?"

To train this filter, they fed it data from real developers. But here's the catch: Real developers are messy.

Sometimes a developer ignores a bad alert just because they are in a rush.
Sometimes they accept a bad suggestion because they are tired.
The data is full of "noise" (mistakes in the labels).

Training a filter on messy data is like trying to teach a student to spot fake news by giving them a textbook written by someone who doesn't know the difference between truth and lies. The filter learns the wrong lessons.

The New Solution: ConceptRM (The "Consensus Council")

The authors of this paper, ConceptRM, came up with a clever way to clean the data without hiring an army of expensive experts to check every single alert.

Think of it like a jury system or a group of detectives solving a mystery.

Step 1: The "Noise-Doping" Experiment

Instead of trying to find the "perfect" data, they intentionally messed up the data on purpose.

Imagine they have a pile of "Good Alerts" (Accept) and "Bad Alerts" (Reject).
They create 6 different training groups.
In Group A, they mix in a tiny bit of "Ignore" (ambiguous) alerts into the "Good" pile.
In Group B, they mix in a lot of "Ignore" alerts.
In Group C, they mix in even more... and so on.

It's like giving 6 different chefs the same recipe but telling them to add varying amounts of salt. Some will make it bland; some will make it salty.

Step 2: The "Co-Teaching" (The Detectives)

They train 6 different AI models (the detectives) on these 6 different "salty" datasets.

The model trained on the "super salty" data becomes very conservative. It only screams "BAD!" if it's 100% sure. It hates making mistakes.
The model trained on the "bland" data becomes aggressive. It screams "BAD!" at almost anything suspicious.

Step 3: The "Consensus" (The Verdict)

Now, they bring all 6 detectives together to vote on the original messy data.

Strict Consensus (The Hardline Judge): If all 6 detectives agree an alert is bad, then it's bad. If even one says "Maybe it's okay," they let it pass. This creates a filter that rarely makes mistakes (low False Positives).
Majority Vote (The Democratic Judge): If 4 out of 6 say it's bad, then it's bad. This catches more bad alerts but might accidentally block a few good ones.

By letting these different models vote, they can figure out which alerts are truly garbage and which ones were just mislabeled by tired humans. They effectively "clean" the data by finding the consensus.

The Result: A Smart Gatekeeper

The final result is a "Gatekeeper" AI that is incredibly good at its job:

It stops the noise: It catches the fake alerts and the trivial nitpicks.
It protects the signal: It rarely blocks a real bug report (unlike other methods that are too aggressive).
It saves money: It only needed a tiny amount of expert human help (about 100 samples) to get started, rather than checking thousands of alerts manually.

The Big Picture Analogy

Imagine a noisy party where everyone is shouting.

The Problem: You can't hear the person you are trying to talk to because of the background noise.
The Old Way: You ask a friend to shout "Quiet!" at everyone. But your friend is drunk and shouts at the wrong people.
ConceptRM: You ask 6 different people to stand in different corners of the room. Each one has a slightly different view of who is shouting. You ask them to vote: "Is this person actually shouting, or are they just talking?" If they all agree it's shouting, then you tell them to be quiet.

Why this matters:
This method allows AI to be a helpful assistant rather than a annoying nuisance. It ensures that when the AI says "There's a fire," you actually stop what you're doing and look, because you know it's not just a piece of toast.

1. Problem Statement

The paper addresses the critical issue of "Alert Fatigue" in applications utilizing intelligent agents, specifically in LLM-based code review.

The Phenomenon: Intelligent agents generate a massive volume of review comments, many of which are false positives (hallucinations, trivial nitpicks, or low-value suggestions).
The Consequence: The overwhelming volume of noise desensitizes human developers, causing them to overlook critical vulnerabilities or valid issues.
The Challenge: Mitigating this requires a Reflection Model (a filter) trained to intercept false alerts. However, training such a model is hindered by noisy training data.
- Data is collected from production environments where user feedback (Accept/Reject/Ignore) is often subjective, inconsistent, or influenced by tight schedules (e.g., developers marking valid comments as "Ignore" to save time).
- Manual annotation to clean this data is prohibitively expensive and slow.
- Standard Learning with Noisy Labels (LNL) techniques (like Co-teaching) fail in the era of Large Language Models (LLMs) because fine-tuning starts from identical pre-trained weights, causing models to converge prematurely rather than maintaining the necessary diversity for noise filtering.

2. Methodology: ConceptRM

The authors propose ConceptRM (Consensus-based Noise Cleaning and Purity-driven co-Teaching for Reflection Modelling). Instead of relying on parameter initialization differences, ConceptRM shifts the source of model divergence to the data space.

Core Components:

Purity-Driven Objective:
- The goal is not just accuracy, but maximizing the Purity-driven Intercept Efficacy (PIE), defined as the ratio of Intercept Rate (IR) to False Positive Rate (FPR).
- The system prioritizes minimizing FPR (blocking valid comments) while maintaining a minimum IR (catching bad comments).
Noise-Doping-Based Data Cleaning (Perturbation):
- The method creates $K$ perturbed training datasets ( $D_1$ to $D_K$ ) by varying the ratio of ambiguous "Ignore" samples injected into the positive class.
- Rationale: "Ignore" samples occupy a semantic grey area between "Accept" and "Reject." By mixing them at different ratios (0% to 50%), the models are forced to learn distinct decision boundaries:
  - High-noise models become conservative (low FPR, low IR) to avoid misclassifying ambiguous "hard positives."
  - Low-noise models become aggressive (high FPR, high IR).
Consensus-Based Biased Relabeling:
- Multiple models ( $M_1...M_K$ ) are trained in parallel on these perturbed datasets.
- A Consensus Strategy is applied to select the optimal model combination and relabel the entire raw dataset. Two strategies are tested:
  - Strict Consensus (SC): A sample is labeled "Negative" (intercepted) only if all models agree. This prioritizes ultra-low FPR.
  - Majority Vote (MV): A sample is labeled based on the majority. This balances IR and FPR.
- Grid Search: The system searches for the optimal subset of models and consensus strategy that maximizes PIE on a small, expert-annotated validation set (anchors).
Final Refinement:
- The selected optimal combination is used to re-inference (relabel) the full raw dataset.
- A final Reflection Model is trained on this cleaned, high-purity corpus.

3. Key Contributions

Novel Data-Centric Divergence: Unlike traditional Co-teaching which relies on random weight initialization (ineffective for fine-tuning LLMs), ConceptRM induces model diversity through data perturbation (varying noise ratios).
Purity-Driven Optimization: It introduces a specific objective function (PIE) that explicitly optimizes for the trade-off between catching bad alerts and preserving good ones, addressing the asymmetric cost of errors in code review.
Low-Resource Efficiency: The method demonstrates that a small amount of expert annotation (as few as 100 samples) is sufficient to guide the consensus mechanism and achieve high performance, making it practical for industrial deployment.
Flexible Deployment: The framework allows users to tune the system's behavior (Conservative vs. Aggressive) simply by switching consensus strategies (SC vs. MV) without retraining the base models.

4. Experimental Results

The method was evaluated on internal industrial data (74k+ samples) and an open-source GitHub dataset across 10 programming languages.

Performance vs. Naive Fine-Tuning:
- ConceptRM significantly reduced the False Positive Rate (FPR).
- On the internal dataset, ConceptRM-SC-1k achieved a PIE of 3.71, outperforming Naive Fine-Tuning (PIE 2.10) by 53.31%.
- On the GitHub dataset, it achieved a PIE of 4.42, outperforming Naive Fine-Tuning by 41.67%.
- FPR was reduced by up to 75.51% compared to naive approaches.
Performance vs. State-of-the-Art (SOTA) LLMs:
- ConceptRM (based on a 30B parameter model) outperformed flagship models (e.g., Qwen-235B, DeepSeek-V3.2, Claude-4.5-Sonnet) in terms of PIE.
- SOTA models exhibited "Aggressive Filtering" (high IR but unacceptably high FPR, e.g., Claude-4.5 had 46.4% FPR), leading to information loss.
- ConceptRM achieved a superior balance, intercepting low-quality comments while preserving high-quality insights.
Data Efficiency:
- Using only 100 expert annotations to guide the cleaning process yielded results nearly identical to using 1,000 annotations, proving the method's robustness in cold-start scenarios.

5. Significance

Mitigating Alert Fatigue: By effectively filtering noise without silencing valid insights, ConceptRM restores trust in AI assistants, preventing the "cry wolf" effect where developers ignore all alerts.
Industrial Applicability: The framework solves the "noisy data" problem common in production environments without requiring massive manual re-annotation costs.
Broader Impact: The consensus-based, purity-driven approach offers a blueprint for building robust Reward Models for Reinforcement Learning (RL) and safety auditors in other high-stakes domains (e.g., medical diagnostics, industrial monitoring) where false positives are costly.
Theoretical Advancement: It provides a new paradigm for fine-tuning LLMs in low-resource, noisy settings by leveraging data space divergence rather than parameter space divergence.