🤖 AI

Triple Expert Learning from Noisy Labels for Semi-Supervised Vision Foundation Model Adaptation

This paper proposes TriNoL, a semi-supervised adaptation framework for Vision Foundation Models that enhances robustness to noisy pseudo-labels by routing unlabeled samples into three confidence-based regions and training specialized LoRA experts for positive, alignment, and negative signals while keeping the backbone frozen.

Original authors: Xuanyu Liu, Zheng Fang, Hongyang He, Yundi Hong, Daizong Liu

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuanyu Liu, Zheng Fang, Hongyang He, Yundi Hong, Daizong Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to recognize animals. You have a few photos with perfect names written on them, but you also have a massive pile of photos with no names at all. To teach the robot, you ask it to guess the names for the unlabeled photos. If it guesses confidently, you write that guess down and use it as a "teacher" for the next round. This is the world of Semi-Supervised Learning, a clever trick in computer science that lets us learn from huge amounts of unlabeled data.

But here's the catch: the robot isn't perfect. Sometimes it guesses confidently but gets it totally wrong. These wrong guesses are called "noisy labels." If you just let the robot learn from all its guesses, it might start believing its own mistakes, getting confused, and learning the wrong things. This is especially tricky when using Vision Foundation Models—these are giant, pre-trained AI brains that already know a lot about the world. We don't want to retrain the whole brain (it's too heavy and expensive); we just want to add a tiny, lightweight "adapter" to help it learn our specific task. The big question is: how do we teach this adapter when the "teachers" (the robot's guesses) are a mix of geniuses, confused students, and total liars?

This is exactly the problem tackled by a new method called TriNoL (Triple-expert learning from Noisy Labels). The researchers realized that treating all unlabeled photos the same way is a mistake. A single, tiny adapter trying to learn from a mix of perfect guesses, shaky guesses, and wild guesses gets confused. It's like trying to teach a single student to be a math genius, a creative writer, and a safety inspector all at once using a jumbled pile of instructions. The instructions for math might contradict the instructions for safety, and the student ends up learning nothing well.

To fix this, the authors propose splitting the learning job into three specialized "experts," each with their own tiny adapter. They sort the unlabeled photos into three groups based on how confident the robot is about its guess:

The Positive Expert (The High-Confidence Geniuses): This expert only looks at photos where the robot is super sure (like 95% confident). These guesses are usually right, so this expert learns hard and fast, sharpening the robot's ability to tell a cat from a dog.
The Alignment Expert (The Confused Middle Ground): This expert handles the photos where the robot is "meh" (maybe 50-70% sure). These are the tricky ones near the edge of categories. Instead of forcing a hard guess, this expert gently nudges the robot to be consistent, helping it understand the blurry lines between categories without forcing it to make a mistake.
The Negative Expert (The Safety Net for Liars): This expert deals with the photos where the robot is barely guessing (low confidence). Instead of ignoring these or trying to force them to be right, this expert learns to avoid the wrong answers. It acts like a filter, making sure the robot doesn't get tricked by its own wild guesses.

The paper suggests that by separating these three groups into three different "learning paths," the system becomes much more robust. The "Positive" path stays clean and strong because it isn't polluted by the noisy guesses. The "Alignment" path helps the robot understand the gray areas, and the "Negative" path protects the system from being misled.

The researchers tested this idea on several datasets, including images of food, birds, and general objects. They found that TriNoL works particularly well when there are very few labeled examples and the unlabeled data is messy. For instance, on a food dataset with only a few labeled images, TriNoL improved accuracy from about 87.58% to 88.42%, beating other methods. They also showed that this improvement isn't just because they made the system bigger; even when they compared TriNoL to a single, much larger adapter with the same amount of computing power, TriNoL still won. This suggests that the secret sauce isn't just having more brainpower, but having the right kind of organization.

However, the authors are careful to note that this isn't a magic bullet for every situation. In cases where there are plenty of labeled images or the data is already very clean, the advantage of having three experts shrinks. They also admit that if the robot's initial confidence scores are completely wrong (miscalibrated), the sorting system might get confused. But for the specific challenge of adapting powerful AI models with limited data and noisy guesses, this "triple-expert" approach offers a promising, structured way to keep the learning on track.

Technical Summary: TriNoL – Triple Expert Learning from Noisy Labels for Semi-Supervised VFM Adaptation

1. Problem Statement

The paper addresses the challenge of adapting Vision Foundation Models (VFMs), such as CLIP and DINO-series encoders, in semi-supervised learning (SSL) settings where labeled data is scarce and unlabeled data relies on pseudo-labels. While VFMs provide strong pretrained representations, their adaptation typically involves freezing the backbone and updating lightweight Parameter-Efficient Fine-Tuning (PEFT) modules, such as LoRA (Low-Rank Adaptation).

The core problem identified is the sensitivity of single-adapter LoRA to noisy pseudo-labels. In standard SSL, pseudo-labels exhibit mixed reliability: high-confidence labels are often correct, medium-confidence labels are ambiguous (near decision boundaries), and low-confidence labels are frequently noisy. Existing methods typically treat pseudo-label confidence as a scalar weight or a binary filter (selecting high-confidence samples and discarding the rest). However, the authors argue that compressing gradients from these three distinct reliability regions into a single low-rank update space causes gradient interference. Reliable gradients, ambiguous gradients, and noisy gradients compete within the same adapter, potentially distorting the adaptation direction and reinforcing incorrect decision boundaries (confirmation bias).

2. Methodology: TriNoL

The authors propose TriNoL (Triple-expert learning from Noisy Labels), a framework that decomposes pseudo-label supervision into three confidence-based regions, each handled by a specialized LoRA expert.

2.1 Architecture

Frozen Backbone: The pretrained VFM backbone ( $\psi$ ) remains frozen.
Triple LoRA Experts: Instead of a single adapter, TriNoL employs three independent LoRA branches ( $\theta^+$ $θ^{+}$ , $\theta^a$ $θ^{a}$ , $\theta^-$ $θ^{-}$ ) inserted into the frozen linear projections:
1. Positive Expert ( $e^+$ ): Handles high-confidence pseudo-labels.
2. Alignment Expert ( $e^a$ ): Handles medium-confidence (ambiguous) pseudo-labels.
3. Negative Expert ( $e^-$ ): Handles low-confidence (noisy) pseudo-labels.
Classifier Head: A shared classifier head ( $C_\omega$ ) is updated alongside the experts.

2.2 Confidence-Based Routing

During training, unlabeled samples are processed through a weakly augmented view to generate pseudo-labels and confidence scores ( $q_i$ ). Two thresholds, $\tau^-$ and $\tau^+$ , partition the unlabeled batch $B_u$ into three sets:

High Confidence ( $B_u^+$ ): $q_i \ge \tau^+$ . Routed to the Positive Expert.
Medium Confidence ( $B_u^a$ ): $\tau^- \le q_i < \tau^+$ . Routed to the Alignment Expert.
Low Confidence ( $B_u^-$ ): $q_i < \tau^-$ . Routed to the Negative Expert.

2.3 Expert-Specific Learning Objectives

Each expert is optimized with a distinct loss function tailored to the reliability of its assigned samples:

Positive Loss ( $L_{pos}$ ): Uses hard pseudo-label supervision (Cross-Entropy) on high-confidence samples to strengthen discriminative adaptation.
Alignment Loss ( $L_{align}$ ): Uses soft alignment (Kullback-Leibler divergence) for medium-confidence samples. It matches the strong-view prediction of the Alignment Expert to the detached weak-view probability ( $sg(p_i^w)$ ). This avoids forcing hard decisions on ambiguous samples while preserving boundary information.
Negative Loss ( $L_{neg}$ ): Designed to suppress over-commitment to unreliable labels. Instead of treating low-confidence pseudo-labels as targets, this loss penalizes the Negative Expert for assigning high probability to the current top pseudo-label. This prevents noisy signals from dominating the update without assuming all low-confidence samples are strictly wrong.

The total objective is a weighted sum: $L = L_{sup} + \lambda_{pos}L_{pos} + \lambda_{align}L_{align} + \lambda_{neg}L_{neg}$ .

2.4 Inference

At inference time, only the Positive Expert is used, as it is trained on ground-truth labels and reliable pseudo-labels. The other experts serve to regularize the training process.

3. Theoretical Analysis

The paper provides theoretical justification for the approach:

Gradient Interference: A single adapter updates based on a mixed gradient $G_{single} = G^+ + G^a + G^-$ . The deviation from the ideal reliable update ( $G^+$ ) is proportional to the magnitude of the ambiguous and noisy gradients.
Contamination Reduction: By routing samples, TriNoL ensures the Positive Expert receives a gradient $G^+_{TriNoL}$ with significantly less contamination from $G^a$ and $G^-$ , leading to a smaller deviation from the reliable target.
Subspace Stability: In the context of low-rank approximation, separating the "clean" signal ( $S$ ) from noise ( $E$ ) and ambiguity ( $A$ ) allows the Positive Expert to approximate a cleaner target matrix, resulting in a low-rank subspace closer to the true optimal direction compared to a single adapter trained on the contaminated matrix.
Optimization Stability: Reducing the bias in the Positive Expert's gradient leads to a smaller steady-state bias radius in local optimization, improving convergence stability.

4. Experimental Results

TriNoL was evaluated on CIFAR-100, FOOD-101, Semi-Aves, and ImageNet under various semi-supervised settings (varying labeled data ratios and noise levels).

Performance: TriNoL achieves state-of-the-art results in low-label and noisy settings. Notable improvements include:
- FOOD-101 (N2): 88.42% vs. 87.58% (V-PET) and 87.27% (FineSSL).
- Semi-Aves (Mixed OOD): 61.82% vs. 61.30% (FineSSL).
- CIFAR-100 (N4): 80.92% vs. 80.66% (FineSSL).
Efficiency: TriNoL uses three rank-8 LoRA experts (totaling ~0.96M parameters), comparable to a single rank-24 LoRA adapter. Despite similar parameter counts and computational costs, TriNoL outperforms the rank-matched single adapter (88.42% vs. 87.73% on FOOD-101 N2), demonstrating that the gain stems from expert specialization rather than increased capacity.
Ablation Studies:
- Removing any of the three experts degrades performance, confirming their complementary roles.
- Replacing confidence-based routing with random routing or shared loss functions yields inferior results, validating the necessity of both routing and specialized objectives.
- Sensitivity analysis indicates optimal performance with thresholds $(\tau^-, \tau^+) = (0.3, 0.7)$ and specific loss weights.

5. Significance and Claims

The paper claims that TriNoL offers a significant shift in how semi-supervised VFM adaptation handles pseudo-label noise. Its primary contributions are:

Identification of a Limitation: It highlights that compressing mixed-reliability pseudo-label gradients into a single low-rank space is a fundamental weakness in current PEFT-based SSL.
Confidence-Aware Routing: It proposes that pseudo-label reliability should control the adaptation pathway, not just the loss weight. By assigning reliable, ambiguous, and unreliable samples to specialized experts, the method reduces gradient interference.
Robustness with Low Cost: TriNoL improves robustness to noisy supervision while maintaining the efficiency of frozen-backbone adaptation, making it suitable for resource-constrained scenarios with limited labels.

The authors acknowledge limitations, noting that the method relies on confidence-based routing (which can be affected by miscalibration or domain shift) and is currently focused on image classification. Future work is suggested for dense prediction and multimodal tasks.