🤖 machine learning

Calibratable Disambiguation Loss for Multi-Instance Partial-Label Learning

This paper proposes a plug-and-play Calibratable Disambiguation Loss (CDL) that modulates a margin-based objective to simultaneously improve classification accuracy and model calibration in Multi-Instance Partial-Label Learning, addressing the reliability issues inherent in existing MIPL approaches.

Original authors: Wei Tang, Yin-Fang Yang, Weijia Zhang, Min-Ling Zhang

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wei Tang, Yin-Fang Yang, Weijia Zhang, Min-Ling Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but you have two big problems. First, you don't get to see the whole crime scene at once; you only get to look at tiny, scattered clues (patches of a photo) and have to guess what happened in the whole room. Second, when you ask a crowd of people for the name of the suspect, they don't give you one name. They give you a list of three or four names, saying, "It could be one of these!" but they won't tell you which one is actually the culprit.

This is the messy reality of Multi-Instance Partial-Label Learning (MIPL). It's a super tricky way for computers to learn from imperfect data. The paper you're reading tackles a specific headache that comes with this: confidence.

The Problem: The Overconfident (or Underconfident) Detective

In the past, computers trying to solve these MIPL puzzles were like detectives who were either too scared to make a call or way too sure of themselves when they were wrong.

The authors found that existing methods (like DEMIPL, ELIMIPL, and MIPLMA) were terrible at knowing how sure they should be. If a computer said, "I'm 90% sure this is a tumor," it was often only right 60% of the time. Or, if it said, "I'm only 25% sure," it might actually be right 80% of the time.

In the real world, this is dangerous. Imagine a doctor using an AI to look at a pathology slide (a picture of cells). If the AI says, "I'm 90% sure this is cancer," the doctor might rush to start treatment. But if the AI is actually only right 60% of the time, that's a disaster. The computer needs to be calibrated: if it says "90%," it should be right 90% of the time.

The paper explicitly argues against a simple fix that people tried first: just taking standard "Focal Loss" (a tool usually used when you know the exact right answer) and slapping it onto this messy MIPL problem. The authors showed that doing this naively made the computer either too timid (under-confident) or too arrogant (over-confident), and it actually made the computer worse at getting the right answer.

The Solution: The "Top vs. Competitor" Margin

So, what did the authors invent? They created a new tool called Calibratable Disambiguation Loss (CDL).

Think of the computer's brain as a student taking a multiple-choice test where the teacher only gave them a list of possible answers (the "candidate set"). The student has to pick the right one from that list.

In the old methods, the student just tried to shout, "I know the answer!" and push their chosen answer to 100% confidence as fast as possible. This led to the "over-confidence" problem.

The new CDL method is like a strict teacher who says: "Don't just shout about your answer. Look at your competitors."

The Competitor: This is the second-best guess the computer is making.
The Rule: The computer is only allowed to get super confident if its top guess is clearly better than the second-best guess.

If the computer is struggling and its top guess and second guess are neck-and-neck (a small "margin"), the CDL says, "Okay, keep working hard, don't get cocky yet." But if the top guess is way ahead of the competitor, the CDL says, "Great job! You can relax and be confident now."

This creates two flavors of the tool:

CDL-CC: Compares the top guess against the second-best candidate (like comparing the #1 and #2 runners in a race).
CDL-CN: Compares the top guess against the strongest non-candidate (like comparing the #1 runner against the best person who wasn't even allowed to run).

The Results: Proven by the Numbers

The authors didn't just guess this would work; they tested it on a bunch of datasets, including real-world pathology images of colorectal cancer and standard image datasets like MNIST.

Here is what they found, with the exact numbers from their experiments:

Better Accuracy: On the Birdsong-MIPL dataset, the old best method (DEMIPL) got 74.36% accuracy. Their new method (DAMCN) jumped to 80.38%. On the SIVAL-MIPL dataset with three wrong labels mixed in, they improved accuracy by a massive 18.58% over the old method.
Much Better Calibration: This is the big win. The old methods had terrible "Expected Calibration Error" (ECE), which measures how far off their confidence is from reality.
- On the Birdsong-MIPL dataset with three wrong labels, the old method (DEMIPL) had an ECE of 53.42%. That's a huge gap between what they said and what was true.
- Their new method (DAMCN) slashed that error down to 4.52%.
- In total, across 60 different test cases on benchmark datasets, their method reduced the calibration error by more than 50% in 26 of those cases. The average reduction was 44.76%.

The Verdict

The paper proves that by adding this "margin" rule—forcing the computer to check how much better its best guess is compared to the runner-up—you can fix the computer's confidence without hurting its ability to get the right answer.

They showed that this works on everything from simple image puzzles to complex medical images where they used deep learning to break pictures into 9, 16, or even 25 tiny patches. The more complex the image, the more their new method helped, with some versions reducing calibration error by 33.32% on the hardest cancer datasets.

So, the next time you see an AI making a medical diagnosis, you can hope it's using a method like this: one that knows exactly how sure it is, because it learned to respect its competitors before it started shouting "I'm right!"

Problem Statement

The paper addresses Multi-Instance Partial-Label Learning (MIPL), a weakly supervised framework that combines challenges from Multi-Instance Learning (MIL) and Partial-Label Learning (PLL). In MIPL, each training example consists of a "bag" of instances where the positive instances are unidentified (MIL challenge), and the true label is hidden within a set of candidate labels (PLL challenge).

While existing MIPL methods focus on label disambiguation (identifying the true label from the candidate set), the authors identify a critical gap: these methods often suffer from poor model calibration. Existing approaches produce confidence scores that do not reflect the empirical probability of correctness. This is particularly problematic in decision-sensitive applications like pathology image classification, where confidence scores guide triage and diagnosis. The authors demonstrate that naively adapting calibration techniques from fully supervised learning (such as Focal Loss or Inverse Focal Loss) to MIPL fails because the absence of ground-truth labels leads to under-confident or over-confident predictions and reduced classification accuracy.

Methodology

The authors propose a plug-and-play Calibratable Disambiguation Loss (CDL) designed to improve both classification accuracy and calibration without compromising the disambiguation process.

1. Core Mechanism:
CDL modulates a momentum-based disambiguation objective using a top-vs-competitor prediction margin. The loss function is defined as:
$L_{CDL} = -\sum_{c \in S_i} w^{(t)}_{i,c} (1 - q^{(t)}_i + \Phi(\hat{p}^{(t)}_i))^\gamma \log(\hat{p}^{(t)}_{i,c})$
Where:

$q^{(t)}_i$ is the predicted probability of the top-ranked candidate label.
$\Phi(\hat{p}^{(t)}_i)$ represents the probability of a "competitor."
The term $(1 - \beta_i)^\gamma$ acts as a modulation factor, where $\beta_i = q^{(t)}_i - \Phi(\hat{p}^{(t)}_i)$ is the margin.

2. Two Variants:
The method introduces two specific instantiations based on how the competitor is defined:

CDL-CC (Candidate-vs-Candidate): The competitor is the second-highest candidate label. This variant emphasizes candidate-level separation.
CDL-CN (Candidate-vs-Non-Candidate): The competitor is the highest probability among non-candidate labels. This variant emphasizes candidate-vs-non-candidate suppression.

3. Adaptive Weighting:
The loss employs a momentum-based weight update strategy for candidate labels. Low-margin bags (ambiguous cases) retain strong disambiguation signals (high loss weight), while well-separated bags receive less incentive for excessive probability sharpening, thereby mitigating over-confidence.

4. Integration:
CDL is designed to be seamlessly integrated into existing embedded-space MIPL frameworks. The authors combine CDL with three established attention mechanisms:

Disambiguation Attention Mechanism (DAM)
Scaled additive Attention Mechanism (SAM)
Margin-aware Attention Mechanism (MAM)
This yields six variants (e.g., DAMCC, SAMCN, MAMCC, etc.).

Theoretical Analysis

The paper provides a rigorous theoretical analysis of CDL from three perspectives:

Loss Level: CDL is shown to be a margin-modulated version of the Momentum-based Disambiguation Loss (MDL). Theoretical bounds prove that CDL preserves the MDL fitting signal for low-margin (ambiguous) bags while down-weighting well-separated predictions.
Calibration Level: The authors derive a lower bound and a pseudo-label confidence-alignment bound. They show that minimizing CDL risk controls the alignment between model confidence and pseudo-label weights, which is a key component of the Expected Calibration Error (ECE).
Optimization Level: Gradient analysis reveals that CDL introduces an additional "margin-shaping" term. This term explicitly promotes an increase in the active top-vs-competitor margin during optimization. Furthermore, through momentum updates, these shaped margins are recursively transferred to future pseudo-label weights, ensuring the calibration effect persists across epochs.

Experimental Results

The authors evaluated their approach on four benchmark datasets (MNIST-MIPL, FMNIST-MIPL, Birdsong-MIPL, SIVAL-MIPL) and seven real-world datasets (including various configurations of CRC-MIPL for colorectal cancer classification).

Classification Accuracy: The proposed methods outperformed six state-of-the-art MIPL baselines (MIPLGP, DEMIPL, ELIMIPL, MIPLMA, PROMIPL, FASTMIPL) in 67 out of 72 benchmark cases. Improvements were particularly significant in challenging scenarios with higher numbers of false-positive labels (e.g., up to 23% improvement on MNIST-MIPL with $r=3$ ).
Calibration Performance: The methods significantly reduced the Expected Calibration Error (ECE). On benchmark datasets, CDL reduced ECE by over 50% in 26 out of 60 cases, with a mean reduction of 44.76%. On real-world datasets, the methods achieved over a 10% reduction in ECE in 33 out of 42 cases.
Reliability Diagrams: Visual analysis confirmed that while baseline methods clustered predictions around low confidence values (e.g., 0.25) despite high accuracy, the CDL variants produced reliability diagrams where predicted confidence closely matched empirical accuracy.

Significance and Contributions

The paper claims four primary contributions:

Problem Identification: It is the first work to identify and characterize model calibration issues in MIPL, demonstrating that existing disambiguation-centered methods produce poorly calibrated confidence estimates.
Methodological Innovation: The proposal of CDL, a plug-and-play loss function with two complementary instantiations (CC and CN) that balance disambiguation and calibration via margin modulation.
Theoretical Foundation: A comprehensive theoretical analysis establishing CDL as a margin-modulated MDL, deriving calibration bounds, and explaining the optimization dynamics through gradient and momentum analysis.
Empirical Validation: Extensive experiments confirming that CDL consistently improves both classification accuracy and calibration performance across diverse benchmark and real-world datasets, including pathology image classification.

The authors conclude that CDL offers a robust solution for MIPL, enabling the construction of predictive models that are not only accurate but also reliable in their confidence estimates, which is crucial for real-world deployment in fields like medical diagnostics.