🤖 machine learning

Mitigating Negative Flips via Margin Preserving Training

This paper proposes a novel training framework that mitigates negative flips in evolving image classification systems by preserving original model margins through logit calibration while employing double-source focal distillation to maintain high accuracy on both old and new classes.

Original authors: Simone Ricci, Niccolò Biondi, Federico Pernici, Alberto Del Bimbo

Published 2026-08-10

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Simone Ricci, Niccolò Biondi, Federico Pernici, Alberto Del Bimbo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef who has spent years perfecting a menu. Your customers know exactly what to expect: the "Spicy Tofu" is always spicy, and the "Creamy Mushroom" is always creamy. Then, you decide to expand the menu. You add exciting new dishes like "Dragon Fruit Sorbet" and "Wasabi Ice Cream." This sounds great, but there's a catch. In the rush to make room for these new flavors, you accidentally tweak the recipe for the old favorites. Suddenly, the Spicy Tofu tastes a bit bland, or the Creamy Mushroom gets a weird aftertaste. Your loyal customers are confused and unhappy, even though the new menu is technically "better" on paper. This is the daily struggle of Artificial Intelligence. As AI systems learn new things, they often forget how to do the old things correctly. In the world of computer science, this specific type of confusion—where an AI gets worse at something it used to get right—is called a "negative flip." It's a frustrating glitch where an update breaks something that wasn't broken.

The paper you are about to read tackles this exact problem. The researchers, working with image classification (teaching computers to recognize pictures of cats, dogs, cars, etc.), noticed that when they added new categories to an AI's brain, the "decision lines" it drew between old and new things got too crowded. It was like trying to fit too many people into a small room; everyone started bumping into each other, and the AI got confused about who was who. To fix this, they invented a new training method called Margin Preserving Training (MPT). Think of it as a clever way to teach the AI to respect the personal space of its old friends while making room for new ones. Instead of just shoving everything together, they use a special "bias" to keep the old categories safe and a "double-teacher" system to make sure the new categories don't get left behind. Their experiments show that this method successfully stops the AI from making those annoying mistakes on old tasks, all while learning the new ones just as well as before.

The Problem: When "Better" Means "Worse"

In the fast-paced world of AI, models are constantly being updated. Developers train a new version of an AI, hoping it will be smarter, faster, and able to recognize more things. Usually, this works great. But sometimes, a strange thing happens. An AI that was perfect at identifying a "Golden Retriever" suddenly starts calling it a "Labrador" after an update. Or, it might confuse a "Stop sign" with a "Speed limit sign" when it wasn't doing that before.

This phenomenon is called a negative flip. It's the opposite of a "positive flip," where an AI finally learns to recognize something it previously got wrong. While positive flips are the goal, negative flips are the hidden cost of progress. They are particularly dangerous because they can cause unpredictable behavior in real-world systems, like a self-driving car suddenly misidentifying a familiar road sign.

The researchers found that these negative flips happen most often when the AI is asked to learn more classes (categories). Imagine a classroom where the teacher has to distinguish between 10 students. It's easy. Now, imagine 100 students walk in. The teacher has to squeeze everyone into the same room. To fit them all in, the "personal space" (or margin) between each student gets smaller. In AI terms, the "margin" is the safety buffer the model keeps between different categories. When you add new classes, the model tries to squeeze everything together to maximize the space for everyone, but this often pushes the old categories too close to the edge, making them easy to confuse.

The Solution: Keeping the Peace with MPT

The authors propose a new method called Margin Preserving Training (MPT). Their goal is simple: update the AI to learn new things without shrinking the safety buffers of the things it already knows.

They tackle this with two main tricks, working together like a well-oiled machine:

1. The "VIP Pass" (Margin-Calibrated Loss)
First, they give the new classes a little "head start" during training. Imagine you are teaching a class of students. The old students (old classes) are already sitting in their seats. The new students (new classes) are standing at the door. If you just tell everyone to sit down, the old students might get pushed out of their chairs to make room.

To prevent this, the researchers give the new students a "VIP pass" (a positive logit bias). This pass tells the AI, "Hey, these new guys are important, give them a little extra space to find their seats." By artificially boosting the confidence for new classes during training, the AI is forced to push the decision boundaries away from the old classes. This ensures the old classes keep their wide, safe "margins" and don't get squeezed. It's like telling the AI, "Don't worry about the new stuff too much yet; make sure the old stuff stays safe."

2. The "Double-Teacher" System (Double-Source Focal Distillation)
There's a problem with the VIP pass, though. If you give the new classes too much of a head start, they might become too confident and ignore the actual features of the image, leading to mistakes on the new categories. The AI might learn the new classes poorly because it was too busy protecting the old ones.

To fix this, the researchers introduce a second teacher.

Teacher A (The Old Model): This teacher knows the old classes perfectly. It helps the new model remember how to handle the old stuff.
Teacher B (A Fresh Model): This is a brand-new model trained from scratch on all the classes (old and new) without any special bias. It knows how to handle the new classes correctly.

The new model learns from both teachers at the same time. It listens to Teacher A to keep the old classes safe and listens to Teacher B to make sure it doesn't mess up the new classes. This "double-source" approach ensures a perfect balance: the old classes stay safe, and the new classes are learned accurately.

What They Found

The researchers tested their idea on two famous image datasets: CIFAR100 (100 categories of small images) and ImageNet1K (1,000 categories of larger, more complex images). They compared their method against several other popular techniques used to fix negative flips.

The results were promising. Their method, MPT, consistently reduced the number of negative flips.

On CIFAR100, using a standard ResNet-18 model, their best version (MPT-KL) reduced the Negative Flip Rate (NFR) to 9.26%, compared to 14.04% for a model with no treatment.
On ImageNet1K, the improvement was even clearer. The NFR dropped to 6.09%, beating the next best method (ELODI) which had 7.61%.

Crucially, they didn't just stop negative flips; they didn't sacrifice overall accuracy either. The models still got the new things right. In fact, on ImageNet1K, their method actually improved the overall error rate on old classes from 28.40% (no treatment) down to 24.68%.

They also ran a "what-if" experiment (an ablation study) to see which part of their recipe was the secret sauce. They found that if they removed the "VIP pass" (the margin bias), the negative flips went back up. If they removed the "Double-Teacher" system, the model struggled with the new classes. It turned out that you really needed both parts working together to get the best results.

Visualizing the Magic

To really see what was happening, the researchers created a visual map of how the AI "saw" the images. They turned the complex math inside the AI into a simple 2D drawing.

Without their method: The clusters of old classes (like "cats") and new classes (like "dogs") were squished together. The lines separating them were thin and shaky. A "cat" might accidentally cross the line into the "dog" zone.
With their method: The old classes stayed in their own neat, spacious circles. The new classes found their own spots without crowding the old ones. The "decision lines" remained wide and clear, just like they were in the original model.

The Takeaway

This paper suggests that the key to updating AI without breaking it lies in respecting the "personal space" of what it already knows. By explicitly preserving the decision margins of old classes and using a smart, dual-teacher system to guide the learning of new ones, we can update AI models more safely.

The authors note that their method does require training an extra "reference" model, which takes a bit more computer power, but it's still much cheaper than other complex methods that require huge groups of models. They also emphasize that the "VIP pass" strength (the bias value) needs to be tuned carefully for different datasets.

In short, MPT offers a way to grow an AI's brain without shrinking its memory. It's a step toward AI systems that can learn continuously, adapting to new worlds without forgetting the lessons of the past.

Technical Summary: Mitigating Negative Flips via Margin Preserving Training

Problem Definition
The paper addresses the phenomenon of "negative flips" in machine learning systems, defined as instances where an updated model misclassifies test samples that were previously classified correctly by an older version. While model updates often aim to reduce overall error rates or incorporate new data, they frequently introduce regression in specific cases. This issue is particularly pronounced in image classification tasks where the number of training classes increases over time. The authors argue that adding new categories reduces the decision margin between adjacent classes and introduces interfering features, leading to negative transfer. Consequently, even high-confidence predictions on previously learned classes can become incorrect due to shifts in decision boundaries, disrupting downstream processes and user trust.

Methodology: Margin Preserving Training (MPT)
The proposed solution, Margin Preserving Training (MPT), consists of two primary components designed to balance the preservation of old class performance with the learning of new classes:

Margin-Calibrated Softmax Cross-Entropy Loss:
To counteract the natural reduction of margins when new classes are introduced, the authors introduce a logit bias term ( $\Delta_i$ ) exclusively for newly introduced classes ( $Y_{new}$ ).
- A positive bias $k > 0$ is applied to the logits of new classes during training.
- This acts as an explicit prior that tilts the decision boundaries to favor previously learned classes ( $Y_{old}$ ), effectively preserving their original decision margins.
- At inference, this bias is removed. Because the model was trained without the bias for old classes, it is forced to learn wider, more robust decision margins for them to compensate for the lack of the artificial prior.
- The loss function is defined as $L_\Delta = -\log \frac{e^{z_y + \Delta_y}}{\sum_{i=1}^C e^{z_i + \Delta_i}}$ , where $\Delta_i = k$ if $i \in Y_{new}$ and $0$ otherwise.
Double-Source Focal Distillation:
The authors note that the margin-calibrated loss alone can lead to underfitting of new classes due to the induced bias. To address this, they propose a distillation strategy that leverages two reference models:
- $\phi_{old}$ : The original model trained on $Y_{old}$ .
- $\phi_{new}$ : A new model trained independently on the full set of classes ( $Y_{all}$ ) using standard cross-entropy.
- The student model is trained using a focal distillation loss ( $L_{FD}$ ) that transfers knowledge from both references. This loss selectively emphasizes instances correctly classified by the reference models, using either Kullback-Leibler (KL) divergence or Logit Matching (LM) as the distance metric.
- The final objective combines the margin-calibrated loss with the distillation terms: $\min_\phi L_\Delta + \lambda L_{FD}(\phi, \phi_{old}) + \lambda L_{FD}(\phi, \phi_{new})$ .

Key Contributions

Novel Approach: The paper introduces MPT, a method that explicitly focuses on preserving decision margins for previously learned classes during incremental updates to mitigate negative flips.
Double-Source Focal Distillation: A strategy is proposed to counteract the potential underfitting of new classes caused by margin calibration, utilizing knowledge from both the old model and a newly trained independent model to ensure balanced performance.
Empirical Validation: Extensive experiments demonstrate that the method consistently reduces negative flip rates while maintaining or improving overall accuracy.

Experimental Results
The authors evaluated MPT on CIFAR100 and ImageNet1K datasets using ResNet-18 and ResNet-50 architectures, as well as a ViT-B/32 fine-tuned on CIFAR100.

Negative Flip Rate (NFR): MPT variants (MPT-KL and MPT-LM) consistently achieved the lowest NFR compared to baselines, including Positive-congruent Training (PCT) variants and ELODI (an ensemble-based distillation method).
Overall Accuracy: The method maintained competitive error rates (ER) on both old and new classes. In scenarios where the new model used a more powerful architecture (ResNet-50 vs. ResNet-18), MPT still outperformed baselines in reducing negative flips without sacrificing overall accuracy.
Qualitative Analysis: Visualization of embedding spaces showed that while standard updates ("No Treatment") caused class clusters to drift and intersect (margin shrinkage), MPT successfully maintained the separation of old classes while properly accommodating new classes.
Ablation Studies: Removing either the margin calibration or the double-source distillation resulted in higher negative flip rates or reduced accuracy on new classes, confirming the necessity of both components.

Significance and Claims
The paper claims that existing approaches to reducing negative flips often overlook the progressive reduction of class margins as new categories are added. By explicitly preserving these margins through logit bias and balancing the learning process via double-source distillation, MPT offers a computationally efficient alternative to ensemble methods (which require multiple models at inference) while achieving superior performance in preventing regression. The authors position their work as the first to systematically address decision margin preservation specifically for the reduction of negative flips in incremental learning settings. They acknowledge that the method requires training an additional reference model and tuning the margin bias parameter $k$ , but assert that the computational overhead is modest compared to ensemble-based approaches.