🤖 machine learning

Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning

This paper introduces Retention-aware Policy Optimization (RaPO), a novel Reinforcement Fine-Tuning method that mitigates catastrophic forgetting in visual continual learning by addressing Trajectory-level Drift Agnosticism through retention reward shaping and cross-task advantage normalization, thereby achieving superior performance over existing approaches.

Original authors: Meng Lou, Hanzhong Guo, Linwei Chen, Yizhou Yu

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Meng Lou, Hanzhong Guo, Linwei Chen, Yizhou Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Goldfish" AI

Imagine you are teaching a very smart robot to recognize different types of birds.

Monday: You show it a picture of a Sparrow. It learns it perfectly.
Tuesday: You show it a picture of a Hawk. It learns the Hawk, but suddenly, it forgets what a Sparrow looks like. It thinks the Sparrow is just a small Hawk.
Wednesday: You show it an Owl. Now it forgets the Sparrow and the Hawk.

This is called Catastrophic Forgetting. The robot is like a goldfish with a 3-second memory; every time it learns something new, it wipes its brain clean of the old stuff. This is a huge problem for AI that needs to learn continuously in the real world.

The Current Solution (and why it's not perfect)

Researchers tried two main ways to fix this:

Supervised Fine-Tuning (SFT): This is like a strict teacher who says, "Memorize this answer exactly." It works okay, but the robot still forgets old answers quickly.
Reinforcement Fine-Tuning (RFT): This is like a coach who says, "Try different ways to solve this problem, and I'll give you a reward if you get it right." Recent studies showed this "coach" method (specifically a technique called GRPO) is better at remembering than the "teacher" method.

However, the authors of this paper found a catch: Even the "coach" method (GRPO) still forgets a lot when the tasks get really hard. It's better than the teacher, but it's not perfect.

The Discovery: The "Drift" Detective

The researchers asked: Why does the "coach" method still forget things?

They looked closely at how the robot thinks. They noticed something weird called Trajectory-level Drift Agnosticism.

The Analogy: Imagine the robot is taking a test. It can solve the current question correctly in two different ways.
- Way A: It uses a logic that is very similar to how it solved questions yesterday.
- Way B: It uses a completely wild, new logic that is totally different from yesterday.
The Problem: The current "coach" (GRPO) only looks at the final score. If both Way A and Way B get the question right, the coach gives them the same reward. It doesn't care that Way B is "drifting" away from the robot's old knowledge.
The Result: Over time, the robot keeps choosing the "wild" ways (Way B) because they get the points. Eventually, it drifts so far away from its original knowledge that it forgets everything it used to know.

The Solution: RaPO (Retention-aware Policy Optimization)

To fix this, the authors created a new method called RaPO. Think of RaPO as a "Smart Coach" who cares about two things: getting the answer right and staying true to who you were yesterday.

RaPO has two main tricks:

1. The "Memory Anchor" (Retention Reward)

How it works: Every time the robot tries to solve a problem, RaPO checks: "How similar is this new way of thinking to how you thought yesterday?"
The Analogy: Imagine you are learning a new dance move.
- If you learn a move that flows naturally from your previous style, the coach gives you a bonus point.
- If you learn a move that is totally chaotic and breaks your previous rhythm, the coach gives you fewer points, even if you still finished the dance.
The Goal: This encourages the robot to learn new things without completely abandoning its old habits. It gently nudges the robot to stay close to its "memory anchor."

2. The "Steady Hand" (Cross-Task Advantage Normalization)

How it works: When the robot switches from learning "Birds" to learning "Cars," the difficulty of the tasks changes suddenly. This can confuse the robot's learning process, making it swing wildly between being too confident and too unsure.
The Analogy: Imagine you are driving a car. Suddenly, you switch from a smooth highway to a bumpy dirt road. If your car's suspension reacts instantly to every bump, you'll crash.
The Fix: RaPO uses a "shock absorber" (an exponential moving average). Instead of reacting to the bumpy road right now, it smooths out the bumps over time. This keeps the robot's learning pace steady, even when the tasks change drastically.

The Results

The researchers tested RaPO on many different visual tasks:

Recognizing new types of images (Image Classification).
Finding objects in pictures (Object Detection).
Understanding videos (Video Classification).

The Outcome:
RaPO was the clear winner. It learned new things just as fast as the other methods but forgot much less.

In one test, the old method forgot about 20% of what it learned.
RaPO only forgot about 4%.

The Bottom Line

This paper doesn't just say "AI is getting smarter." It specifically solves the problem of how to teach an AI to keep learning new things without deleting its old memories.

They found that simply rewarding the AI for being "correct" isn't enough. You also have to reward it for being consistent with its past self. By adding a "memory check" and a "stabilizer" to the learning process, they created an AI that can grow without losing its identity.

Note: The paper focuses entirely on computer vision tasks (images and videos) and does not claim these methods are currently used for medical diagnosis, surveillance, or other specific real-world applications yet. It is a foundational study to help future AI systems learn continuously.

Technical Summary: Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning

Problem Statement
While recent studies suggest that Reinforcement Fine-Tuning (RFT), particularly methods like Group Relative Policy Optimization (GRPO), is inherently more resilient to catastrophic forgetting than Supervised Fine-Tuning (SFT) in Multi-modal Large Language Models (MLLMs), its efficacy in challenging visual continual learning (VCL) settings remains an open problem. Specifically, in rehearsal-free scenarios such as Class-Incremental Learning (CIL) and Domain-Incremental Learning (DIL), standard RFT still suffers from non-negligible forgetting. The authors identify a specific bottleneck termed Trajectory-level Drift Agnosticism: among candidate rollouts that achieve identical task rewards, the Kullback-Leibler (KL) divergence from the preceding-task policy varies substantially. Purely task-reward-driven behavior fails to distinguish between these trajectories, leading to credit assignment that inadvertently favors high-drift solutions and accelerates the loss of historical knowledge.

Methodology: Retention-aware Policy Optimization (RaPO)
To address trajectory-level drift agnosticism, the authors propose RaPO, a simple yet effective RFT method that explicitly mitigates forgetting through trajectory-level reward shaping. RaPO consists of two core components:

Retention Reward: This component converts trajectory-level distribution drift into a continuous reward signal. For a rollout $y_i$ generated by the current policy $\pi_t$ , the drift is measured as the length-normalized log-probability ratio against a frozen anchor policy $\pi_{t-1}$ (the policy from the preceding task).
- The drift metric is defined as $\bar{D}_{drift}(y_i) = \max(\frac{1}{m_i}\sum \log \frac{\pi_t}{\pi_{t-1}}, 0)$ .
- This drift is mapped to a bounded, positive reward $R_{ret}(y_i) = \exp(-\alpha \bar{D}_{drift}(y_i))$ , where lower drift yields a higher reward.
- The total reward is formulated as $R_{total} = R_{task} + \lambda R_{ret}$ . Crucially, this reward is added before the group-relative advantage calculation. This ensures that among rollouts with comparable task performance, those that remain closer to the anchor policy (preserving prior knowledge) receive higher advantages, steering the policy toward low-drift regions.
Cross-Task Advantage Normalization (CTAN): To counter optimization instability caused by abrupt shifts in reward distributions at task boundaries, RaPO replaces the per-batch reward standard deviation with a persistent Exponential Moving Average (EMA) $\hat{\sigma}$ .
- $\hat{\sigma}$ is updated across optimization steps and task boundaries ( $\hat{\sigma} \leftarrow \beta \hat{\sigma} + (1-\beta)\sigma_{batch}$ ).
- The advantage is computed as $A_i = \frac{R_{total}(y_i) - \mu_{group}}{\hat{\sigma} + \epsilon}$ .
- This mechanism prevents the advantage scale from collapsing or exploding when transitioning between tasks, ensuring stable credit assignment throughout the continual learning stream.

Key Contributions

Empirical Identification of Drift Agnosticism: The paper provides the first systematic analysis showing that vanilla GRPO suffers from trajectory-level drift agnosticism in visual VCL, where equal-reward trajectories exhibit vastly different degrees of distributional drift relative to previous tasks.
RaPO Framework: The authors introduce RaPO, which integrates a Retention Reward and CTAN to explicitly penalize high-drift rollouts and stabilize optimization scales, respectively.
Comprehensive Evaluation: The method is evaluated across five diverse visual continual learning settings: class-incremental image classification, domain-incremental image classification, class-incremental object detection, domain-incremental object detection, and class-incremental video classification.
First Systematic Exploration: To the best of the authors' knowledge, this is the first work to systematically explore the potential of verifier-based RFT for visual continual learning.

Experimental Results
Extensive experiments on datasets including ImageNet-R, ImageNet-A, TinyImageNet, CUB-200, COCO 2017, UCF-101, and Kinetics-200 demonstrate that RaPO consistently outperforms strong baselines (including SFT, EWC, LwF, and standard GRPO).

Image Classification: On ImageNet-R (10 tasks), RaPO achieved an accuracy of 85.92% with only 4.69% forgetting, significantly outperforming GRPO (74.67% accuracy, 20.02% forgetting).
Object Detection: On COCO (5 tasks), RaPO improved box AP from 14.64% to 19.31% while reducing forgetting from 6.67% to 1.39%.
Generalization: RaPO showed robust performance across video classification and domain-incremental settings, maintaining strong plasticity for new tasks while preserving historical knowledge.
Ablation Studies: The authors confirmed that both the Retention Reward and CTAN are complementary; removing either degrades performance. The method was also shown to be robust to hyperparameter choices and effective when applied to other policy optimization algorithms (e.g., SAPO).

Significance and Claims
The paper claims that while RFT is naturally more resilient to catastrophic forgetting than SFT, it is insufficient for challenging visual VCL without explicit intervention against trajectory-level drift. RaPO addresses this by modifying the reward signal to prioritize knowledge-preserving rollouts without requiring architectural changes or data replay. The authors position this work not as a pursuit of state-of-the-art benchmarks through complex engineering, but as a foundational exploration of how RFT can be adapted for continual learning. They conclude that RaPO offers a simple, effective strategy that substantially reduces catastrophic forgetting while preserving the strong reasoning and adaptation capabilities of MLLMs, hoping to inspire further research in RFT-based continual learning.