🤖 machine learning

Natural Gradient Descent for Online Continual Learning

This paper proposes a novel Online Continual Learning approach that leverages Natural Gradient Descent with a Kronecker Factored Approximate Curvature (KFAC) approximation of the Fisher Information Matrix to accelerate convergence and mitigate catastrophic forgetting across various datasets and existing methods.

Original authors: Joe Khawand, David Colliaux

Published 2026-03-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Joe Khawand, David Colliaux

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a student trying to learn a new language every single day, but with a very strict rule: you can only study the new language for one hour, and once that hour is over, you must throw away your textbook and can never look at it again.

This is the challenge of Online Continual Learning (OCL). In the real world, AI systems (like the ones in your phone or car) face this exact problem. They need to learn new things constantly without forgetting everything they learned yesterday. If they forget, it's called "Catastrophic Forgetting." It's like a student who learns French, then spends an hour studying Spanish, and suddenly forgets how to say "hello" in French.

The Problem: The "Stumble" of Standard Learning

Most AI models learn using a method called Stochastic Gradient Descent (SGD). Think of SGD as a hiker trying to find the bottom of a valley (the best solution) in the dark.

How it works: The hiker feels the slope under their feet and takes a step downhill.
The flaw: The hiker assumes the ground is flat and uniform. But in reality, the "landscape" of learning is bumpy and curved. Sometimes, a small step in the wrong direction on a steep curve sends the hiker flying off a cliff, causing them to forget everything they knew before.

The Solution: The "Smart Map" (Natural Gradient Descent)

The authors of this paper propose a smarter way to learn called Natural Gradient Descent (NGD).

Instead of just feeling the slope, imagine the hiker has a GPS map that shows the shape of the terrain.

The Metaphor: If the ground is a steep, curved hill, a normal hiker might slide down too fast and crash. The "Smart Hiker" (NGD) looks at the map, sees the curve, and adjusts their step to walk along the curve rather than sliding off it.
The Result: They reach the bottom (the best learning state) much faster and with fewer mistakes. They don't lose their footing as easily when the terrain changes (which happens constantly in online learning).

The Secret Weapon: KFAC (The "Cheat Sheet")

There's a catch. Calculating this perfect "map" (mathematically called the Fisher Information Matrix) is incredibly hard and slow for a computer, like trying to draw a map of the entire world while running a marathon.

To solve this, the authors use a trick called KFAC (Kronecker Factored Approximate Curvature).

The Analogy: Instead of drawing a detailed map of every single tree and rock, KFAC says, "Let's just approximate the general shape of the hills and valleys." It breaks the complex map into two smaller, easier-to-handle pieces and multiplies them together.
The Benefit: It gives the AI a "good enough" map that is fast to compute, allowing it to take those smart, curved steps without getting bogged down in heavy math.

What Did They Find?

The researchers tested this "Smart Hiker" approach on several difficult datasets (like recognizing different types of animals or handling blurry photos). Here is what happened:

Faster Learning: The AI learned new tasks much quicker. It didn't need to wander around aimlessly; it knew exactly which way to step.
Better Retention: Because the steps were more careful and calculated, the AI forgot less about its previous lessons.
Works with Everything: They tried this method alongside other popular "study hacks" (like keeping a small notebook of old examples). In almost every case, adding this "Smart Map" made the other methods work even better.

The One Catch

The paper notes a small trade-off. While the AI learned new things faster and got smarter overall, it sometimes forgot the very specific details of old tasks a tiny bit more than usual. However, the authors argue that the massive gain in overall performance and speed is worth this small cost.

The Big Picture

In simple terms, this paper says: "Stop teaching AI to walk in a straight line on a curved world. Give it a map that understands the curves."

By using this new "Smart Map" technique, we can build AI systems that learn continuously, adapt to new situations instantly, and don't suffer from amnesia every time they encounter something new. This is a huge step toward making AI that can truly learn and grow alongside us, just like humans do.

1. Problem Statement

The paper addresses the challenges of Online Continual Learning (OCL) for image classification. Unlike standard machine learning settings that assume independent and identically distributed (i.i.d.) data, OCL requires models to learn from a continuous stream of data where:

Constraints: The model can only iterate once over the current data (single-pass), uses very small batch sizes, and has no access to previous tasks or large storage buffers.
Scenarios: The model faces Online Class Incremental (OCI) learning (new classes arrive over time) or Online Domain Incremental (ODI) learning (data distribution shifts, e.g., noise, blur, occlusion).
Core Challenge: The primary issue is catastrophic forgetting, where the model loses performance on previous tasks as it learns new ones. While existing strategies (like Experience Replay) exist, achieving rapid convergence and high accuracy in this constrained, non-stationary setting remains difficult.

2. Methodology

The authors propose replacing the standard first-order optimizer (Stochastic Gradient Descent - SGD) with Natural Gradient Descent (NGD) approximated via Kronecker Factored Approximate Curvature (KFAC).

Natural Gradient Descent (NGD): Unlike SGD, which assumes a Euclidean metric in parameter space, NGD accounts for the geometry of the probability distribution space. It uses the Fisher Information Matrix (FIM) to precondition the gradient, ensuring updates respect the underlying data distribution.
- Update Rule: $\theta_{t+1} = \theta_t - \alpha \cdot F^{-1}(\theta_t) \cdot \nabla L(\theta_t)$
KFAC Approximation: Calculating and inverting the full FIM is computationally prohibitive for deep networks. The authors use KFAC, which approximates the FIM as the Kronecker product of two smaller matrices ( $A \otimes B$ ) for each layer, significantly reducing storage and computation costs.
Regularization & Stability: To prevent numerical instability (common in the early "exploration phase" of OCL), the authors implement Tikhonov regularization (damping). They found that a high damping factor ( $\lambda = 1.0$ ) was crucial for stability in the OCL setting.
Integration: The method is applied as a drop-in replacement for the optimizer in existing OCL frameworks (e.g., ER, A-GEM, MIR) and combined with standard OCL "tricks" (e.g., Labels Trick, Nearest Class Mean, Separated Softmax).

3. Key Contributions

Novel Application: This is the first work to apply NGD-KFAC specifically to the Online Continual Learning domain for image classification.
Performance Enhancement: The authors demonstrate that NGD-KFAC significantly improves the performance of existing OCL methods across various benchmarks.
Convergence Analysis: The study highlights that NGD-KFAC facilitates faster convergence, which is critical in single-pass OCL scenarios where the model cannot revisit data.
Comprehensive Evaluation: The method was tested on multiple datasets (Split CIFAR-100, Split MiniImageNet, CORe50) and both OCI and ODI settings, showing robustness across different memory buffer sizes and OCL tricks.

4. Experimental Results

The experiments were conducted on ResNet-18 with a fixed small batch size (10) and a single epoch per task.

Overall Performance:
- On Split CIFAR-100 (OCI), the average end accuracy across all methods improved from 19.58% (without NGD-KFAC) to 22.20% (with NGD-KFAC).
- In ODI settings (e.g., MiniImageNet with noise/occlusion), accuracy improvements were substantial. For example, on MiniImageNet-Occlusion with a 10k buffer, accuracy jumped from 13.7% to 26.7% when using ER + NGD-KFAC.
Impact of Memory Size: The benefits of NGD-KFAC were consistent across buffer sizes (1k, 5k, 10k), with larger buffers yielding higher absolute gains.
Interaction with OCL Tricks:
- Nearest Class Mean (NCM): Showed the most synergy, increasing accuracy from 20.89% to 25.31% while keeping forgetting relatively low.
- Labels Trick & Separated Softmax: Also saw significant accuracy boosts (e.g., Separated Softmax improved from 18.81% to 22.95%).
- Forgetting Trade-off: The authors noted a slight increase in "Average Forgetting" metrics in some configurations. However, they argue this is offset by the significantly higher Average End Accuracy. The faster convergence allows the model to adapt to new tasks more effectively, even if the initial drop on old tasks is slightly steeper.
Damping Sensitivity: The study found that higher damping coefficients (specifically $\lambda = 1.0$ ) yielded the best results, stabilizing the optimization process in the volatile OCL environment.

5. Significance and Conclusion

This paper bridges the gap between theoretical second-order optimization and practical Online Continual Learning.

Efficiency: It proves that second-order methods, often considered too computationally expensive for real-time or resource-constrained settings, can be effectively approximated (via KFAC) to solve OCL problems.
Robustness: By accounting for the curvature of the parameter space, NGD-KFAC provides a more principled update path than SGD, leading to better adaptation to non-stationary data streams.
Future Direction: The results suggest that integrating second-order optimizers with robust OCL architectures (like NCM) is a promising path toward reducing catastrophic forgetting and achieving performance closer to offline-trained models in dynamic environments.

In summary, the authors successfully demonstrate that NGD-KFAC is a powerful, plug-and-play optimization strategy that enhances the state-of-the-art in Online Continual Learning, offering a viable solution to the trade-off between rapid convergence and catastrophic forgetting.