🤖 machine learning

FOGO: Forgetting-aware Orthogonalization Optimizer

FOGO is a scalable optimizer that mitigates both short-term and long-term forgetting by spectrally orthogonalizing momentum updates and resolving gradient conflicts with a compact, random-projection-based memory, thereby improving convergence and knowledge retention across diverse training scenarios without storing past data.

Original authors: Toan Nguyen, Yang Liu, Trung Le, Celso de Melo, Flora D. Salim

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Toan Nguyen, Yang Liu, Trung Le, Celso de Melo, Flora D. Salim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Overcrowded Room"

Imagine you are trying to learn a new skill, like juggling. You have a coach (the AI optimizer) who tells you how to move your hands.

In standard AI training, the coach pays the most attention to the easiest, most common moves. If you are juggling 100 red balls and only 1 blue ball, the coach spends 99% of their time correcting your red-ball technique. The blue ball gets ignored.

The paper argues that this isn't just a problem when learning new tasks one after another (Continual Learning). It happens every single step of standard training. The "loud" instructions (common data) drown out the "quiet" but useful instructions (rare data). Over time, the AI forgets how to handle the rare stuff because it was never revisited. This is called Forgetting.

The Solution: FOGO (The Smart Coach)

The authors introduce FOGO, a new "coach" for AI that solves this problem. It doesn't just tell the AI what to do; it remembers what the AI used to do and protects those memories.

FOGO works like a three-part system:

1. The "Equalizer" (Spectral Orthogonalization)

Imagine a music mixer where the bass (common data) is so loud it drowns out the violin (rare data).

What FOGO does: It acts like a smart equalizer. It flattens the volume of the loud bass so the violin can be heard. It ensures that even the rare, quiet directions of learning get a fair chance to influence the AI's brain, rather than being crushed by the dominant trends.

2. The "Pocket Notebook" (Compact Codebook Memory)

Usually, to remember the past, an AI needs to store huge amounts of old data (like a massive library of past examples). This is slow and expensive.

What FOGO does: Instead of storing the whole library, FOGO keeps a tiny Pocket Notebook.
- It uses a trick called Random Projection. Imagine taking a complex 3D sculpture and casting a shadow of it on a 2D wall. You lose some detail, but you keep the shape and the distance between the parts.
- FOGO casts the AI's learning directions into this tiny 2D shadow space. It writes down the "gist" of important past directions as simple dots (centroids) in this notebook.
- Because it's just a notebook of dots, it takes up almost no space (megabytes instead of gigabytes) and is very fast to read.

3. The "Traffic Cop" (Conflict Resolution)

Every time the AI tries to learn something new, FOGO checks its Pocket Notebook.

The Check: "Hey, if you move your hand this way (new instruction), will you break the memory of how you held the blue ball (old instruction)?"
The Fix:
- Short-term: If the new move is too similar to a dominant trend, FOGO gently nudges it away so the rare stuff isn't ignored right now.
- Long-term: If the new move threatens to erase a critical memory from a previous task, FOGO acts as a traffic cop. It blocks the move or redirects it so it slides around the old memory instead of crashing into it.

How It Works in Real Life (According to the Paper)

The authors tested FOGO in four different scenarios, and it consistently outperformed the standard coaches (Adam and Muon):

The Unfair Classroom (Class-Imbalanced Learning): When an AI is trained on data where some classes are rare (like 10% of the data), standard AI forgets the rare classes. FOGO remembered them much better, improving accuracy on those rare items without hurting the common ones.
The Shape-Shifting World (Continual Visual Learning): When the AI had to learn to recognize objects in different styles (e.g., photos, cartoons, sketches) one after another, FOGO didn't forget the old styles when learning the new ones. It kept its knowledge intact better than other methods, even without storing old pictures.
The Talking Robot (LLaVA-7B Fine-tuning): When fine-tuning a large multimodal model (one that sees and speaks), FOGO helped the robot remember how to answer different types of questions (counting, reasoning, location) without losing its ability to do the previous ones.
The Language Learner (GPT-2 Pretraining): Even when just learning to predict the next word in a sentence (standard training), FOGO learned faster and reached a lower error rate than the standard methods.

The Bottom Line

The paper claims that forgetting is a universal problem, not just a "continual learning" problem. It happens whenever loud data drowns out quiet data.

FOGO fixes this by:

Making sure the quiet voices are heard (Orthogonalization).
Keeping a tiny, efficient memory of what matters (Random Projection Codebook).
Checking every step to ensure new learning doesn't erase old learning (Conflict Resolution).

The result is an AI that learns faster, remembers more, and doesn't need a massive hard drive full of old data to do it.

Technical Summary: FOGO (Forgetting-aware Orthogonalization Optimizer)

1. Problem Statement

The paper argues that forgetting is not merely a failure mode specific to continual learning (CL) but a fundamental optimization phenomenon inherent in standard stochastic training. The authors posit that during standard training, dominant mini-batch gradients suppress rare but useful update directions, causing short-term forgetting at every step. When these suppressed directions are not revisited, the losses compound into long-term forgetting, which is the classical failure mode of continual learning.

Existing solutions often treat standard training and task-sequential CL as separate problems. Gradient-interference methods (e.g., OGD, GEM) attempt to resolve conflicts but often rely on storing raw data or full-rank gradient bases, which do not scale to billion-parameter models. Conversely, matrix-aware optimizers (e.g., Muon) improve conditioning via orthogonalization but lack mechanisms to explicitly preserve historical update directions over time.

2. Methodology: FOGO

The authors introduce FOGO, a scalable optimizer designed to continuously detect and resolve gradient interference across both standard and continual learning regimes. FOGO operates without storing raw data, relying instead on a compact, projected memory of gradient directions.

Core Mechanisms

Spectral Orthogonalization:
FOGO orthogonalizes momentum updates to prevent dominant singular directions from monopolizing the optimization trajectory. By flattening the singular spectrum (using Newton–Schulz iterations), it ensures that weaker but coherent update directions remain visible and contribute to the update, rather than being overwritten by high-energy modes.
Dual-Stream Momentum:
To capture both stable trends and emerging directions, FOGO maintains two momentum states:
- Slow Stream ( $B^{(s)}$ ): Captures persistent trends.
- Fast Stream ( $B^{(f)}$ ): Tracks recent changes.
  These are fused via spherical interpolation (SLERP) to balance plasticity and stability.
Compact Directional Memory via Random Projection:
Instead of storing raw gradients, FOGO projects high-dimensional updates into a low-dimensional space ( $d \ll |\theta|$ ) using a fixed Rademacher random projection matrix $\Pi$ .
- Theoretical Basis: Leveraging the Johnson–Lindenstrauss lemma, this projection preserves pairwise distances between gradient vectors with high probability, maintaining the geometric structure of the gradient distribution.
- Codebook: The projected updates are summarized into a spherical codebook (centroids) using Exponential Moving Average (EMA) statistics. This codebook acts as a memory of representative past directions.
Conflict Resolution:
At each step, FOGO compares the current orthogonalized update against the codebook to resolve conflicts:
- Short-Term Filtering: Identifies and suppresses components that are anti-aligned (conflicting) with stored directions or overly aligned (risking collapse) with dominant local directions. This is achieved via lightweight orthogonal correction in the projected space.
- Long-Term Protection: After a task is completed, important centroids are frozen with importance weights. During subsequent tasks, updates are constrained to remain nearly orthogonal to these frozen directions, preventing catastrophic forgetting of sensitive past-task knowledge.
Proximal Lifting:
Since the random projection $\Pi$ is not invertible, the corrected update in the projected space is lifted back to the original high-dimensional parameter space via a proximal least-squares step. This ensures the correction is consistent with the projected constraints while minimizing distortion in the original space.

3. Key Contributions

Unified View of Forgetting: The paper reframes forgetting as a general optimization issue arising from gradient interference in both standard and continual learning, rather than a distinct CL problem.
Scalable Optimizer-Level Memory: FOGO introduces a memory mechanism that operates entirely in a low-dimensional projected space. This allows for the tracking of gradient geometry and conflict resolution without storing raw data or performing expensive full-rank decompositions, making it scalable to large models (e.g., LLaVA-7B, GPT-2).
Dual-Mechanism Design: The combination of spectral orthogonalization (to expose rare directions) and directional memory (to preserve them) addresses both short-term suppression and long-term accumulation of forgetting.
Theoretical Guarantees: The paper provides theoretical analysis (Theorem 1, Corollaries 2 & 3) demonstrating how off-subspace directions are suppressed under standard momentum and how FOGO's memory protection preserves learned directions while leaving unconstrained subspaces free for adaptation.

4. Experimental Results

FOGO was evaluated across four distinct settings, consistently outperforming Adam and Muon (and other CL baselines like UPGD, CBP, FIRE):

Class-Imbalanced Classification (CIFAR-10): In a setting with 5 rare classes (10% data), FOGO improved rare-class accuracy by ~2–3% and reduced rare-class forgetting by ~6% compared to Adam and Muon. Ablation studies confirmed that short-term filtering is the primary driver of this improvement.
Continual Visual Learning:
- Domain-IL (PACS): FOGO achieved the lowest average forgetting (AF) without replay, outperforming Muon by 2.67 points in AF. With replay (ER), it further improved performance.
- Class-IL (CIFAR-10): FOGO achieved higher accuracy and lower forgetting than Adam, Muon, and UPGD.
Continual Fine-tuning (LLaVA-7B): On the Continual VisualQA benchmark (10 reasoning tasks), FOGO improved average performance by ~2 points and reduced total forgetting by ~10 points compared to AdamW and Muon. Analysis showed FOGO induces more balanced expert feature variance, mitigating expert collapse.
Language Pretraining (GPT-2): In standard unsupervised pretraining on OpenWebText, FOGO converged faster than Adam and Muon, particularly in the early stages (first 30K steps), suggesting benefits even in non-continual settings.

5. Significance and Claims

The paper claims that forgetting is a general optimization problem solvable at the optimizer level. By resolving gradient interference through spectral orthogonalization and a compact, data-free directional memory, FOGO offers a practical route to retaining knowledge across both standard and continual learning scenarios.

The authors emphasize that their approach does not require storing raw data or previous model snapshots, making it suitable for large-scale models where memory is a constraint. While FOGO introduces modest computational overhead compared to Adam (due to dual-stream orthogonalization and projection operations), the paper argues this is acceptable given the consistent gains in convergence, performance, and knowledge retention. The work suggests that future solutions to forgetting should integrate optimization strategies with architectural and data strategies, with FOGO addressing the optimization axis.