Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learning

Inspired by the fly olfactory circuit, Fly-CL is a bio-inspired framework that enhances pre-trained model-based continual representation learning by efficiently resolving multicollinearity to reduce training time while maintaining or exceeding state-of-the-art performance.

The Gist

The Big Problem: The "Amnesia" of AI

Imagine you are teaching a student (an AI) to recognize animals. First, you show them pictures of dogs. They learn perfectly. Then, you show them pictures of cats. If you just keep teaching them new things, they often forget the dogs. This is called Catastrophic Forgetting.

To fix this, scientists usually use a "frozen" teacher (a pre-trained model) that has already seen millions of images. The AI just needs to learn how to sort these images into new buckets. However, there's a catch:

• The Multicollinearity Problem: Imagine the "dog" bucket and the "wolf" bucket are sitting right next to each other, almost touching. It's hard to tell them apart. In math terms, the features for different classes are too similar (correlated), making the AI confused.
• The Speed Problem: Fixing this confusion usually requires heavy, slow math. It's like trying to organize a library by hand-sorting every single book one by one. It takes too long for real-time applications.

The Solution: The Fly's Nose

The researchers looked at nature for a solution. They studied the olfactory (smell) system of a fruit fly.

A fly's brain is tiny, yet it can instantly distinguish between thousands of different smells (like a rose vs. a rotting apple) without getting confused or forgetting. How?

1. Expansion: When a smell enters the fly's nose, it gets blasted into a massive, high-dimensional space. Think of it like taking a small, crowded room and suddenly expanding it into a giant stadium.
2. Sparsity (The "Top-K" Rule): In this stadium, only a few specific seats light up for a specific smell. Most seats stay dark. This separates the smells clearly.
3. Inhibition: If two smells are too similar, the fly's brain actively suppresses the weak signals, forcing a clear "winner" to emerge.

Introducing Fly-CL: The Fly-Inspired Framework

The authors built Fly-CL, a new method that copies this fly brain process to help AI learn continuously without forgetting and without slowing down.

Here is how it works, step-by-step:

1. The "Stadium" Expansion (Sparse Random Projection)

Instead of keeping the AI's features in a small, crowded room, Fly-CL projects them into a massive, empty stadium (a high-dimensional space).

• Analogy: Imagine you have a messy pile of 100 colored marbles. It's hard to sort them. Fly-CL throws them into a giant field. Suddenly, the red marbles are far away from the blue ones. They are no longer tangled up.
• The Trick: To save time, the AI doesn't look at every seat in the stadium. It uses a "Sparse" map, meaning it only checks a few random spots. This makes the math incredibly fast.

2. The "Winner-Take-All" Filter (Top-K Operation)

Once the features are in the stadium, the system applies a filter. It keeps only the top K strongest signals and turns off the rest.

• Analogy: Imagine a noisy party where everyone is shouting. The fly brain (and Fly-CL) puts on noise-canceling headphones that only let the loudest, clearest voices through. All the background chatter (noise) is silenced. This makes the "dog" signal distinct from the "wolf" signal.

3. The "Smart Librarian" (Streaming Ridge Classification)

Finally, the system needs to learn which "seat" in the stadium belongs to which animal.

• The Old Way: Previous methods tried to calculate the perfect seat for every animal by re-reading the entire library every time a new book arrived. This was slow.
• Fly-CL's Way: It uses a "streaming" method. It updates its knowledge instantly as new data comes in, like a librarian who updates the catalog card the moment a book is returned, without re-shelving the whole library. It uses a mathematical trick (Ridge Classification) that naturally keeps the "dog" and "wolf" seats far apart, solving the confusion problem automatically.

Why is this a Big Deal?

The paper shows that Fly-CL is a superhero in two ways:

1. It's Lightning Fast: Because it skips the heavy math and uses the "sparse" (fewer connections) approach, it trains 90% faster than the current best methods.
   • Real-world impact: An AI on a self-driving car or a smartphone could learn new objects in real-time without lagging.
2. It's Smarter: Despite being faster, it doesn't sacrifice accuracy. In fact, it often gets better scores than the slower, more complex methods because it separates the data so cleanly.

The Takeaway

The researchers realized that for decades, AI engineers were trying to solve complex math problems with brute force. By looking at a tiny fruit fly, they realized the answer was expansion and sparsity.

Fly-CL is like giving the AI a "fly brain" upgrade: it takes messy, confusing data, spreads it out in a giant stadium, silences the noise, and instantly sorts it into clear categories. It proves that sometimes, the best way to build a super-intelligent machine is to copy the tiny, efficient brains of nature.

Technical Summary

1. Problem Statement

The paper addresses the challenges of Continual Learning (CL) using pre-trained models, specifically focusing on Class Incremental Learning (CIL) where models must learn new classes sequentially without access to previous data (exemplar-free).

• Current Paradigm: Representation-based CL methods typically use a frozen pre-trained encoder to extract features, compute class prototypes (centroids), and perform classification via similarity matching (e.g., cosine similarity).
• Key Bottlenecks:
  1. Multicollinearity: Features from different classes often exhibit high correlation in the embedding space. This leads to ambiguous decision boundaries and degrades the discriminative power of similarity matching.
  2. Computational Cost: Existing methods to mitigate multicollinearity (e.g., complex decorrelation techniques or extensive cross-validation for regularization) are computationally prohibitive, making them unsuitable for real-time, low-latency applications.
  3. Scalability: Many state-of-the-art (SOTA) methods rely on parameter updates (prompts/adapters) or storing multiple models, which increases storage and inference overhead.

2. Methodology: Fly-CL

Inspired by the fly olfactory circuit, the authors propose Fly-CL, a bio-inspired framework that achieves progressive decorrelation of features while maintaining extreme computational efficiency. The framework consists of three main stages:

A. Sparse Random Projection & Top-K Operation (PN $\to$ KC)

Mimicking the transformation from Projection Neurons (PNs) to Kenyon Cells (KCs) in the fly brain:

• High-Dimensional Expansion: Input features $v \in \mathbb{R}^d$ are projected into a much higher-dimensional space $h \in \mathbb{R}^m$ ($m \gg d$) using a fixed sparse random projection matrix $W$.
  • Sparsity: Each row of $W$ has only $p$ non-zero entries (sampled from $\mathcal{N}(0,1)$), reducing projection complexity from $O(m \cdot d)$ to $O(m \cdot p)$.
• Winner-Take-All Inhibition: A Top-K operation is applied to the high-dimensional vector, retaining only the $k$ largest activations and zeroing out the rest.
  • Effect: This mimics lateral inhibition in the fly brain, suppressing noisy dimensions and enhancing the separation of discriminative features.
  • Theoretical Guarantee: The authors prove that as long as $k$ is sufficiently large relative to $m$, the performance degradation is bounded and negligible (Theorem 4.2).

B. Streaming Ridge Classification (KC $\to$ MBON)

Mimicking the transformation from Kenyon Cells (KCs) to Mushroom Body Output Neurons (MBONs):

• Hebbian Learning Equivalence: The authors theoretically demonstrate that the biological Hebbian learning rule (with anti-Hebbian error correction and weight decay) is mathematically equivalent to Ridge Classification in the CL setting.
• Online Update: Instead of batch processing, the method maintains streaming statistics (Gram matrix $G$ and cross-term matrix $S$) updated incrementally as new tasks arrive.
• Adaptive Regularization: To avoid the high cost of grid-search cross-validation ($O(lm^3)$), the framework employs an Adaptive Generalized Cross-Validation (GCV) method. This analytically approximates the optimal regularization parameter $\lambda$ using Singular Value Decomposition (SVD) of the current task's data, reducing complexity to $O(n_t^2 m)$.
• Efficient Solver: The classifier matrix $C$ is computed using Cholesky factorization (exploiting the positive-definiteness of the regularized Gram matrix) rather than standard LU decomposition, halving the computational cost.

C. Inference

During inference, the system performs a sparse projection and Top-K operation on the input, followed by a simple dot product with the learned modulated prototypes $C$.

3. Key Contributions

1. Bio-Inspired Efficiency: Proposes the first CL framework that explicitly models the fly olfactory circuit's decorrelation mechanism (sparse expansion + lateral inhibition) to solve the multicollinearity problem in pre-trained models.
2. Theoretical Novelty: Establishes a theoretical bridge between biological Hebbian learning and Ridge Classification, proving that the KC $\to$ MBON pathway supports decorrelation.
3. Algorithmic Optimization: Introduces a highly efficient pipeline combining sparse random projection, adaptive GCV for regularization selection, and Cholesky factorization, drastically reducing training time.
4. State-of-the-Art Performance: Achieves accuracy comparable to or better than SOTA methods while significantly reducing computational overhead.

4. Experimental Results

The authors evaluated Fly-CL on CIFAR-100, CUB-200-2011, VTAB, ImageNet-R, and ImageNet-A using ViT-B/16 and ResNet-50 backbones.

• Accuracy:
  • On ViT-B/16, Fly-CL achieved 93.89% (CIFAR-100), 93.84% (CUB), and 96.54% (VTAB), outperforming or matching SOTA methods like RanPAC and EASE.
  • On ResNet-50, it improved accuracy by 1.89% over the best baseline on CIFAR-100, whereas other methods (like F-OAL) suffered significant degradation on CNNs.
• Efficiency (Training Time):
  • Drastic Reduction: Fly-CL reduced the post-extraction training time ($\tau_{post}$) by 91% on CIFAR-100 and 83% on CUB compared to the most efficient baselines.
  • Absolute Speed: On CUB-200-2011, Fly-CL took only 0.35 seconds per task for post-extraction training, compared to 33.83s for RanPAC and 122.48s for EASE.
• Robustness:
  • Demonstrated superior performance under severe domain shifts (ImageNet-R/A).
  • Showed robustness in longer task sequences (up to 20 tasks) and varying backbone architectures.
• Memory: Fly-CL exhibited the lowest peak memory usage among all compared methods.

5. Significance

• Bridging Biology and AI: The paper provides a concrete example of how biological neural circuit motifs (specifically the fly olfactory system) can inspire efficient algorithms for modern AI problems, validating the utility of "neuromorphic" principles in deep learning.
• Real-World Deployment: By drastically reducing training time and memory footprint without sacrificing accuracy, Fly-CL makes representation-based continual learning viable for edge computing and real-time applications where latency and resource constraints are critical.
• Scalability: The framework's compatibility with various backbones (Transformers and CNNs) and its ability to handle large-scale pre-trained models (tested up to Qwen2.5-VL-7B) suggest it is a scalable solution for the era of foundation models.

In conclusion, Fly-CL successfully resolves the trade-off between accuracy and efficiency in continual learning by leveraging biological principles of sparse coding and decorrelation, offering a fast, robust, and theoretically grounded alternative to existing parameter-heavy or computationally expensive methods.