Automatic Construction of Pattern Classifiers Capable of Continuous Incremental Learning and Unlearning Tasks Based on Compact-Sized Probabilistic Neural Network

This paper proposes a novel, hyperparameter-free probabilistic neural network model that utilizes a simple one-pass algorithm to automatically construct compact classifiers capable of achieving high performance in standard classification tasks while supporting continuous incremental learning and unlearning without iterative matrix approximations.

The Gist

Imagine you are running a library that needs to sort books into categories.

The Old Way: The "Hard-Drive" Library (Deep Learning)

Most modern AI (like Deep Learning) works like a massive, rigid library built all at once.

• The Problem: If you want to add a new genre of books (Incremental Learning), you have to reorganize the entire library. If you try to just shove new books on the shelves, you accidentally knock over old books, and the librarian forgets where they were (this is called "Catastrophic Forgetting").
• The Fix: To stop forgetting, the library has to keep a "backup copy" of every old book it ever saw. This takes up a huge amount of space and is very slow to manage.
• The Tuning: Before opening, you have to spend weeks tweaking the size of the shelves, the lighting, and the sorting rules (Hyperparameters). If you get it wrong, the whole system fails.

The New Way: The "Modular" Library (This Paper's Solution)

The authors, Tetsuya Hoya and Shunpei Morita, propose a new kind of library called a Compact-Sized Probabilistic Neural Network (CS-PNN). Think of it not as one giant building, but as a set of modular, pop-up tents.

Here is how their system works, using simple analogies:

1. No Blueprints Needed (No Hyperparameter Tuning)

Usually, building an AI is like trying to bake a cake without a recipe; you have to guess how much sugar and flour to use.

• The CS-PNN Solution: This system is like a self-expanding tent. You don't need to guess the size. You just start with one tent. As new guests (data) arrive, the system automatically adds more tents or rearranges the existing ones. It figures out the size and shape as it goes, requiring zero manual tuning.

2. The "One-Pass" Construction

• Old Way: The old library tries to memorize every single book by reading them over and over again (iterative training).
• New Way: The CS-PNN is a one-pass learner. It looks at a book once. If the book fits in an existing tent, it goes in. If the book is weird and doesn't fit, the system instantly builds a new, small tent just for that book. It never looks back. It's incredibly fast.

3. Learning New Things (Incremental Learning)

Imagine you are a librarian and suddenly, "Science Fiction" becomes a new category.

• Old Way: You have to tear down the whole library and rebuild it, or risk losing the "History" section.
• New Way: The CS-PNN just pops up a new tent labeled "Science Fiction." The "History" tent stays exactly where it is, untouched. Because each category has its own dedicated space (subnet), adding new categories doesn't corrupt the old ones.

4. Forgetting Things on Purpose (Unlearning)

Sometimes, you need to remove a category. Maybe "Science Fiction" was a mistake, or you need to delete specific books due to privacy laws (Unlearning).

• Old Way: You have to "un-train" the AI, which is like trying to erase a memory from a human brain without hurting the rest of their mind. It's hard and often leaves scars.
• New Way: With the CS-PNN, you just take down the specific tent for that category. The rest of the library remains perfectly intact. It's like removing a Lego block without breaking the whole castle.

5. The "Smart Radius" (The Secret Sauce)

The magic behind this system is how it decides how big each tent should be.

• In the past, the size of the tent was fixed based on a guess.
• In this new system, the size of the tent shrinks or grows dynamically based on how crowded the area is. If the "Science Fiction" section gets too crowded with new books, the system automatically adjusts the size of the tents to fit them all perfectly without them spilling over into other sections.

The Results: What Did They Find?

The authors tested this on nine different "libraries" (datasets), ranging from recognizing handwritten digits (like MNIST) to identifying letters.

1. Smaller Footprint: The new system used far fewer "tents" (hidden units) than the old Probabilistic Neural Networks, making it much more compact.
2. Just as Smart: Even though it was smaller, it performed just as well as the heavy, complex Deep Learning models (MLPs) in standard tests.
3. The Real Winner: When it came to adding and removing categories continuously, the Deep Learning models failed miserably (they forgot things or got confused). The CS-PNN kept its cool, adding and removing categories effortlessly without losing its memory of the other categories.

The Bottom Line

This paper introduces a flexible, self-building AI that doesn't need a PhD to set up. It's like a Lego set that builds itself:

• You don't need to plan the whole structure in advance.
• You can add new rooms whenever you want.
• You can knock down rooms you don't need anymore without the whole building collapsing.
• It's fast, efficient, and perfect for a world where data is constantly changing.

It's a move away from "rigid, heavy brains" toward "agile, modular brains" that can learn and unlearn in real-time.

Technical Summary

1. Problem Statement

The paper addresses three critical challenges in modern machine learning, particularly within the context of deep learning (DL) and pattern classification:

• Hyperparameter Sensitivity: Traditional Deep Neural Networks (DNNs) and even standard Probabilistic Neural Networks (PNNs) require extensive manual tuning of hyperparameters (e.g., learning rates, batch sizes, number of hidden units, radius values for RBFs). This process is resource-intensive, time-consuming, and often relies on trial-and-error.
• Catastrophic Forgetting in Incremental Learning: Standard DNNs suffer from "catastrophic forgetting" when trained on new data sequentially. New data corrupts the representation of previously learned classes. While "replay" methods (e.g., iCaRL) exist, they require storing subsets of old data, which is not a true solution for memory-constrained or privacy-sensitive environments.
• Lack of Unified Unlearning Mechanisms: Existing methods for "machine unlearning" (removing specific data or classes from a trained model) often require complex matrix operations, model snapshots, or retraining from scratch. There is a lack of unified schemes that can dynamically handle both incremental learning (adding classes/data) and unlearning (removing classes/data) without heavy computational overhead or hyperparameter re-tuning.
• Scalability of Standard PNNs: While PNNs naturally support incremental learning due to their local data representation, the standard PNN requires a hidden unit for every single training sample. This leads to massive network sizes, overfitting, and slow inference times, especially with large datasets.

2. Methodology

The authors propose a Compact-Sized Probabilistic Neural Network (CS-PNN) constructed via a novel, automatic, one-pass algorithm.

Core Architecture

• Model: A variant of the PNN featuring a three-layer structure: Input, Hidden (Radial Basis Function - RBF), and Output.
• Topology: The hidden-to-output layer is topologically equivalent to a collection of independent sub-networks (subnets), where each subnet corresponds to a specific class. This structural modularity allows for easy addition or removal of classes.

The Proposed Algorithm

The construction and reconstruction of the CS-PNN rely on a data-driven, one-pass algorithm with no hyperparameter tuning.

1. Dynamic Radius Calculation:

   • Unlike standard PNNs that use a fixed, heuristically chosen radius ($\sigma$), the CS-PNN updates the radius dynamically based on the current state of the network.
   • Formula: $\sigma = \frac{d_{max}}{k}$
     • $d_{max}$: The maximum distance between the current input pattern and all existing centroid vectors (centroids of RBFs). This value is updated in parallel with feed-forward computation.
     • $k$: The current number of classes accommodated in the network.
   • This allows the network to adapt to varying pattern spaces as new data arrives or classes are removed.

2. Automatic Network Growth (Construction/Reconstruction):

   • The algorithm processes training data in a single pass.
   • Condition for Adding a New RBF: A new RBF unit (with a new centroid) is added only if the incoming training data is incorrectly classified by the current network.
   • Condition for Updating: If the data is correctly classified, the algorithm updates the centroid of the most activated RBF within the corresponding class subnet using a running average (Eq. 4).
   • Result: This eliminates the need for a threshold hyperparameter ($\theta$) used in previous approaches to decide when to add units.

3. Unlearning Mechanism:

   • Instance-wise Unlearning: Specific RBF units responsible for unwanted samples are simply removed from the network.
   • Class-wise Unlearning (CDL): Entire subnets corresponding to specific classes are unloaded.
   • Efficiency: Unlearning involves no retraining or complex matrix operations; it is a structural removal. The radius parameter ($\sigma$) is automatically recalculated during the subsequent testing or reconstruction phase.

3. Key Contributions

• Hyperparameter-Free Construction: The proposed method requires zero manual hyperparameter tuning. The network size, structure, and radius parameters are determined entirely by the data and the algorithm.
• True Incremental and Decremental Learning: The CS-PNN supports continuous class-incremental learning (CIL) and class-decremental learning (CDL) without catastrophic forgetting or the need to store replay buffers.
• Compactness: The algorithm generates a significantly smaller number of hidden units (RBFs) compared to the original PNN (which uses one unit per training sample), effectively solving the scalability and overfitting issues of standard PNNs.
• Unified Framework: It provides a single, unified scheme for standard classification, incremental learning, and unlearning tasks, contrasting with DNNs which often require distinct, complex strategies for these scenarios.

4. Experimental Results

The authors evaluated the CS-PNN on nine public datasets (including MNIST, Isolet, Letter-Recognition, and UCI datasets) across three scenarios:

A. Standard Classification

• Performance: The CS-PNN achieved classification accuracy comparable to Multi-Layer Perceptron (MLP) networks and the original PNN.
• Efficiency: The CS-PNN used 4% to 46% of the hidden units required by the original PNN.
• Comparison: While MLPs sometimes achieved slightly higher accuracy, they required significant hyperparameter tuning and were unstable across runs. The CS-PNN provided stable, high-performance results with a fixed, automatic construction process.

B. Class Incremental Learning (CIL)

• Setup: Classes were added sequentially (1 to 4 classes at a time).
• Comparison: The CS-PNN was compared against an MLP trained with iCaRL (a popular replay-based method).
• Outcome:
  • The iCaRL-based DNN suffered from catastrophic forgetting, with accuracy dropping drastically (e.g., from ~80% to ~25%) as more classes were added.
  • The CS-PNN maintained stable performance and even improved as more classes became available, demonstrating its ability to track the evolving pattern space without forgetting.

C. Continuous Unlearning and Incremental Learning (CUIL)

• Setup: The network underwent cycles of removing classes (unlearning) and re-adding them (incremental learning).
• Outcome: The CS-PNN successfully handled these dynamic structural changes.
  • Classification accuracy remained relatively stable despite the fluctuation in network size.
  • The number of RBF units dynamically adjusted to the complexity of the current pattern space.
  • The system demonstrated robustness in "forgetting" and "re-learning" without manual intervention.

5. Significance and Conclusion

The paper presents a significant step toward autonomous and adaptive machine learning systems.

• Practicality: By removing the need for hyperparameter tuning and replay buffers, the CS-PNN offers a practical solution for real-world applications where data distributions change over time (non-stationary environments) and where data privacy (unlearning) is paramount.
• Computational Efficiency: The algorithm avoids iterative optimization (like backpropagation) and complex matrix inversions, relying instead on simple distance calculations and centroid updates.
• Future Outlook: The authors suggest that while the testing mode of PNNs can be computationally heavy due to the number of RBFs, parallel computing environments can mitigate this, potentially making CS-PNNs as fast as DNNs for inference.

In summary, the CS-PNN offers a lightweight, self-configuring, and robust alternative to deep learning for dynamic pattern classification tasks, effectively balancing performance, model size, and adaptability.