💻 computer science

SimGrowNet: Efficient Neural Growth with Similarity-Based Redundancy Control

SimGrowNet is an efficient neural growth method inspired by human neurogenesis that progressively expands compact models during training while minimizing neuron redundancy through similarity-based constraints, thereby enhancing feature diversity, mitigating shortcut learning, and outperforming existing baselines on multiple image recognition datasets.

Original authors: Taigo Sakai, Kazuhiro Hotta

Published 2026-08-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Taigo Sakai, Kazuhiro Hotta

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, computers learn to recognize the world by building vast networks of digital connections, mimicking the way the human brain processes information. These systems, known as deep neural networks, have become incredibly skilled at tasks like identifying objects in photographs. However, to make them smarter, researchers often simply make them bigger, adding more and more connections until the system becomes a massive, energy-hungry machine that takes a long time to train. This approach has a significant downside: as the network grows, it often starts to repeat itself. Just as a team of workers might become inefficient if everyone is assigned the exact same task, these digital networks can develop redundant parts where multiple connections learn to spot the same simple details, like the color of the sky, rather than the complex shape of the object itself. This repetition wastes computing power and can lead the system to make mistakes by focusing on the wrong clues.

Researchers at Meijo University in Japan have proposed a different way to build these intelligent systems, one that grows more like a living organism than a static machine. Instead of starting with a massive network and trying to cut it down, or adding new parts without a plan, they developed a method called SimGrowNet that starts small and expands carefully. The core idea is inspired by how the human brain continues to form new neurons throughout adulthood. The researchers realized that when new connections are added to a digital network, they often end up doing the exact same job as the old ones. To fix this, they introduced a rule that forces every new connection to be different from the ones already there. They did this by measuring how similar the new parts were to the old ones and gently nudging them to become distinct, ensuring that the network learns a wide variety of features rather than just repeating the same few.

The team tested this approach on three different sets of images, ranging from simple drawings of animals to complex photographs of real-world scenes. They compared their new method against older techniques that simply added random new connections or copied existing ones. The results showed that by keeping the new parts unique, the network became much better at its job. On a standard test of recognizing one hundred different categories of images, the new method improved accuracy by nearly two percent compared to the previous best approach. On another test involving a thousand categories, it also outperformed the competition. These numbers might seem small, but in the field of artificial intelligence, even a fraction of a percent can mean the difference between a system that is useful and one that is not. More importantly, the researchers could see exactly why it worked. By using a visualization tool that highlights what the computer is looking at, they found that the new method focused on the entire object, such as the body and head of a bird, rather than getting distracted by the background or just a small edge.

This shift in focus suggests that the network was no longer taking shortcuts. In many cases, older methods would learn to guess the answer based on a single, easy clue, like assuming a picture of a bird is actually a plane because both have a blue sky in the background. The new method, by forcing the connections to be diverse, prevented the system from relying on these simple tricks. It learned to look at the whole picture. The researchers also discovered that this benefit came not just from adding more parts, but from the way those parts were controlled. Even when they stopped adding new connections and just applied the rule to a fixed network, the system still performed better. This indicates that the key to success was not just having more capacity, but having a network where every part had a unique role to play.

The study also addressed a common problem in training these systems: instability. When a network changes its structure rapidly, it can sometimes become confused or lose the knowledge it had already learned. The researchers added a safety mechanism to their method that prevented the new connections from changing too wildly, ensuring that the system remained stable while it grew. They found that the timing of these changes mattered; growing the network too frequently or too slowly could reduce the benefits, so they had to find a specific rhythm that worked best for each type of image. While the method requires some careful tuning to get right, the results suggest that building efficient, smart networks does not require massive scale. Instead, it requires a focus on diversity and quality.

This work offers a promising path forward for creating artificial intelligence that is both powerful and efficient. By learning to grow in a way that avoids repetition, these systems can achieve high performance without the massive energy costs associated with today's largest models. The researchers suggest that this approach could be particularly useful for devices with limited power, such as smartphones or sensors, where every bit of computing power counts. While the method was tested on image recognition, the principle of ensuring diversity in learning could apply to other areas where computers need to understand complex patterns. The study concludes that the future of efficient artificial intelligence may lie not in building bigger, but in growing smarter, ensuring that every new part of the system brings something unique to the table.

Technical Summary of SimGrowNet: Efficient Neural Growth with Similarity-Based Redundancy Control

Problem Statement
Deep neural networks have achieved remarkable performance in image recognition, yet their increasing size leads to substantial computational costs and prolonged training times. Conventional optimization strategies, such as fine-tuning, knowledge distillation, and pruning, often face limitations regarding stability or the risk of discarding critical information. Furthermore, existing dynamic network growth methods—specifically those that add neurons randomly or duplicate existing ones—often introduce neuronal redundancy. This redundancy causes new neurons to extract features overlapping with existing ones, leading to inefficient use of model capacity, biased feature extraction (e.g., focusing on background regions rather than the target object), and "shortcut learning," where models rely on superficial cues rather than robust semantic features.

Methodology: SimGrowNet
Inspired by biological neurogenesis, SimGrowNet proposes a constructive approach where a compact neural network progressively expands during training. Unlike previous growth methods, SimGrowNet explicitly controls redundancy through similarity-based constraints. The core mechanism involves two primary steps executed after neuron addition or duplication:

Cosine Similarity Computation: The method computes a pairwise cosine similarity map ( $C_{sim}$ ) between the weight vectors of neurons in a layer. This map is derived from the normalized weight matrix ( $W_{norm}$ ), where $C_{sim} = W_{norm}W_{norm}^T$ . Cosine similarity is chosen because it captures the directional alignment of weight vectors, representing functional similarity in feature extraction regardless of magnitude.
Similarity Minimization and Regularization: The objective is to minimize the mean absolute cosine similarity ( $\mu_{sim}$ ) among neurons (excluding self-similarity). By driving this value toward zero, the method encourages neurons to become orthogonal, thereby promoting functional diversity and preventing redundant feature extraction.

To ensure training stability and prevent the optimization of diversity from overwriting valid pre-existing feature detectors, a weight-change regularization term ( $L_{weight}$ ) is introduced. This term penalizes abrupt changes in the sum of weights between training steps.

The total loss function is formulated as:
$\min L_{total} = L_{task} + \alpha(\mu_{sim} + \lambda \cdot L_{weight})$
where $L_{task}$ is the standard task-specific loss (e.g., Cross-Entropy), $\alpha$ is a weighting coefficient, and $\lambda$ balances the weight regularization.

Key Contributions
The paper outlines three primary contributions:

Similarity-Regularized Growth: A strategy that minimizes inter-neuron cosine similarity post-expansion to reduce redundancy in newly added neurons, ensuring they learn distinct, complementary features.
Stability Mechanism: The incorporation of a weight-change regularization term to stabilize dynamically expanded networks and prevent weight instability during the growth phase.
Redundancy-Aware Framework: A formulation linking neuron-similarity control to improved feature diversity and the mitigation of shortcut learning, applicable to compact CNN expansion.

Experimental Results
SimGrowNet was evaluated on CIFAR-10, CIFAR-100, and ImageNet-1k datasets using VGG16 and MobileNet architectures. It was compared against Random growth, Splitting Steepest Descent (SSD), and the Firefly algorithm.

Accuracy Improvements: SimGrowNet consistently outperformed baseline methods. Notable improvements included:
- CIFAR-10: +1.1% accuracy over Firefly (MobileNet: 92.56% vs. 91.89%).
- CIFAR-100: +1.95% accuracy over Firefly (MobileNet: 67.49% vs. 66.42%).
- ImageNet-1k: +0.9% accuracy over Firefly (MobileNet: 67.0% vs. 66.1%).
Qualitative Analysis (Grad-CAM): Visualizations demonstrated that while baseline methods (like Firefly) often focused on localized or background regions (e.g., sky for a bird), SimGrowNet attended to broader, semantically relevant object regions (e.g., the bird's body or head). This indicates a reduction in shortcut learning.
Ablation Studies:
- The combination of similarity minimization and weight regularization yielded the highest accuracy, confirming the necessity of both components.
- The method improved accuracy even in fixed-architecture settings (without actual neuron growth), suggesting that the similarity-based redundancy control itself is a primary driver of performance gains.
- Under data-scarce conditions (e.g., 10% of training data), SimGrowNet showed significant robustness, outperforming conventional models by 4.1% on CIFAR-10 and 5.8% on CIFAR-100.
Hyperparameter Sensitivity: Optimal performance was observed with an update iteration count ( $N$ ) of 15 and a penalty coefficient ( $\lambda$ ) of 0.01.

Significance and Claims
The authors claim that SimGrowNet offers a sustainable alternative to the paradigm of training massive models and subsequently pruning them. By constructing compact, high-performance models from scratch with controlled redundancy, the method aligns with "Green AI" principles, making it suitable for edge computing and resource-constrained environments.

The paper emphasizes that the primary benefit is not just marginal accuracy gains but improved model reliability. By forcing neurons to learn diverse features, SimGrowNet mitigates shortcut learning, ensuring the model relies on robust semantic features rather than background artifacts. The authors note that while the method introduces hyperparameters ( $N$ and $\lambda$ ) requiring tuning, the approach effectively enhances representation diversity and learning efficiency without modifying the fundamental network structure. Future work is identified as applying this method to Continual Learning to mitigate catastrophic forgetting and exploring its applicability to Reinforcement Learning and other architectures like Transformers.

More like this