💻 computer science

CNN Regularization and Model Capacity: An Empirical Investigation of L1, L2, and Dropout in Hand Sign Image Classification

This empirical study demonstrates that in hand sign image classification, increasing CNN capacity does not guarantee better performance and that dropout regularization, particularly at architecture-dependent rates, significantly outperforms L1 and L2 penalties, with a medium-sized model achieving the highest accuracy of 96.66%.

Original authors: Chaitanya Patil

Published 2026-08-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chaitanya Patil

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of computer vision, machines are learning to see. They do this by using digital brains called convolutional neural networks, which are designed to recognize patterns in images just as human eyes and brains do. These systems are incredibly powerful, capable of identifying everything from street signs to medical scans. However, there is a catch. When these digital brains become too large and complex, they can start to memorize the specific examples they are shown rather than learning the general rules of how to recognize an object. This is known as overfitting. Imagine a student who memorizes the answers to a specific practice test but fails the real exam because they cannot apply what they learned to new questions. To prevent this, scientists use techniques called regularization. These are methods that gently constrain the learning process, forcing the system to find simpler, more robust patterns that work on data it has never seen before. The question researchers have been asking is whether making the digital brain bigger always helps, or if there is a sweet spot where the size of the brain and the strength of the constraints work best together.

A researcher at the Indian Institute of Technology Kharagpur set out to answer this question by building a series of digital brains to recognize hand signs. The task was to teach a computer to distinguish between six different hand gestures, each representing a number from one to six. The researcher created three versions of the same computer vision system, each with a different level of capacity. The first was a small system, the second was medium-sized, and the third was large. The only difference between them was the number of internal connections they possessed; the large system had many more pathways for information to travel through than the small one. The goal was to see if simply adding more connections would make the system smarter, or if it would just make it more prone to memorizing the training data.

The experiments began with a surprising discovery. When the systems were left to learn without any constraints, the medium-sized system performed the best. It correctly identified the hand signs in the test images about 91.67 percent of the time. The large system, despite having the most capacity to learn, performed worse, achieving only about 83.33 percent accuracy. This happened because the large system was so powerful that it memorized the specific details of the training images, including random noise, rather than learning the true shape of the hand signs. It was like a student who studied the practice test so hard that they remembered the exact order of the questions but failed to understand the underlying concepts. The medium system, with fewer connections, was forced to find a balance, learning the essential features without getting distracted by the noise.

To improve these results, the researcher applied different techniques to stop the systems from overfitting. One method involved adding a penalty for having large internal weights, which is similar to encouraging the system to keep its internal settings simple. Another method involved randomly turning off parts of the system during training, forcing it to rely on different pathways to solve the problem. This technique, known as dropout, proved to be the most effective tool. When applied to the medium-sized system, it pushed the accuracy up to 96.66 percent, the highest score achieved in the entire study.

The study also revealed that the best way to use these tools depends on the size of the system. For the small system, a gentle amount of random turning off worked well. For the medium system, a moderate amount was best. But for the large system, which had the most capacity to memorize, the researchers found that a stronger dose of random turning off was necessary to get good results. This suggests that larger systems need stronger constraints to keep them from memorizing the training data. The researcher also tested combining different methods, but found that simply stacking them together did not automatically make the system better. In fact, the best results came from using a single, well-tuned method rather than a complex mix.

The findings were checked for reliability by running the experiments multiple times with different starting conditions, and the main results held up. The study concludes that building a bigger computer vision system does not guarantee better performance. Instead, the size of the system must be matched with the right amount of constraint. A medium-sized system with the right balance of constraints can outperform a much larger one. This work provides a clear, practical guide for how to tune these digital brains, showing that in the quest for artificial intelligence, sometimes less is more, and the strength of the rules matters just as much as the size of the brain.

Technical Summary: CNN Regularization and Model Capacity

Problem Statement
Convolutional Neural Networks (CNNs) are powerful tools for image classification but face significant challenges in generalization when model capacity is high relative to the size of the training dataset. High-capacity models often memorize training examples rather than learning generalizable patterns, leading to overfitting. While regularization techniques such as L1 penalization, L2 weight decay, and dropout are standard methods to mitigate overfitting, the specific interaction between model capacity and the effectiveness of these strategies remains an active area of investigation. It is unclear whether larger models require proportionally stronger regularization or if smaller models benefit from weaker constraints to avoid underfitting.

Methodology
This study presents a systematic empirical investigation using a six-class hand sign image classification task. The research employs a controlled experimental framework with the following components:

Dataset: A publicly available dataset of 1,080 training images and 120 test images (64×64×3 RGB) representing hand signs 1 through 6. Images were normalized by dividing pixel values by 255.0, and labels were one-hot encoded.
Architecture: Three CNN architectures were designed with identical depth but varying width (capacity):
- Small: 16 → 32 → 64 filters/units.
- Medium: 32 → 64 → 128 filters/units.
- Large: 64 → 128 → 256 filters/units.
  All models utilized two convolutional layers (3×3 kernels, ReLU, 2×2 max pooling), a fully connected layer, and an output layer with softmax.
Training Configuration: Models were trained using the Adam optimizer (learning rate 0.001) for 1,000 iterations with He Normal initialization. The primary loss function was cross-entropy.
Regularization Strategies: The study evaluated:
- L1 Regularization: Coefficients ( $\lambda_1$ ) of 0.001, 0.01, and 0.05.
- L2 Regularization: Coefficients ( $\lambda_2$ ) of 0.1, 0.3, 0.6, 1.0, and 2.0.
- Dropout: Rates of 0.1, 0.3, and 0.5 applied to the first fully connected layer.
- Combined L1+L2: Simultaneous application with $\lambda_1=0.05$ and $\lambda_2=0.6$ .
Evaluation: Primary metrics were test accuracy and training accuracy. Experiments were conducted primarily with random seed 42, with reproducibility checks performed using seeds 1137 and 2024.

Key Results
The experiments yielded several distinct findings regarding the interplay between capacity and regularization:

Capacity vs. Performance: Increasing model capacity did not automatically improve test accuracy. The unregularized Medium model achieved the highest baseline accuracy (91.67%), outperforming the Large model (83.33%) and the Small model (84.16%). The Large model achieved 100% training accuracy but failed to generalize, suggesting it memorized noise.
Dropout Efficacy: Dropout produced the strongest performance across all configurations. The optimal dropout rate shifted toward stronger regularization as model capacity increased:
- Small Model: Best performance at 0.1 or 0.3 (90.83%).
- Medium Model: Best performance at 0.3, achieving the study's peak accuracy of 96.66%.
- Large Model: Best performance at 0.5 (95.00%).
L1 and L2 Sensitivity:
- L2: Moderate regularization ( $\lambda_2=0.1$ ) improved the Medium model to 95.00%, but performance degraded significantly at high strengths ( $\lambda_2=2.0$ ), indicating over-regularization.
- L1: Showed architecture-dependent sensitivity. For the Small model, stronger L1 ( $\lambda_1=0.05$ ) improved accuracy to 91.66%, whereas the Large model performed best with the weakest L1 ( $\lambda_1=0.001$ ).
Combined Regularization: The combination of L1 and L2 did not outperform the strongest individual regularizers, yielding a mean test accuracy of 87.50% for the Medium model.
Reproducibility: Selected checks confirmed the stability of the main findings, though some variation was observed (e.g., Medium model with dropout 0.3 dropped from 96.66% to 93.34% under seed 1137).

Significance and Claims
The paper positions itself as an exploratory empirical study rather than a source of universal theory. Its primary contributions are:

Providing a controlled comparison of regularization strategies across three distinct capacity levels within a single architecture family.
Demonstrating that model capacity alone is not a predictor of generalization performance; a medium-sized model can outperform a larger one if the latter is prone to overfitting without appropriate constraints.
Establishing an empirical trend where larger models benefit from stronger dropout rates, aligning with the intuition that higher capacity requires more aggressive regularization to prevent co-adaptation.
Highlighting that regularization methods do not necessarily compose linearly, as the combined L1+L2 approach failed to surpass individual methods.

The author explicitly states that these conclusions are restricted to the specific architectures, dataset, hyperparameter ranges, and conditions investigated. They do not claim to provide a universal scaling law for regularization but offer empirical observations to inform future investigations into capacity-regularization interactions.

Technical Summary: CNN Regularization and Model Capacity

More like this