An Empirical Study of Data Scale, Model Complexity, and Input Modalities in Visual Generalization

Technical Summary: An Empirical Study of Data Scale, Model Complexity, and Input Modalities in Visual Generalization

Problem Statement
Modern deep neural networks frequently exhibit over-parameterization and achieve strong performance in computer vision, yet the theoretical sources of their generalization capabilities remain difficult to explain using traditional statistical learning theory. While classical theories suggest that increasing model complexity beyond a certain point degrades generalization, empirical observations show that over-parameterized models often maintain or even improve test performance. This study addresses the lack of understanding regarding the specific roles of three fundamental, controllable variables: training data scale, model complexity, and input modalities. The research seeks to determine how these factors interact to influence generalization performance, particularly in the context of the "black-box" nature of deep learning.

Methodology
The study employs a two-stage empirical approach, moving from a controlled low-dimensional setting to complex image classification tasks.

1. Preliminary Experiment (Synthetic Data):

   • Objective: To establish a controlled baseline for observing the interaction between data scale and model complexity.
   • Setup: A one-dimensional nonlinear function $f(x) = \sin(2\pi x) + 0.5 \cos(5\pi x)$ was constructed with added Gaussian noise.
   • Variables: Model complexity was proxied by the degree of polynomial fitting ($D \in [0, 100]$). Data scale was varied between small ($N=50$) and large ($N=150$) training sets.
   • Metrics: Training, validation, and test Mean Squared Error (MSE) were analyzed to observe fitting behavior and generalization trends, specifically looking for "double descent" phenomena.

2. Main Experiments (Image Classification):

   • Datasets: CIFAR-10 (10 classes) and CIFAR-100 (100 classes), both consisting of $32 \times 32$ color images.
   • Models: A range of architectures was tested, including a Multi-Layer Perceptron (MLP), AlexNet, and a series of ResNet models (ResNet-18, 34, 50, 101, 152).
   • Controlled Variables:
     • Data Scale: Training sets were subsampled to sizes of 5k, 10k, 20k, 30k, and 50k using a fixed random seed to ensure nested subsets.
     • Input Modalities: Experiments compared baseline RGB inputs against:
       • Grayscale inputs (color information removed).
       • Enhanced inputs where explicit priors (gradients, edge maps, wavelet decompositions) were concatenated with the original RGB channels.
   • Constraints: To isolate the effects of the target variables, explicit regularization techniques (data augmentation, weight decay, Dropout) were excluded. Models were trained using Adam optimizer and cross-entropy loss.
   • Metrics: Primary metric was Test Top-1 Accuracy; Test Loss was used as an auxiliary metric to analyze overfitting and prediction confidence.

Key Results

• Data Scale vs. Model Complexity:

  • Increasing the training data scale consistently improved generalization performance across all model architectures and datasets.
  • Increasing model complexity (parameter count) did not provide stable gains. In many cases, particularly with smaller data scales or more difficult tasks (CIFAR-100), larger models (e.g., ResNet-152) performed worse than smaller counterparts (e.g., ResNet-18).
  • Even simple MLPs achieved near-perfect training accuracy, demonstrating high fitting capacity, but their test performance was significantly inferior to convolutional networks, highlighting that fitting ability does not equate to generalization.
  • In the CIFAR-100 experiments, larger models showed a tendency to overfit (increasing test loss) when data was scarce, whereas ResNets with residual connections generally maintained better stability than MLPs or AlexNet.

• Input Modalities:

  • Color Information: Removing color information (converting to grayscale) consistently degraded performance across all models, confirming that color is a critical feature for CIFAR classification.
  • Explicit Priors: The introduction of explicit prior features (gradients, edges, wavelets) yielded inconsistent results:
    • MLP: Showed clear improvements with prior features, suggesting that fully connected networks benefit from auxiliary structural information they cannot learn from raw pixels alone.
    • Convolutional Networks (ResNet): Showed no stable gains; in several instances, performance degraded. This suggests that CNNs, due to their inductive biases (local connectivity, weight sharing), can already extract these features from raw data, and simple channel concatenation may disrupt or fail to integrate the new information effectively.

Key Contributions

1. Empirical Isolation of Factors: The study provides a controlled empirical analysis separating the effects of data scale, model complexity, and input modalities without the confounding variables of standard regularization techniques.
2. Validation of Data Scale Dominance: It reinforces the finding that, in the absence of regularization, increasing data scale is a more reliable method for improving generalization than increasing model capacity.
3. Architecture-Dependent Input Sensitivity: The research demonstrates that the utility of input modalities is highly dependent on model architecture. While explicit priors aid models lacking inductive bias (MLPs), they do not necessarily benefit models with strong built-in biases (CNNs) when introduced via simple concatenation.
4. Observation of Non-Monotonicity: The preliminary experiment confirmed that the relationship between model complexity and test error is non-monotonic (resembling double descent), and that high-complexity models require denser data constraints to avoid oscillatory behavior in sample gaps.

Significance and Claims
The paper claims that improving generalization performance is not merely a function of scaling up data, parameters, or input information in isolation. Instead, it emphasizes the alignment between model architecture and data characteristics.

• Modest Claims: The authors explicitly state that their preliminary experiment is a phenomenological reference rather than direct proof of double descent mechanisms in deep networks.
• Limitations: The study acknowledges that conclusions are drawn from small-resolution datasets (CIFAR) and a single random seed, without the use of standard training strategies like data augmentation.
• Core Insight: The primary significance lies in the observation that "over-parameterization" does not automatically guarantee better generalization; rather, the effectiveness of a model depends on whether its inductive bias matches the structure of the data and the nature of the input information. For convolutional networks, simply adding more input channels (priors) is insufficient; the architecture must be designed to effectively utilize that information.

The study concludes that future work should focus on designing model architectures that better align with data structures and exploring more sophisticated methods for integrating prior knowledge (e.g., attention mechanisms) rather than simple concatenation.