What do CNNs Learn in the First Layer and Why? A Linear Systems Perspective

1. Problem Statement

Deep Convolutional Neural Networks (CNNs) have demonstrated remarkable success in image recognition, often attributed to their ability to learn good representations. A well-documented empirical observation is that the first layer of CNNs, regardless of architecture, initialization, or training dataset, converges to a highly consistent set of filters. These filters typically exhibit sensitivity to intermediate spatial frequencies while being insensitive to very low and very high frequencies.

The central question addressed by the authors is: Why does this consistency occur?

• Hypothesis A (Semantic Optimization): The consistency arises because these specific filters are optimal for the semantic task of object recognition (e.g., ignoring noise at high frequencies and illumination at low frequencies).
• Hypothesis B (Algorithmic/Statistical Bias): The consistency is an emergent property of the training algorithm (gradient descent) and the statistical properties of natural images, independent of the semantic label information.

The authors aim to distinguish between these hypotheses and provide a theoretical explanation for the learned energy profiles.

2. Methodology

A. Quantifying Consistency: The Energy Profile

To rigorously measure the similarity between different CNN first layers, the authors propose a metric based on Linear Systems Theory.

• Filter Bank Perspective: They treat the first layer as a bank of filters.
• PCA Basis: They define a set of Principal Component Analysis (PCA) vectors $\{u_i\}$ derived from the training image patches. These vectors represent the dominant spatial frequencies.
• Energy Profile Definition: For a set of filters $\{w_k\}$, the energy profile $e$ is a vector where the $i$-th component represents the average energy of the filters projected onto the $i$-th PCA vector:
  $$e_i^2 = \frac{1}{K} \sum_{k=1}^K (w_k^T u_i)^2$$
• Similarity Metric: They measure the correlation coefficient between the energy profiles of different trained models. This metric is invariant to filter permutation, rescaling, and orthogonal transformations.

B. Empirical Experiments

The authors conducted extensive experiments to test the "Semantic Optimization" hypothesis:

1. Frozen First Layer: They trained modern CNNs (VGG variants) with the first layer weights frozen to random initialization. They observed that performance (loss and accuracy) remained comparable to fully trained networks, suggesting the first layer does not need to be optimized for semantics to achieve good results.
2. Random Labels: They trained CNNs with randomly assigned labels. Despite the lack of semantic structure, the networks learned energy profiles nearly identical to those trained with true labels.
3. Cross-Architecture/Dataset: They compared models (VGG, ResNet, etc.) across different datasets (CIFAR-10, CIFAR-100, CelebA, ImageNet) and initializations, finding correlation coefficients $>0.9$ for trained models and $\approx 0$ for random initializations.

C. Theoretical Analysis: Linear CNNs

To explain the empirical findings, the authors derived an analytical formula for a simplified depth-2 linear CNN trained with Mean Squared Error (MSE) loss.

• Assumptions:
  • Input patches are uncorrelated with each other.
  • Labels are uncorrelated with individual PCA components of the input patches (a condition valid for random labels and approximately true for natural images).
• Derivation: By analyzing the gradient descent dynamics in the PCA space, they derived the energy profile $e_i$ at iteration $t$:
  $$e_i \propto \left| \frac{1 - (1 - \eta \lambda_i^2)^t}{\eta^2 \lambda_i^2} \right| \lambda_i$$
  Where $\lambda_i$ is the energy profile of the training patches (eigenvalues of the covariance matrix) and $\eta$ is the learning rate.
• Limit Behavior: As the number of iterations $t \to \infty$, the term $(1 - \eta \lambda_i^2)^t \to 0$, leading to:
  $$e_i \propto \frac{1}{\lambda_i}$$
  This inverse relationship ($1/\lambda_i$) corresponds to a whitening transformation, which removes correlations (redundancy) from the input.

3. Key Contributions

1. Quantitative Consistency Metric: Introduced the "Energy Profile" based on PCA projections, providing a robust, invariant method to compare linear representations across different CNNs.
2. Refutation of Semantic Necessity: Demonstrated that the consistent first-layer representation is not driven by the need to solve the classification task. This is proven by the fact that:
   • Networks with frozen random first layers perform well.
   • Networks trained on random labels learn the same profiles as those trained on true labels.
3. Analytical Formula for Energy Profiles: Derived a closed-form solution for the energy profile of linear CNNs. The formula depends only on the second-order statistics of the input patches ($\lambda_i$), the learning rate, and the number of iterations. It is independent of the specific labels used.
4. Whitening as an Emergent Property: Showed that gradient descent on classification tasks implicitly drives the first layer toward partial whitening (spatial decorrelation) of the input. This occurs as a byproduct of minimizing the loss, not as an explicit objective.
5. Generalization to Nonlinear CNNs: Validated that the analytical formula derived for linear CNNs fits the energy profiles of complex, nonlinear architectures (ResNet, VGG, DenseNet) trained on real-world datasets with high correlation ($>0.9$).

4. Results

• High Correlation: Trained models across different seeds, architectures, and datasets showed energy profile correlations often exceeding 0.98. In contrast, random initializations showed correlations near 0 with trained models.
• Random vs. True Labels: Models trained with random labels achieved energy profile correlations of 0.90–0.96 with models trained on true labels, confirming that the learning bias is driven by input statistics and the optimizer, not the labels.
• Frozen Layer Performance: Freezing the first layer to random weights resulted in negligible performance degradation on deep networks (e.g., VGG-19), proving that the specific learned filters are not strictly necessary for the task.
• Theoretical Fit: The analytical formula (Eq. 3) fitted the empirical energy profiles of real-world CNNs with correlations consistently above 0.9, even on complex datasets like ImageNet.
• Manipulation Experiments:
  • When the authors artificially correlated labels with specific PCA components, the consistency between true and random label models broke down, confirming the theory's reliance on the uncorrelated assumption.
  • When input statistics were altered (scaling specific PCA components), the learned profiles shifted exactly as predicted by the formula.

5. Significance

This paper fundamentally shifts the understanding of representation learning in deep CNNs:

• Implicit Bias: It highlights that the "inductive bias" of gradient descent, combined with the second-order statistics of natural images, is sufficient to produce a "good" first-layer representation (whitening) without explicit redundancy reduction objectives.
• Whitening without Preprocessing: It explains why CNNs often perform well even without explicit whitening preprocessing steps; the network learns to perform approximate whitening implicitly during training.
• Transfer Learning: The findings suggest that the first layers of models trained on random labels are highly similar to those trained on true labels. This provides a theoretical basis for why random-label models can still be useful for transfer learning or feature extraction.
• Simplification of Complexity: By using a linear model to explain the behavior of complex nonlinear deep networks, the paper demonstrates that the core mechanism of early representation learning is governed by simple statistical principles rather than complex semantic optimization.

In summary, the authors conclude that the first layer of CNNs learns to whiten the input (remove spatial redundancy) because gradient descent naturally drives the system toward this solution given the statistics of natural images, regardless of the semantic content of the labels.