Learning Color Equivariant Representations

1. Problem Statement

Convolutional Neural Networks (CNNs) have achieved remarkable success in image classification but often lack robustness to perceptual variations, specifically changes in color (hue, saturation, and luminance).

• Current Limitations: Standard CNNs are sensitive to color shifts caused by different lighting conditions, sensors, or data collection protocols (e.g., medical imaging from different labs).
• Existing Solutions:
  • Data Augmentation: Increases training time and does not guarantee robustness to unseen shifts.
  • Color Invariance: Converting images to grayscale or enforcing invariance discards color information, which is often a critical cue for classification.
  • Previous GCNNs (CEConv): Lengyel et al. (2024) proposed a Group Convolutional Neural Network (GCNN) for hue equivariance. However, their approach lifts the filters rather than the input. In the RGB color space, rotating filters to simulate hue shifts often produces invalid RGB values (values outside the [0, 1] or [0, 255] cube). These invalid values must be projected back, breaking the mathematical property of equivariance and introducing artifacts.

2. Methodology

The authors propose a novel Group Convolutional Neural Network architecture that achieves strict equivariance to hue, saturation, and luminance shifts by leveraging the geometric structure of the HSL (Hue, Saturation, Luminance) color space.

A. Geometric Formulation

The authors model color variations as group actions:

1. Hue Group ($H_N$): Modeled as the discrete cyclic group $C_N$ (2D rotation group). A hue shift corresponds to a rotation in the hue channel.
2. Saturation Group ($S_N$) & Luminance Group ($L_N$): Modeled as discrete subsets of the integer translation group $(\mathbb{Z}, +)$. Shifts in saturation and luminance are treated as translations.
3. Combined Action: The network handles the product space of these groups (e.g., Hue-Saturation group).

B. The Core Innovation: Input Lifting

The most significant technical contribution is the design of the Lifting Layer.

• CEConv Approach (Flawed): Lifts the filters by rotating them in RGB space. This causes invalid RGB values for most discretizations ($N \neq 1, 3$), requiring projection that breaks equivariance.
• Proposed Approach (Correct): Lifts the input image directly.
  • The input image is converted to HSL space.
  • The lifting layer applies the group action (hue rotation or saturation/luminance shift) directly to the pixel values of the input image.
  • Since HSL values are naturally bounded and cyclic (for hue) or linear (for saturation/luminance), this transformation is invertible and produces no invalid values.
  • The network then performs group convolutions on this lifted representation.

C. Network Architecture

• Lifting Layer: Transforms the input $f^0$ to the group space $f^1$ via $f^1(gx, j) = \phi(h_j, f^0)(x)$, where $\phi$ is the group action on the image.
• Group Convolution: Subsequent layers perform convolutions over the product space $Z^2 \times G$ (spatial grid $\times$ color group), ensuring that a shift in the input color space results in a corresponding shift in the feature map space.
• Pooling: Final layers use group pooling to produce invariant representations for classification.

3. Key Contributions

1. Resolution of Invalid RGB Artifacts: By lifting the input image in HSL space rather than rotating RGB filters, the authors eliminate invalid color values. This improves equivariance error by over three orders of magnitude compared to CEConv.
2. Extension to Saturation and Luminance: The paper generalizes color equivariance beyond just hue, introducing networks that are equivariant to saturation and luminance shifts, which previous works ignored.
3. Theoretical Rigor: The authors provide formal proofs that the proposed discretized hue and saturation groups satisfy group axioms and that the defined actions satisfy group action properties.
4. Sample Efficiency: The architecture demonstrates higher sample efficiency, achieving comparable or better performance with significantly less training data compared to conventional CNNs.

4. Experimental Results

The authors evaluated their models (Hue-N, Sat-N, and combined variants) against baselines (Z2CNN, ResNet, and CEConv) on several datasets:

• Hue-Shift MNIST:
  • Generalization: On out-of-distribution (OOD) test sets with unseen hue shifts, the proposed model maintained low error (1.8%), while the standard CNN failed catastrophically (57%).
  • Sample Efficiency: The performance gap between the proposed model and the baseline widened as the amount of training data decreased.
• 3D Shapes (Local Hue Shift):
  • The model achieved near-perfect accuracy (0.00% error) on OOD local hue shifts, significantly outperforming CEConv, which suffered from feature map artifacts.
• Camelyon17 (Saturation Shift in Medical Imaging):
  • In a real-world medical dataset with domain shift due to varying tissue staining (saturation), the Sat-3 model reduced classification error from 28.91% (ResNet50) to 16.08%.
• Real-World Datasets (Caltech-101, Oxford Pets, Stanford Cars, CIFAR):
  • The color-equivariant networks consistently outperformed CEConv and were competitive with or superior to ResNet baselines, particularly on fine-grained classification tasks where color is a discriminative feature.
• Color Sorting:
  • The authors demonstrated a unique application: sorting images by color using the learned feature maps. The equivariant structure allowed for natural hue-based sorting, a task conventional CNNs failed to perform reliably.

5. Significance and Conclusion

This paper bridges the gap between geometric deep learning and perceptual robustness. By treating color variations as geometric transformations within a group structure, the authors create networks that are equivariant by design rather than relying on brittle heuristics or data augmentation.

• Impact: The work solves a critical flaw in previous color-equivariant networks (invalid RGB values), making the approach theoretically sound and practically superior.
• Applications: The method is particularly valuable for domains with high color variability, such as medical imaging (histology slides), remote sensing, and autonomous driving, where lighting and sensor conditions vary but the underlying semantic content remains constant.
• Future Work: The authors acknowledge that GCNNs are computationally more expensive than standard CNNs due to the filter orbit computation and suggest future work on continuous group representations to further optimize efficiency.