Style-Based Neural Architectures for Real-Time Weather Classification

1. Problem Statement

The paper addresses the challenge of real-time weather condition classification (specifically distinguishing between sunny, rain, snow, and fog) from single images.

• Context: Accurate weather detection is critical for automated agriculture, renewable energy optimization, autonomous vehicles, and climate modeling.
• Hypothesis: The authors posit that weather conditions primarily alter the visual style (appearance, texture, brightness, and color) of objects within an image, rather than changing the semantic content (the objects themselves). Therefore, architectures designed for neural style transfer may be more effective for this task than traditional deep learning models that prioritize high-level semantic content.
• Limitation of Existing Methods: Standard deep networks (like full ResNet50) often ignore appearance variations to achieve robust object recognition, potentially discarding the subtle stylistic cues (e.g., wetness, foggy outlines) necessary for weather detection. Furthermore, existing style transfer methods using Gram matrices (Gatys et al.) often lose spatial coherence.

2. Methodology

The authors propose three distinct neural network architectures inspired by style transfer techniques (Gatys, Isola, and PatchGANs) to capture stylistic features effectively.

A. Model 1: Truncated ResNet50

• Concept: A simplified version of the ResNet50 architecture, retaining only the first 9 layers.
• Rationale: Based on the observation that deeper layers in classification networks focus on semantic content, while shallower layers capture low-level features (edges, textures, colors) essential for style.
• Optimization: The truncation point was determined via an evolutionary algorithm. The model uses pre-trained MoCoV3 weights and is retrained specifically for weather classification.
• Goal: To extract high-frequency features essential for capturing subtle stylistic details without the computational overhead of deep layers.

B. Model 2: Truncated ResNet50 + Gram Matrix + Attention

• Concept: Builds upon Model 1 but incorporates Gram Matrices (a core component of Neural Style Transfer) for every layer of the truncated network.
• Mechanism:
  • Computes the Gram matrix for feature maps at each layer to capture correlations between filters (representing texture and style).
  • Introduces an Attention Mechanism to automatically learn and weight the importance of these Gram matrices across different layers.
• Rationale: This allows the model to dynamically select the most relevant stylistic expressions (fine details from shallow layers vs. global style from deeper layers) for classification, optimizing the extraction of weather-specific features.

C. Model 3: Multi-Patch Simple PatchGAN

• Concept: Adapts the PatchGAN architecture (used in Pix2Pix and CycleGAN) for detection rather than generation.
• Mechanism: Instead of a single PatchGAN, this model employs multiple PatchGANs with different patch sizes (small, medium, large).
• Rationale: Different patch sizes allow the network to analyze stylistic features at various scales. The final decision is an aggregation of these multi-scale evaluations, improving robustness against varying image resolutions and weather intensities.

3. Key Contributions

1. Novel Architectures: Introduction of three style-based architectures specifically adapted for weather classification, moving away from standard semantic-heavy CNNs.
2. Truncation Strategy: Demonstration that truncating deep networks (ResNet50 to 9 layers) significantly improves performance for appearance-based tasks by focusing on high-frequency features.
3. Attention-Weighted Gram Matrices: The first application of a learned attention mechanism to weight Gram matrices from different layers for classification, optimizing stylistic feature selection.
4. Real-Time Capability: All models are optimized for real-time inference (>20 images/second on GPU), making them suitable for autonomous vehicles and surveillance.
5. Cross-Domain Generalization: Demonstration that these style-based models generalize well to other appearance-based tasks, such as melanoma detection in medical imaging.

4. Experimental Results

The models were trained on a custom Cerema dataset (132,000 images: 33k per class) and tested on multiple public datasets (MWI, Image2Weather, Kaggle) to evaluate generalization.

• Performance on Training/Test Set (Cerema):

  • Truncated ResNet50: Achieved an F1-score of 98.84% (Precision/Recall: 98.84%).
  • Truncated ResNet50 + Gram + Attention: Achieved an F1-score of 98.29%.
  • Multi-Patch PatchGAN: Achieved an F1-score of 93.88%.
  • Comparison: These results outperform state-of-the-art methods like ResNet15 (96.03%) and Vision Transformers with attention (92.73%).

• Generalization Capability:

  • The models maintained strong performance on unseen datasets (e.g., 79.06% F1 on Kaggle850, 94.29% on Kaggle2000), demonstrating robustness to domain shifts.

• Interpretability:

  • Grad-CAM visualizations confirmed that the models focus on weather-relevant areas (e.g., wet roads, foggy skies) rather than irrelevant background objects, though some errors were noted (e.g., confusing white gravel with snow).

• Secondary Application (Melanoma Detection):

  • When applied to the SIIM-ISIC Melanoma dataset, the Truncated ResNet50 achieved an AUC of 0.9758, surpassing the winning EfficientNet model (AUC 0.9600) from the 2020 challenge, validating the "style-based" hypothesis for medical imaging.

5. Significance and Limitations

• Significance:

  • The paper validates the hypothesis that weather is a stylistic attribute, proving that style-transfer-inspired architectures are superior to traditional detection models for this specific task.
  • It offers a computationally efficient, real-time solution for critical infrastructure (autonomous driving, agriculture).
  • It establishes a new paradigm for "appearance-based classification" applicable to medical and industrial defect detection.

• Limitations:

  • Multi-Weather Scenarios: The models struggle when an image contains simultaneous weather conditions (e.g., rain and fog), often forcing a categorical choice.
  • False Positives: The models can be misled by non-weather visual elements that share similar textures (e.g., white gravel misclassified as snow).
  • Future Work: The authors propose developing multi-task models to handle mixed weather conditions and quantify intensity/visibility, moving beyond simple classification.

Conclusion

This paper successfully bridges the gap between neural style transfer and weather detection. By truncating deep networks and leveraging attention mechanisms on Gram matrices, the authors created models that are not only state-of-the-art in accuracy and generalization but also highly efficient for real-time deployment. The successful transfer of these models to melanoma detection further underscores the versatility of style-based feature extraction in computer vision.