The effects of image augmentations when training machine learning models in astronomy

1. Problem Statement

Machine learning (ML) is increasingly vital in astronomy for processing massive datasets from surveys like Euclid, LSST, and DESI. However, training deep neural networks (DNNs) is computationally expensive and time-consuming.

• The Assumption: Astronomers often assume that adding image augmentations (random transformations like rotation, flipping, zooming) to training data always improves model performance by increasing data diversity and preventing overfitting.
• The Gap: It is unclear whether these augmentations provide diminishing returns as dataset sizes grow, or if they unnecessarily inflate training times without significant accuracy gains.
• The Goal: To quantify the impact of specific image augmentations and training dataset sizes on the performance of a fixed-size deep learning model (Zoobot) used for galaxy morphology classification.

2. Methodology

Data Source

• Dataset: 230,000 galaxy images from the Galaxy Zoo DECaLS (GZD-5) survey, derived from the Dark Energy Camera Legacy Survey (DECaLS).
• Preprocessing: Images (originally 424×424 pixels) were converted to grayscale, rescaled based on Petrosian radii, and cropped to 224×224 pixels.
• Labels: Morphological classifications derived from volunteer votes (citizen science). Two test sets were used:
  1. All Test Galaxies: Galaxies with ≥34 votes.
  2. Confident Galaxies: A subset where volunteers agreed on the answer ≥80% of the time (high-confidence labels).

Model Architecture

• Model: Zoobot, a convolutional neural network based on EfficientNet B0 (5.3 million parameters).
• Task: Binary classification for 10 distinct morphological questions (e.g., "Is it smooth or featured?", "Does it have spiral arms?").
• Training Split: 80% training, 20% testing (randomly split but consistent across all experiments).

Experimental Design

The authors created 24 distinct model configurations by varying two factors:

1. Dataset Size: Training on 10%, 25%, 50%, and 100% of the available ~180,000 training images.
2. Augmentation Strategy: Six different transformation pipelines:
   • Original: Full pipeline (Grayscale → Random Rotation → Random Crop → Zoom/Stretch → Horizontal Flip).
   • Rotation + Flip: Rotation + Flip (No Zoom/Stretch).
   • Rotation: Rotation only (No Flip/Zoom).
   • Flips: Vertical and Horizontal flips (No Rotation/Zoom).
   • Zoom: Zoom + Stretch + Crop (No Rotation/Flip).
   • None: Only Cropping (No random transformations).

• Training Protocol: Early stopping with a patience of 8 epochs (training stops if validation loss does not improve for 8 epochs). Models were trained on a Tesla V100S GPU.

3. Key Results

A. Impact of Dataset Size vs. Augmentation

• Small Datasets (10-25%): Image augmentations significantly improve performance. The "None" configuration (no augmentations) performed universally worst, often lagging by >10% in accuracy compared to augmented models.
• Large Datasets (50-100%): The performance gap between augmented and non-augmented models diminishes drastically.
  • At 100% dataset size, the "None" configuration often performed nearly as well as the "Original" configuration.
  • For many questions, the accuracy curves of different augmentation strategies converged, suggesting the model reached a saturation point where the raw data diversity was sufficient to fill the model's capacity.

B. Model Capacity and Saturation

• The study identified a saturation point for the Zoobot architecture. Once the model is trained on a sufficiently large and diverse dataset, adding more data or complex augmentations yields negligible performance gains.
• Different morphological questions reached saturation at different rates. Complex tasks (e.g., "Spiral Arm Count") showed more variance and lower absolute accuracy than simpler tasks (e.g., "Smooth or Featured").

C. Specificity of Augmentations

• No "Best" Augmentation: No single augmentation strategy (e.g., Rotation vs. Flips) consistently outperformed the others across all 10 questions when trained on large datasets.
• Simplicity vs. Complexity: Simpler augmentations (like flipping) often yielded similar accuracy to complex ones (like rotation + zoom) but required fewer training epochs.

D. Training Time and Efficiency

• Time Cost: Augmentations significantly increased training time.
  • The "None" configuration was the fastest (e.g., ~13 mins for 10% data, ~2.5 hours for 100% data).
  • The "Original" configuration was the slowest (e.g., ~56 mins for 10% data, ~7.5 hours for 100% data).
• Epochs Required: Models with complex augmentations (rotation, zoom) required significantly more epochs to converge because the model had to "see" a larger variety of transformed images.
• Efficiency Trade-off: For large datasets, the marginal accuracy gain from complex augmentations did not justify the substantial increase in computational time (often doubling or tripling training duration).

4. Key Contributions

1. Quantification of Diminishing Returns: The paper provides empirical evidence that the benefit of image augmentation is inversely proportional to dataset size. For large datasets, augmentations offer little to no performance boost.
2. Identification of Saturation Points: It demonstrates that for a fixed model architecture (Zoobot/EfficientNet B0), there is a data limit beyond which performance cannot be improved by augmentation alone.
3. Resource Optimization Guidelines: The authors propose that for large-scale astronomical surveys, astronomers should consider simpler augmentations (or even minimal ones) to save computational resources, as complex transformations rarely yield superior results on large datasets.
4. Validation of "Confident" Subsets: The study confirms that models perform significantly better on "confident" galaxies (high inter-voter agreement), suggesting that training on ambiguous data limits model ceiling.

5. Significance and Conclusion

This work challenges the "more is better" heuristic in astronomical machine learning.

• For Large Surveys (e.g., LSST, Euclid): As datasets grow to billions of objects, the marginal utility of complex image augmentations decreases. Researchers can reduce training time and carbon footprint by using simpler augmentation pipelines without sacrificing model accuracy.
• For Small Datasets: Augmentations remain critical for preventing overfitting and improving generalization.
• Future Work: The authors suggest that while this model (Zoobot) reached saturation, more complex architectures might require larger datasets before hitting their own saturation points. However, the principle of diminishing returns for augmentations likely holds true across architectures.

Final Takeaway: Astronomers should carefully evaluate their dataset size and model capacity before applying computationally expensive augmentations. In many cases, simpler transformations or raw data are sufficient, allowing for significant savings in time and energy.