💻 computer science

Contrastive Heliophysical Image Pretraining for Solar Dynamics Observatory Records

The paper introduces SolarCHIP, a family of contrastively pretrained visual backbones tailored to multi-instrument Solar Dynamics Observatory data that addresses unique solar imaging challenges through a multi-granularity objective, achieving state-of-the-art performance in cross-modal translation and flare classification while significantly improving label efficiency.

Original authors: Shiyu Shen, Zhe Gao, Taifeng Chai, Yang Huang, Bin Pan

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shiyu Shen, Zhe Gao, Taifeng Chai, Yang Huang, Bin Pan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching AI to "See" the Sun

Imagine the Solar Dynamics Observatory (SDO) as a high-tech security camera system constantly filming the Sun. It doesn't just take one photo; it takes thousands of pictures every day using different "lenses" (instruments). Some lenses see heat (like a thermal camera), while others see magnetic fields (like a compass map).

For years, scientists had to teach a new computer program (AI) from scratch for every single job they wanted it to do—like predicting solar flares or translating images from one lens to another. This was like hiring a new student for every class and making them read a textbook from page one every time, even though they were all studying the same subject.

SolarCHIP is a new "pre-trained" AI brain. Instead of starting from zero, the researchers taught this AI to understand the Sun's unique language first. Now, it can be used as a smart assistant for many different solar tasks, saving time and needing fewer examples to learn.

The Three Big Problems SolarCHIP Solves

The authors say that teaching AI about the Sun is harder than teaching it about cats or dogs because of three specific challenges:

The "Multi-Lens" Puzzle: The Sun looks very different depending on which instrument you use. A magnetic map (HMI) looks nothing like a heat map (AIA), even though they are looking at the exact same spot at the exact same time.
- Analogy: Imagine looking at a person through a night-vision camera and a thermal camera. They look totally different, but it's the same person. SolarCHIP learns that these different views are actually the same "solar scene."
The "Slow Motion" Problem: The Sun changes very slowly. If you take a picture of the Sun now and one an hour later, they look almost identical, just shifted slightly because the Sun is spinning. It's hard for AI to tell them apart.
- Analogy: It's like trying to spot the difference between two photos of a parked car taken 10 minutes apart. The AI needs to learn to ignore the tiny shifts and focus on what actually matters.
The "Needle in a Haystack" Problem: Most of the Sun is quiet and boring. The exciting stuff (like solar flares) happens in tiny, specific spots.
- Analogy: If you are looking for a specific red dot on a giant, mostly gray wall, a standard AI might get confused by the gray wall. SolarCHIP is trained to zoom in on those tiny, important red dots without getting distracted by the background.

How SolarCHIP Learns (The "Three-Step" Training)

To teach the AI, the researchers used a special training method called Contrastive Learning. Think of this as a game of "Match the Pair" played at three different levels of detail:

The "Global" Match (The Big Picture):
The AI is shown a pair of images taken at the exact same time: one from the magnetic lens (HMI) and one from the heat lens (AIA). It is told, "These two belong together." It learns to recognize that even though they look different, they represent the same moment in time.
- Goal: To understand the overall "vibe" of the Sun at a specific moment.
The "Patch" Match (The Local Neighborhood):
The AI breaks the image into small squares (patches). It looks at a specific square in the magnetic image and the exact same square in the heat image. It learns that the magnetic field in that specific spot corresponds to the heat in that same spot.
- Goal: To ensure the AI understands that "North" in one image matches "North" in the other, preserving the map's layout.
The "Intra-Sample" Match (The Detail Detective):
This is the tricky part. The AI looks at one image and compares different squares within that same image. It learns that the quiet, boring background squares are different from the active, exciting flare squares.
- Goal: To prevent the AI from ignoring the quiet parts. It forces the AI to learn the unique "fingerprint" of every single spot on the Sun, not just the bright spots.

What Can SolarCHIP Do Now?

The researchers tested this new AI brain on two specific tasks:

Translating Solar Languages (Cross-Modal Translation):
They used SolarCHIP to translate images. For example, they fed it a magnetic map (HMI) and asked it to "draw" what the heat map (AIA) would look like, and vice versa.
- Result: It did a great job. It could predict the heat patterns based on the magnetic map, even filling in details that were missing. It's like being able to look at a blueprint of a house and accurately guess what the interior paint colors would look like.
Predicting Solar Flares (Classification):
They used SolarCHIP to look at a full image of the Sun and guess if a solar flare (a massive explosion) was happening, and how big it was (ranging from tiny "A" class to massive "X" class).
- Result: It was very accurate, even when they gave it very few examples to learn from (a "low-resource" setting). This is crucial because big solar flares are rare, and scientists often don't have enough data to train normal AI models.

The Bottom Line

SolarCHIP is a reusable, pre-trained tool for solar scientists. Instead of building a new AI from scratch for every new solar project, scientists can now use SolarCHIP as a foundation. It understands the Sun's unique mix of magnetic fields and heat, handles the slow changes over time, and spots the tiny details in a massive image. The authors have made the code and the "brain" available for everyone to use, hoping to speed up research into space weather and solar safety.

1. Problem Statement

Deep learning has transformed solar image analysis, but current approaches face significant limitations when applied to data from the Solar Dynamics Observatory (SDO):

Lack of Domain-Specific Backbones: Most models train task-specific encoders from scratch or rely on pretraining with natural images (e.g., ImageNet), which fails to capture the unique physical characteristics of solar data.
Unique Data Challenges: SDO data presents three specific challenges that natural image pretraining does not address:
1. Multimodal Sensing: Data comes from multiple instruments (AIA and HMI) observing different physical phenomena (temperature/height vs. magnetic fields) that are not pixel-aligned in appearance but are physically correlated.
2. Weak Inter-class Separability: Solar evolution is slow; consecutive images often look like rigid translations of the disk, making it hard to distinguish between different time steps or event classes based on global morphology alone.
3. Strong Intra-class Variability & Sparsity: Relevant activity (flares, eruptions) is sparse and localized, while the majority of the image is low-contrast background. Standard models often fail to extract fine-grained features from these sparse signals.

2. Methodology: SolarCHIP

The authors propose SolarCHIP, a family of contrastively pretrained visual backbones designed specifically for SDO data. The framework employs a multi-granularity contrastive pretraining strategy that aligns features across modalities (AIA and HMI) and scales (global and local).

A. Data Preprocessing

Input: SDO images (HMI and 10 AIA passbands) are downsampled to $1024 \times 1024$ .
Normalization: A signed log-transform is applied to handle the heavy-tailed distribution of solar activity (rare high-magnitude events vs. quiet background).
Augmentation: Geometric transformations (rotation, flip) are applied synchronously across all modalities to preserve co-registration, avoiding photometric distortions like color jitter.

B. Architecture

The framework uses an Autoencoder architecture (supporting both CNN and Vision Transformer backbones) consisting of:

Encoders: Modality-specific encoders extract token sequences (Class token + Patch tokens).
Decoders: Reconstruct the input images to preserve fine-grained spatial and photometric details.
Projections: Separate linear heads project Class tokens and Patch tokens for contrastive learning.

C. Multi-Granularity Contrastive Objectives

The core innovation is a joint loss function combining three contrastive components:

Global Class-Level Alignment (Time-Indexed):
- Aligns the global [CLS] tokens of co-temporal AIA-HMI pairs.
- Goal: Enforce temporal discrimination. Images from the same time are pulled together; images from different times are pushed apart.
Local Patch-Level Alignment (Position-Aware):
- Aligns patch tokens at the same fixed spatial index across co-temporal AIA-HMI pairs.
- Goal: Enforce modality-invariant features at specific locations (e.g., a magnetic field patch in HMI corresponds to a specific EUV patch in AIA).
Intra-Sample Spatial Discrimination:
- Treats patches at different spatial indices within the same image as distinct classes.
- Goal: Prevent feature collapse in inactive regions. By using other patches in the same image as "hard negatives," the model is forced to learn discriminative features for both active and quiet regions, not just rely on global scene cues.

Total Loss: $L = L_{rec} + \lambda_1 L_{cls} + \lambda_2 L_{pat} + \lambda_3 L_{int}$

3. Key Contributions

SolarCHIP Framework: The first publicly released family of pretrained visual backbones tailored specifically for SDO multi-instrument data, including weights and training code.
Novel Pretraining Objective: A unified, multi-granularity contrastive framework that jointly aligns global (time-indexed) and local (position-aware) tokens across modalities, addressing the sparsity and weak separability of solar data.
Comprehensive Evaluation: Demonstrated state-of-the-art (SOTA) performance on two distinct downstream tasks:
- Cross-modal Translation: Generating AIA images from HMI (and vice versa) using ControlNet.
- Flare Classification: Full-disk solar flare recognition (A, B, C, M, X classes).

4. Experimental Results

Cross-Modal Translation: SolarCHIP achieved SOTA performance in translating between HMI and AIA passbands. Visualizations show high fidelity in preserving physical structures (e.g., coronal jets) and magnetic field directions, outperforming standard baselines and SOTA translation methods (measured by MSE, PSNR, SSIM).
Flare Classification: The model significantly outperformed task-specific baselines and ImageNet-initialized models in classifying full-disk flare levels.
Few-Shot Learning (Low-Resource): In scenarios with limited labeled data (down to 5% of the training set), SolarCHIP-pretrained models maintained robust performance, whereas models trained from scratch suffered severe degradation. This highlights the efficiency of the pretraining strategy in reducing label requirements.
Ablation Studies: Confirmed that all three contrastive components (Class, Patch, Intra-sample) are essential. Removing any single component led to a measurable drop in downstream performance, proving that global alignment, local consistency, and intra-sample discrimination are all necessary for optimal representation.

5. Significance and Impact

Standardization: SolarCHIP provides a "plug-and-play" feature extractor for the heliophysics community, standardizing representations and reducing the need to train models from scratch for every new task.
Efficiency: It drastically lowers computational and data requirements, enabling accurate AI systems even when labeled solar data is scarce.
Foundation for Future Work: By releasing pretrained weights, the authors enable the development of more complex foundation models for solar science, including long-term activity synthesis and improved space weather forecasting for satellite and power grid protection.
Scientific Insight: The approach successfully bridges the gap between distinct physical modalities (magnetic fields vs. thermal emissions), proving that contrastive learning can capture the underlying physical topology of solar phenomena better than generic natural image pretraining.