🤖 AI

Step-adaptive multimodal fusion network with multi-scale cloud feature learning for ultra-short-term solar irradiance forecasting

This paper proposes a step-adaptive multimodal fusion network that integrates multi-scale cloud feature extraction via InceptionNeXt, dynamic low-frequency compensation, and TempAttnLSTM temporal modeling to significantly improve ultra-short-term solar irradiance forecasting accuracy over existing methods.

Original authors: Jingxin Zhang Xiaoqin Wang

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jingxin Zhang Xiaoqin Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict exactly how much sunlight will hit a solar panel in the next few hours. This is tricky because clouds are like mischievous dancers: they move fast, change shape, and can suddenly block the sun, causing the power output to spike or crash.

The paper introduces a new "super-forecaster" called IST (Intelligent Spatio-Temporal) that solves this problem by acting like a team of three specialized experts working together, rather than just one person guessing.

Here is how the IST system works, broken down into simple concepts:

1. The Problem: Why Old Methods Fail

Previous attempts to predict solar power had three main flaws:

The "One-Track Mind" Problem: Some models only looked at numbers (like wind speed or past temperature). They missed the visual reality of clouds moving across the sky.
The "Blurry Lens" Problem: Other models looked at cloud photos but used standard tools that couldn't see both tiny, broken clouds and huge, massive cloud systems at the same time. It's like trying to see both a single ant and a whole mountain in the same photo without zooming in or out.
The "One-Size-Fits-All" Problem: Most models treated a 15-minute prediction the same as a 4-hour prediction. But predicting the immediate future requires focusing on fast-moving details, while predicting further out requires understanding the big, slow-moving trends. Old models used a fixed strategy for both, which didn't work well.

2. The Solution: The IST "Dream Team"

The authors built a system that combines ground-based sky cameras (taking pictures of the clouds) with weather station data (numbers like temperature and wind). It uses three specific tools to process this information:

Expert A: The "Multi-Lens Photographer" (InceptionNeXt)

Instead of using a standard camera lens, this part of the system uses a special "multi-lens" approach.

The Analogy: Imagine looking at a cloud through four different glasses at once:
1. A square glass to see small, detailed textures (like the edges of a fluffy cloud).
2. A long horizontal strip to see clouds stretching out sideways.
3. A long vertical strip to see clouds stretching up and down.
4. A clear glass to keep the original picture intact.
The Result: This allows the computer to understand clouds of all shapes and sizes simultaneously, capturing both the tiny details and the big structures without getting confused.

Expert B: The "Smart Memory Keeper" (SALFCU)

This is the paper's most unique invention. It acts like a librarian who knows exactly how much detail you need based on when you are asking.

The Analogy: Think of predicting the weather like reading a story.
- For the next 15 minutes (Short-term): You need to know the immediate action. "Is that dark spot moving right toward the sun?" This requires High-Frequency details (sharp, fast changes).
- For the next 4 hours (Long-term): You don't need to know if a single leaf is moving; you need to know the overall plot. "Is the whole storm system moving away?" This requires Low-Frequency information (the big, slow trends).
The Magic: Old models tried to use the same amount of detail for both. This new "Smart Memory Keeper" automatically adjusts. It focuses on sharp details for the near future and switches to broad, smooth trends for the distant future. It ensures the model doesn't lose the "big picture" when looking at the "small details."

Expert C: The "Time Traveler" (TempAttnLSTM)

Once the images and weather numbers are combined, this final expert looks at the sequence of events over time.

The Analogy: Imagine watching a movie of the clouds. This expert doesn't just watch frame-by-frame; it looks for patterns. It asks, "Does this cloud movement usually happen at this time of day?" or "How does the wind usually affect the clouds 30 minutes from now?"
The Result: It connects the dots between the past, the present, and the future, allowing it to make a confident guess about what will happen in the next few hours.

3. The Results: How Well Did It Work?

The researchers tested this "Dream Team" in two ways:

On Public Data (NREL): Using a massive, standard dataset of cloud photos and weather records.
On Real Life (Shandong, China): Using actual data from a real solar power station in the field.

The Outcome:
The IST model beat every other method they tested.

It made fewer mistakes (lower error rates) than models that only looked at numbers or only looked at pictures.
It was better than other "hybrid" models that tried to combine both but lacked the special "Smart Memory Keeper" and "Multi-Lens" tools.
In the real-world test, it was particularly good at avoiding "extreme errors" (wildly wrong guesses), keeping its predictions tight and reliable even when the weather was chaotic.

Summary

In short, this paper presents a new way to predict solar power by teaching a computer to see clouds like a photographer with multiple lenses, remember the right amount of detail based on how far into the future it's looking, and understand the story of how the weather changes over time. This makes it much more reliable for keeping the power grid stable when the sun is being unpredictable.

Technical Summary: Step-Adaptive Multimodal Fusion Network for Ultra-Short-Term Solar Irradiance Forecasting

Problem Statement
Ultra-short-term solar irradiance forecasting is critical for photovoltaic (PV) system dispatch and grid stability. However, existing approaches face three primary limitations:

Inadequate Spatial Dynamics: Single time-series models fail to capture the spatial migration and shading effects of cloud clusters under complex weather conditions.
Insufficient Feature Representation: Standard convolutions and single-scale feature extraction modules cannot adaptively represent the multi-scale and multi-directional properties of diverse cloud structures (e.g., small broken clouds vs. large stratus clouds).
Static Compensation Strategies: Conventional low-frequency compensation methods often rely on fixed strategies that fail to adapt to the differing feature requirements of various prediction steps. Short-range forecasting requires high-frequency local details for rapid cloud movements, while long-range forecasting relies on low-frequency global trends for overall evolution patterns.

Methodology: The IST Framework
To address these challenges, the authors propose IST, a multimodal fusion model that integrates ground-based cloud images and meteorological time-series data. The architecture consists of three core components:

InceptionNeXt for Multi-Scale Spatial Extraction:
- Ground-based sky images are processed using an InceptionNeXt network.
- This module employs a parallel multi-branch depthwise convolution design, including:
  - Square convolution ( $k_s \times k_s$ ): Extracts local textures and edge details.
  - Horizontal ( $1 \times k_b$ ) and Vertical ( $k_b \times 1$ ) band convolutions: Capture directional structural information and large-scale cloud systems.
  - Identity mapping: Preserves primitive feature data.
- These branches are concatenated and refined via a Multi-Layer Perceptron (MLP) to generate high-dimensional spatial features that adaptively model anisotropic cloud characteristics.
Step-Adaptive Low-Frequency Compensation Unit (SALFCU):
- This unit introduces a parallel low-frequency memory branch to counteract the loss of global information in deep networks.
- It utilizes a Discrete Wavelet Transform (DWT) to separate features into low-frequency (structural) and high-frequency (detail) components.
- A step-aware dynamic gating mechanism is embedded to generate an adaptive weighting coefficient, $\alpha(H)$ , based on the prediction step $H$ .
- Functionality: For early prediction steps, the model prioritizes high-frequency local details to track rapid cloud changes. For later steps, it dynamically increases the weight of low-frequency global trends to capture overall cloud evolution.
TempAttnLSTM for Temporal Modeling:
- The fused multimodal features (refined cloud images + meteorological time-series) are fed into a Temporal Attention LSTM (TempAttnLSTM).
- This module enhances standard LSTMs with Multi-Head Attention and Learnable Periodic Queries.
- The learnable query matrix ( $\beta_t$ ) follows a periodic pattern, allowing the model to incorporate global periodic priors and long-range dependencies.
- Future time-oriented vectors guide the LSTM to generate targeted multi-step predictions.

Key Contributions

Multimodal Fusion Framework: The paper presents the IST framework, which jointly models cloud-derived spatial features and meteorological time-series information to improve accuracy in capturing non-stationary irradiance variations.
InceptionNeXt Adaptation: The application of InceptionNeXt for cloud image feature extraction allows for the adaptive capture of multi-scale and multi-directional cloud features, overcoming the limitations of single-convolution models.
Step-Adaptive Compensation: The novel SALFCU module dynamically regulates the balance between high-frequency and low-frequency features based on the specific forecasting step, addressing the static nature of previous compensation strategies.

Experimental Results
The model was evaluated on two datasets: the public NREL dataset (2021–2023) and a practical dataset from a PV station in Shandong, China (May–December 2024).

NREL Dataset (0–4 hour horizon):
- The IST model achieved the lowest Mean Absolute Error (MAE) of 91.45 and Root Mean Square Error (RMSE) of 149.28, outperforming nine baseline models (including Tri-TimesNet-TempAttnLSTM, BiLSTM, iTransformer, PatchTST, 3DCNN, and multimodal variants).
- It achieved an $R^2$ of 0.7618.
- Ablation studies confirmed that removing either the image or time-series inputs resulted in the largest performance drops, validating the necessity of multimodal fusion.
Shandong Practical Dataset:
- The model demonstrated superior performance in real-world engineering scenarios, achieving an MAE of 50.86 and RMSE of 68.53.
- Compared to the best baseline, it reduced MAE by 12.7% and RMSE by 13.7%.
- Error density distributions showed that IST produced a sharper, zero-centered peak with narrower tails compared to baselines, indicating fewer extreme prediction errors.

Significance and Claims
The paper claims that the IST framework effectively bridges the gap between spatial cloud dynamics and temporal meteorological trends. By specifically addressing the need for step-adaptive feature balancing, the model overcomes the limitations of fixed fusion strategies in multi-step forecasting. The authors assert that the integration of multi-scale spatial extraction (InceptionNeXt), dynamic frequency compensation (SALFCU), and periodic temporal attention (TempAttnLSTM) provides a robust solution for ultra-short-term solar irradiance forecasting, demonstrating significant improvements in stability and accuracy over state-of-the-art time-series, image-based, and multimodal approaches.