⚡ electrical engineering

A Multimodal Transformer Framework with State-Aware Adaptive Fusion for Ultra-Short-Term Solar Irradiance Forecasting with Sky Cloud Images

This paper proposes a multimodal transformer framework that integrates sky cloud images and numerical sequences through state-aware adaptive fusion to significantly enhance the accuracy and robustness of ultra-short-term solar irradiance forecasting.

Original authors: Houhang Zhao, Aiyi Song, Yu Jin, Haoxiang Yu

Published 2026-08-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Houhang Zhao, Aiyi Song, Yu Jin, Haoxiang Yu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the sun as a giant, unpredictable spotlight on a stage. Sometimes it shines steadily, but often, mischievous clouds drift in front of it, dimming the light in seconds. For the people running our power grids, this is a high-stakes game of catch. They need to know exactly how much sunlight will hit their solar panels in the next few minutes to keep the lights on without wasting energy. This is the world of "ultra-short-term solar forecasting."

To play this game, scientists usually look at two things. First, they check the "history book" of the sun's past behavior, looking at numbers like temperature and wind speed to guess what comes next. It's like trying to predict the weather by only looking at yesterday's calendar. Second, they look at the "live camera feed" of the sky, watching clouds move in real-time. It's like watching the clouds roll by your window. For a long time, experts had to choose one or the other, or just mash them together clumsily. But what if you could have a super-smart referee who knows exactly when to trust the history book and when to trust the live camera? That is the challenge this paper tackles: building a system that doesn't just add information together, but learns to listen to the right voice at the right time.

The Smart Referee: TSGAN

The authors of this paper, a team of researchers from universities in China and Hong Kong, have built a new kind of AI model they call TSGAN. Think of TSGAN as a super-smart referee standing between two teams: the "Number Team" (historical data like past sunlight and wind) and the "Picture Team" (photos of the sky and clouds).

In the old days, if you wanted to combine these two teams, you might just tape their reports together and hope for the best. But the researchers realized that's like asking a referee to listen to both teams shouting at the same time, regardless of who is actually right. Sometimes the Number Team knows the most (like on a calm, sunny day), and sometimes the Picture Team is the only one that matters (like when a giant cloud suddenly blocks the sun).

TSGAN solves this with a clever trick called "State-Aware Adaptive Fusion." Imagine a smart referee who has a special pair of glasses. These glasses let the referee look at the current situation and instantly decide: "Right now, the clouds are moving fast, so I'll trust the Picture Team 90% and the Number Team 10%." But five minutes later, if the sky clears up, the referee flips a switch: "Now the clouds are gone, so I'll trust the Number Team's history more." The model doesn't just mix the data; it dynamically filters and weighs the information based on what the weather is actually doing at that exact second.

How They Tested It

To see if their "smart referee" was any good, the team used a massive dataset from Albuquerque, New Mexico. They grabbed a year's worth of data from 2021, which included:

The Numbers: Past records of sunlight, wind, and temperature.
The Pictures: Sky images taken every 10 minutes.

They set up a challenge where the model had to predict how much sunlight would hit the ground in 10, 20, and 30 minutes. They pitted TSGAN against a bunch of other "players," including old-school math models, standard AI that only looks at numbers, and AI that only looks at pictures.

The Results: A Clear Victory

The results were quite impressive. TSGAN didn't just do okay; it crushed the competition.

Beating the Numbers: When compared to models that only used historical numbers (like the popular Transformer or LSTM models), TSGAN made significantly fewer mistakes. For example, in the first quarter of the year, TSGAN's error rate (measured by something called MAE) was 28.42, while the best single-number model was around 46.23. That's a huge jump in accuracy.
Beating the Pictures: Models that only looked at sky images (like CNNs) also struggled. They could see the clouds, but they didn't know the "rules" of how the sun usually behaves. TSGAN combined the visual "what's happening now" with the numerical "what usually happens," resulting in much more stable predictions.
Beating Other Mixers: Even when compared to other methods that tried to mix numbers and pictures, TSGAN won. The researchers found that simply sticking the data together (a method called "concatenation") wasn't enough. TSGAN's ability to choose which data to trust was the secret sauce.

The paper also ran a statistical test called the Diebold-Mariano (DM) test to make sure these wins weren't just luck. The results showed that TSGAN's better performance was statistically significant in 96.88% of the comparisons at a very high level of confidence. In plain English, the model really is better, not just by a little bit, but in a way that is mathematically proven.

Why It Matters

The paper suggests that this approach is a game-changer for handling tricky weather. When the weather is stable, the model leans on the history. When a storm rolls in or clouds dart across the sun, the model instantly switches to trusting the live camera feed. This prevents the "lag" that happens when a model is too slow to react to sudden changes.

The authors conclude that by teaching the AI to be a flexible referee rather than a rigid blender, they can predict solar power much more accurately. This isn't just about getting a better score on a test; it's about helping power grids run smoother, storing energy more efficiently, and keeping our lights on even when the sky is playing hide-and-seek. While the study was limited to one location and specific timeframes, the method suggests a powerful new way to handle the unpredictable nature of our sun.

Technical Summary: A Multimodal Transformer Framework with State-Aware Adaptive Fusion for Ultra-Short-Term Solar Irradiance Forecasting

Problem Statement
Accurate ultra-short-term solar irradiance forecasting is critical for integrating photovoltaic (PV) power into the grid and ensuring system stability. However, solar irradiance is subject to significant stochastic fluctuations driven by both regular diurnal cycles and rapid, random local weather changes, particularly cloud formation and dissipation. Traditional forecasting methods relying solely on structured numerical time series (e.g., historical irradiance and meteorological data) often fail to capture leading indicators of sudden irradiance drops caused by cloud occlusion, leading to prediction lags and cumulative errors. Conversely, methods relying exclusively on sky-cloud images provide rich spatial and temporal visual information regarding cloud evolution but lack the historical physical constraints and absolute irradiance context necessary to anchor predictions. Existing multimodal approaches often utilize static fusion strategies (e.g., simple concatenation or fixed-weight averaging) that fail to dynamically adapt to varying weather conditions, limiting their ability to exploit the complementary nature of visual and numerical data.

Methodology
The paper proposes TSGAN (Transformer Selective Gated Attention Network), a multimodal framework designed to synergistically fuse sky-cloud images and numerical sequences for ultra-short-term Global Horizontal Irradiance (GHI) forecasting. The architecture consists of three primary components:

Unified State Encoder (USE): This module maps heterogeneous inputs into a shared latent space.
- Numerical Branch: Utilizes a Transformer encoder to process historical sequences of GHI, Direct Normal Irradiance (DNI), Diffuse Horizontal Irradiance (DHI), and meteorological variables (wind, temperature, pressure). It captures deep temporal dependencies and global evolutionary patterns.
- Visual Branch: Employs a hybrid CNN-Transformer architecture. A CNN extracts local spatial features (e.g., cloud edges, solar occlusion) from individual sky frames, while a temporal Transformer models the dynamic evolution of cloud sequences across time steps.
State-Aware Adaptive Fusion Mechanism: This is the core innovation of TSGAN. Instead of fixed-weight fusion, the model employs a Selective Gated Attention Network.
- It constructs a joint state representation by concatenating the numerical and image state vectors.
- A lightweight gating network generates modality-selection weights based on this joint state. These weights are dynamic, varying per sample and per feature dimension.
- The mechanism adaptively filters and fuses features: if the numerical trend is stable, it prioritizes historical patterns; if the sky image indicates rapid cloud movement or occlusion, it enhances the contribution of visual priors. This transforms the fusion process into a "modality-state identification—modality-importance allocation—complementary information fusion" decision process.
Predictive Decoder: The fused state representation is fed into a decoder to output GHI forecasts for future time steps (10, 20, and 30 minutes ahead).

Key Contributions

Collaborative Representation System: The paper introduces a framework that jointly models visual weather information and historical physical patterns, addressing the information gaps inherent in single-modality approaches regarding spatial perception or temporal evolution.
Dual-Branch State Encoding: A specialized encoding structure is constructed for heterogeneous inputs, utilizing Transformers for temporal dependency in numerical data and a CNN-Transformer hybrid for spatiotemporal cloud dynamics in images.
State-Aware Adaptive Fusion: The authors propose a selective gated attention network that shifts multimodal fusion from static feature combination to dynamic information selection. This allows the model to adaptively adjust fusion weights based on sample-state awareness and modality reliability under different weather conditions.

Experimental Results
The model was evaluated using the Open Energy Data Initiative (OEDI) dataset (Albuquerque, NM), utilizing 10-minute resolution data from 2021. Performance was assessed across four quarters and three forecasting horizons (1, 2, and 3 steps) using MAE, SMAPE, NRMSE, and $R^2$ .

vs. Numerical Baselines: TSGAN outperformed traditional machine learning (RF, XGBoost, MLP) and deep learning baselines (GRU, LSTM, Transformer, PatchTST). The proposed framework achieved an average MAE of 47.312, significantly reducing errors compared to single-modality numerical models. The advantage was most pronounced in one-step-ahead forecasts where visual priors correct for sudden fluctuations.
vs. Image-Only Models: TSGAN surpassed single-image models (CNN, ResNet, CNN-LSTM), reducing MAE by 32.38% and NRMSE by 23.32%. The results indicate that while images capture cloud dynamics, they require numerical constraints to accurately determine absolute irradiance levels and reduce distributional bias.
vs. Other Fusion Strategies: Compared to direct concatenation, standard gated fusion, and Cross-Attention mechanisms, TSGAN reduced average MAE by 18.39% and improved $R^2$ by 0.0260. Crucially, TSGAN demonstrated greater stability across seasons and forecasting horizons, with narrower error distributions, suggesting the adaptive mechanism effectively mitigates error accumulation in longer horizons.
Statistical Significance: Diebold-Mariano (DM) tests confirmed that TSGAN's performance improvements are statistically significant (at the 1% level for 96.88% of comparisons) against all benchmark models.

Significance and Claims
The paper claims that TSGAN offers a paradigm shift from "information superposition" to "state selection" in multimodal solar forecasting. Its significance lies in:

Mechanistic Improvement: It addresses the lack of leading information in traditional time-series models by incorporating visual priors for sudden changes, while simultaneously solving the ambiguity of absolute irradiance levels in pure image models through numerical constraints.
Adaptability: The state-aware fusion mechanism allows the model to dynamically identify the reliability of different modalities under seasonal and weather variations (e.g., relying more on history in stable conditions and more on images during rapid cloud movement), thereby reducing information bias.
Practical Utility: The framework provides a more timely and accurate basis for PV power smoothing, energy storage control, and short-term grid dispatch.

The authors note that current findings are limited by the specific dataset scope and forecasting horizons, suggesting future work could explore continuous cloud-motion characterization, integration with satellite/infrared imagery, and probabilistic interval forecasting.

The Smart Referee: TSGAN

How They Tested It

The Results: A Clear Victory

Why It Matters

More like this