Meteorological data and Sky Images meets Neural Models for Photovoltaic Power Forecasting

This paper proposes a hybrid deep learning approach that integrates sky images, historical photovoltaic data, and meteorological variables to enhance the accuracy and robustness of both short-term nowcasting and long-term solar power forecasting, particularly under cloudy conditions.

The Gist

Imagine you are trying to predict exactly how much electricity a solar farm will produce tomorrow. It's a bit like trying to guess how much water will flow through a river, but instead of rain, the "water" is sunlight, and the "river" is the power grid.

The problem is that clouds are tricky. They move fast, change shape, and can suddenly block the sun, causing the power output to spike or crash. This is bad news for the power grid, which needs a steady flow of electricity to keep the lights on.

This paper presents a new, smarter way to predict this power output by acting like a super-charged weather forecaster that uses three different "senses" instead of just one.

The Three Senses of the New System

Think of the old ways of predicting solar power as trying to drive a car while wearing blinders, looking only at a map (historical data) or only at the road ahead (sky images). This new system combines three different tools into one "super-vision":

1. The Camera (The Eyes):
   The system uses a 360-degree fisheye camera pointed at the sky. It takes pictures of the clouds every minute. This is like having a security guard watching the sky, telling you, "Hey, a big dark cloud is moving right toward the sun!"

   • The Paper's Data: They used a massive dataset called SKIPP'D, which has thousands of these sky photos.

2. The Weather Station (The Feel):
   Just looking at a cloud isn't enough; you need to know what the air feels like. The system pulls in real meteorological data (like wind speed, air pressure, and different types of radiation) from a global weather database called ERA5.

   • The Analogy: If the camera sees a cloud, the weather station tells you if the wind is pushing that cloud fast or slow, or if the air is thick with moisture. It's like feeling the humidity on your skin to know if a storm is coming, even before you see the rain.

3. The Sun Calculator (The Compass):
   The system also calculates exactly where the sun should be in the sky at any given second, based on the time of day and location.

   • The Analogy: This is like knowing the sun's schedule. Even if the sky is gray, the system knows, "The sun is supposed to be high and bright right now, so if the power is low, it must be a thick cloud blocking it."

How the "Brain" Works

The researchers built a Neural Network (a type of AI brain) that acts like a master chef.

• The Ingredients: It takes the sky photo, the weather numbers, and the sun's position.
• The Recipe: It mixes them all together. In the past, chefs (AI models) might have tried to cook with just the photo or just the numbers. This paper says, "Let's throw everything in the pot!"

They tested this in two scenarios:

• Nowcasting (The "Right Now" Guess): Looking at the current sky image and weather to guess what happens in the next few minutes. This is crucial for sudden changes, like a cloud suddenly blocking the sun.
• Forecasting (The "Next Hour" Guess): Looking at the last 15 minutes of history to predict the next 15 minutes. This helps the power grid plan ahead.

The Big Discovery: Why Cloudy Days are the Real Test

The most interesting part of the paper is what they found out about cloudy days.

• Sunny Days: Predicting power on a clear day is easy. It's like driving on a straight, empty highway. The old methods worked okay here.
• Cloudy Days: This is the "off-road" driving. Clouds make the power jump up and down wildly (these jumps are called "ramp events"). The old methods struggled here, often getting lost.

The Magic Ingredient: The researchers found that adding Surface Long-Wave Radiation (basically, the heat radiating down from the clouds) and Wind Data made a huge difference.

• The Analogy: Imagine you are trying to guess how fast a car is moving. Looking at the car (the image) helps. But if you also feel the wind in your face (wind data) and feel the heat from the engine (radiation data), you can guess the speed much better, even if the car is hidden in fog.

The Result

By combining the Sky Camera, the Weather Data, and the Sun's Position, the new model became much better at:

1. Predicting the "Ramp Events": It can tell the power grid, "Get ready! The power is about to drop in 2 minutes because a cloud is moving in," allowing them to prepare.
2. Handling Cloudy Weather: It stopped getting confused when the sky was gray and messy.
3. Being Understandable: Because it uses real physics (wind, heat, sun position), we can understand why it made a prediction, rather than it being a "black box" mystery.

The Bottom Line

This paper shows that to predict solar energy accurately, you can't just look at the sky. You have to feel the wind, measure the heat, and know the sun's schedule all at the same time. By teaching an AI to use all these senses together, we can make solar power more reliable, helping us switch to green energy without worrying that the lights will flicker when a cloud passes by.

Technical Summary

1. Problem Statement

The rapid integration of photovoltaic (PV) solar energy into power grids is hindered by the inherent variability of solar irradiance, primarily caused by dynamic cloud cover. This variability leads to:

• Ramp Events: Sudden, sharp fluctuations in power generation that challenge grid stability.
• Forecasting Limitations: Existing models struggle to accurately predict these ramp events and perform reliably under cloudy conditions.
• Data Silos: Current approaches often rely solely on sky images (visual data) or purely numerical/statistical models, failing to leverage the synergistic potential of combining visual, historical, and physical meteorological data.

The authors aim to address these gaps by developing a hybrid multimodal approach that improves both nowcasting (instantaneous prediction) and short-term forecasting (15-minute ahead prediction), specifically targeting robustness in cloudy weather and the accurate prediction of ramp events.

2. Methodology

The proposed framework integrates three distinct data modalities into deep neural networks: Sky Images, Meteorological Data, and Analytical Solar Position.

A. Data Sources

1. Visual Information (SKIPP'D Dataset):
   • Ground-based 360-degree fisheye sky images (2048×2048 pixels, downsampled to 64×64).
   • Captured at 20 Hz but aggregated to 1-minute intervals.
   • Includes 20 selected days (10 sunny, 10 cloudy) with corresponding 1-minute PV power output.
   • Forecast Dataset: A derived subset using a 2-minute sampling interval, utilizing sequences of 16 consecutive images (15-minute history) for prediction.
2. Meteorological Information (ERA5 Reanalysis):
   • Data extracted from four grid points (NW, NE, SW, SE) surrounding the camera location at Stanford University.
   • Key Variables:
     • Instantaneous: Total Cloud Cover (tcc), Surface Pressure (sp), Wind components (100u, 100v), Wind Gust (i10fg).
     • Radiative Fluxes: Surface Solar Radiation Downwards (ssrd), Surface Thermal Radiation Downwards (strd), Net Thermal Radiation (str), Top Net Solar Radiation (tsr), Direct Radiation (fdir).
   • Preprocessing: Data is converted to local time, aligned with image timestamps, and normalized (mean=0, std=1). Accumulative variables are converted to hourly rates.
3. Analytical Solar Position:
   • Instead of relying on image processing to detect the sun (which fails in heavy cloud), the authors calculate the sun's position analytically using pvlib.
   • Parameters: Solar Azimuth ($\alpha$) and Elevation ($\epsilon$).
   • These are transformed into relative coordinates $(x, y)$ on the image plane, accounting for camera orientation and tilt, providing a robust physical prior.

B. Neural Architecture

The study proposes two distinct models with a shared backbone but different input structures:

1. Nowcast Model ($f_N$):
   • Input: Single current sky image ($I_i$) + Meteorological data ($M_i$).
   • Output: Instantaneous PV power ($P_i$).
   • Architecture: Two convolutional blocks (3×3 filters, batch norm, ReLU, max-pooling) followed by fully connected (FC) layers. Meteorological data is concatenated with flattened image features before the final FC layers.
2. Forecast Model ($f_F$):
   • Input: Sequence of 16 historical images + 16 historical PV values + Meteorological data ($M$).
   • Output: PV power 15 minutes ahead ($P_{t+T}$).
   • Architecture: Similar CNN backbone, but explicitly incorporates historical context sequences and concatenates them with meteorological variables in the latent feature space.

Training: Both models use the Adam optimizer with a learning rate of $3 \times 10^{-6}$, minimizing Mean Squared Error (MSE).

3. Key Contributions

• Multimodal Fusion: This is the first study to systematically integrate ERA5 meteorological reanalysis data (specifically surface long-wave radiation and wind) with fisheye sky images and analytical solar position for PV forecasting.
• Analytical Solar Position: The authors propose a robust method to infer sun position analytically rather than via image segmentation, significantly improving reliability during overcast conditions where visual detection fails.
• Hybrid Architecture: The design injects physical meteorological variables directly into the latent feature space of the neural network, allowing the model to learn interactions between visual cloud patterns and physical atmospheric states.
• Dual-Task Framework: Simultaneous development of models for both instantaneous nowcasting and 15-minute forecasting, demonstrating how historical context changes the optimal feature set.

4. Experimental Results

The models were evaluated using Root Mean Squared Error (RMSE) and Mean Absolute Error (MAE) on sunny, cloudy, and overall scenarios.

• Baseline Performance: The proposed baseline (MSUNSET, using only images) outperformed state-of-the-art models (SUNSET [17] and RSUNSET [27]) in cloudy scenarios for the forecast task, though results varied by dataset partitioning.
• Impact of Meteorological Variables:
  • Surface Long-wave Radiation Downwards (strd): Proved to be the most significant single variable for forecasting, particularly improving accuracy on cloudy days.
  • Wind + Solar Position: The combination of wind gusts ($i10fg$), wind components ($wind100$), and analytical sun position yielded the best nowcasting results, outperforming competing methods.
  • Top Net Solar Radiation (tsr): Surprisingly improved cloudy day predictions in nowcasting, likely acting as an indicator of atmospheric attenuation.
• Cloudy vs. Sunny Conditions:
  • The models showed marked improvement in cloudy conditions (where variability is highest). For example, in forecasting, the RMSE for cloudy days dropped significantly when adding strd.
  • In sunny conditions, the models were generally stable, but adding too many complex variables sometimes interfered with pattern learning (e.g., strd was less relevant for sunny nowcasting).
• Ramp Event Prediction: The inclusion of wind and radiation data significantly enhanced the model's ability to predict sudden power fluctuations (ramp events), a critical requirement for grid stability.

5. Significance and Conclusion

This work demonstrates that integrating diverse data modalities is essential for reliable solar energy prediction.

• Grid Stability: By improving the accuracy of ramp event prediction and short-term forecasts, the proposed method supports more efficient grid operation and better management of solar variability.
• Robustness: The hybrid approach overcomes the limitations of image-only models, which often fail in foggy or heavily overcast conditions. The addition of physical meteorological data provides a "safety net" for prediction when visual cues are ambiguous.
• Interpretability: The use of analytical solar position and physical variables (like wind and radiation) adds a layer of physical interpretability to the "black box" deep learning models, making the predictions more trustworthy for grid operators.

In summary, the paper establishes that a physics-aware, multimodal deep learning approach is superior to purely visual or statistical methods for photovoltaic power forecasting, particularly in challenging weather conditions.