On the Importance of Clearsky Model in Short-Term Solar Radiation Forecasting

This paper proposes a novel "Clearsky-Free" short-term solar radiation forecasting approach using Extreme Learning Machines (ELM) that directly learns from raw Global Horizontal Irradiance data to eliminate the limitations and complexities of traditional clearsky models while achieving accuracy and robust uncertainty quantification comparable to or better than existing benchmarks.

The Gist

Imagine you are trying to predict exactly how much sunlight a solar panel will get tomorrow. This is crucial for running a city's power grid efficiently. If you guess wrong, you might waste energy or cause blackouts.

For years, scientists have tried to solve this by using a "perfect world" reference point. They ask: "How much sun would we get if there were absolutely no clouds, dust, or pollution?" They call this the Clear Sky Model. Then, they try to figure out how much the actual weather (clouds, haze) reduces that perfect amount.

The Problem with the Old Way:
Think of the old method like trying to bake a cake by first calculating the perfect, theoretical recipe for a cake in a vacuum, and then trying to subtract the "messiness" of your actual kitchen (humidity, bad oven, stale flour).

• It's complicated.
• It requires knowing exactly how much dust is in the air (which is hard to measure).
• It breaks down at sunrise and sunset (when the sun is low, the math gets messy).
• If your "perfect recipe" calculation is slightly off, your whole prediction fails.

The New "Clear-Sky-Free" Approach:
This paper proposes a smarter, simpler way. Instead of calculating a perfect theoretical sky and then subtracting errors, the authors say: "Just look at the raw sunlight data from the past and let a smart computer figure out the pattern."

They used a specific type of AI called an Extreme Learning Machine (ELM). Here is how it works, using a few analogies:

1. The "Muscle Memory" Analogy

Imagine a professional basketball player.

• The Old Way (Clear Sky Model): The player tries to calculate the physics of the ball, the wind speed, the humidity, and the exact angle of the hoop before every shot. It's slow and prone to calculation errors.
• The New Way (ELM): The player doesn't do the math. They just rely on muscle memory. They have shot thousands of balls before. They know that if the ball feels a certain way and the hoop looks a certain way, they should shoot this way. They learned the "feel" of the game directly from experience, not from a physics textbook.

The ELM model does exactly this with sunlight. It looks at the last few hours of sun data and says, "Based on how the sun behaved yesterday and this morning, here is what it will do next." It implicitly learns about clouds, dust, and humidity without needing to measure them explicitly.

2. The "Weather Forecast" Analogy

• Old Method: To predict if it will rain, you first calculate what the weather would be like if the atmosphere were empty, then try to add clouds back in.
• New Method: You just look at the sky right now and the last few days. You see the clouds moving. You don't need to know the chemical composition of the clouds to know it's going to rain in 10 minutes. The AI looks at the raw data (the "sky") and learns the patterns of change directly.

Why is this a Big Deal?

The authors tested this new method against the old "Clear Sky" methods using data from 76 weather stations in Spain. Here is what they found:

• It's Faster: The AI learns the patterns in seconds. The old method takes a long time to calculate all the atmospheric physics.
• It's More Accurate: The AI made fewer mistakes, especially for predictions 1 to 6 hours ahead.
• It's More Robust: It doesn't break down at sunrise or sunset, where the old math often gets confused.
• It's Simpler: You don't need expensive sensors to measure dust or water vapor in the air. You just need a sensor that measures sunlight.

The "Uncertainty" Bonus

The paper also shows that this AI can tell you how sure it is.

• Old Way: "I think it will be 500 Watts." (But maybe it's 400 or 600).
• New Way: "I think it will be 500 Watts, and I'm 90% sure it will be between 480 and 520."
  This helps energy companies manage risk better.

The Bottom Line

This paper is essentially saying: "Stop over-engineering the problem."

Instead of building a complex, fragile machine to simulate the perfect sky and then subtracting the real world, just let a smart, fast computer learn the rhythm of the sun directly from the data. It's like teaching a child to ride a bike by letting them feel the balance, rather than making them study the physics of gyroscopes first.

The result? A cheaper, faster, and more accurate way to predict solar power, which helps us use more clean energy and less fossil fuel.

Technical Summary

1. Problem Statement

Solar irradiance forecasting is critical for grid stability, energy trading, and renewable integration. Traditional short-term forecasting methods typically rely on Clearsky-Based models. These approaches involve a multi-step process:

1. Calculating theoretical clear-sky irradiance ($GHI_{cs}$) using models like McClear, REST2, or SPARTA.
2. Computing the Clearsky Index (CSI) as the ratio of measured Global Horizontal Irradiance (GHI) to $GHI_{cs}$.
3. Forecasting the CSI and rescaling it back to GHI.

Limitations of Current Approaches:

• Synchronization Errors: Mismatches between measured data and model outputs, particularly at low solar elevations (sunrise/sunset).
• Data Dependency: Reliance on atmospheric inputs (Aerosol Optical Depth, water vapor, ozone) which are often unavailable, inaccurate, or delayed in real-time scenarios.
• Mathematical Instability: The CSI becomes undefined at night and prone to large errors during transitional periods (dawn/dusk) due to division by near-zero values.
• Complexity: Requires complex intermediate calculations and synchronization with Numerical Weather Prediction (NWP) systems.

The authors hypothesize that these limitations can be bypassed by a Clearsky-Free approach that learns atmospheric attenuation (aerosols, clouds, water vapor) directly from raw historical data using advanced machine learning, eliminating the need for explicit Clearsky normalization.

2. Methodology

Data Collection and Preprocessing

• Dataset: High-resolution (30-min) GHI data from 76 stations in the SIAR agroclimatic network in Spain over four years.
• Quality Control: Strict filtering for sensor malfunctions and outliers. The McClear model was used in hindcast mode (2-day delay) solely for data validation and anomaly detection, not for real-time forecasting inputs.
• Inputs: The model uses endogenous features (lagged GHI values) and temporal features (hour of day, day of year). It does not use external atmospheric data (AOD, water vapor) as direct inputs.

Modeling Framework

The study compares a Clearsky-Free approach against traditional Clearsky-Based benchmarks.

• Proposed Model: Extreme Learning Machine (ELM)

  • Architecture: A single-hidden-layer neural network.
  • Training: Unlike deep learning, ELM uses random initialization for input-to-hidden weights and analytically computes output weights via Ridge Regression (pseudo-inverse), avoiding iterative backpropagation.
  • Configuration: Optimized per horizon (e.g., ~78 input neurons, ~472 hidden neurons for 1-hour horizon). Uses a hybrid activation function (60% Sigmoid, 40% Gaussian).
  • Probabilistic Extension: Uses Quantile Regression (QR) and non-parametric lookup tables based on forecast residuals to generate prediction intervals.

• Benchmarks:

  • Naive: Persistence (P), Clearsky (CS), Smart Persistence (SP).
  • Statistical: AR(p) (with and without Clearsky normalization), CLIPER, Exponential Smoothing (ES), ARTU.
  • Ensemble: Combination model (COMB).

Evaluation Metrics

• Deterministic: Normalized Root Mean Square Error (nRMSE), Normalized Mean Absolute Error (nMAE), Bias (nMBE), and $R^2$.
• Probabilistic: Continuous Ranked Probability Score (CRPS), Prediction Interval Coverage Probability (PICP), Mean Interval Length (MIL), and Interval Score (IS).
• Statistical Significance: Mann-Whitney U test to compare error distributions.

3. Key Contributions

1. Validation of Clearsky-Free Forecasting: The paper demonstrates that a data-driven model can implicitly learn the effects of atmospheric extinction (aerosols, ozone, water vapor) and cloud cover directly from raw GHI time series, rendering explicit Clearsky models unnecessary for short-term forecasting.
2. Superiority of ELM: The study proves that ELM outperforms traditional autoregressive models (AR) and Clearsky-based benchmarks, particularly in medium (180 min) and long (360 min) horizons.
3. Elimination of Stationarization: The work challenges the necessity of stationarizing solar data via CSI. It shows that modern ML architectures can handle non-stationary, periodic data directly, simplifying the pipeline and removing synchronization errors.
4. Probabilistic Robustness: The ELM-based probabilistic approach provides tighter prediction intervals with comparable or better coverage than Quantile Regression, offering better precision for energy trading.

4. Key Results

• Deterministic Performance:

  • ELM vs. AR: ELM consistently outperformed the standard AR model across all metrics and horizons (p < 0.05).
  • ELM vs. Clearsky Models: ELM significantly outperformed the "Smart Persistence" and "Clearsky" benchmarks.
  • Ranking: The final ranking of models from best to worst was: ELM > AR > rAR (AR with Clearsky) > CLIPER > SP > ARTU > ES > COMB > CS > P.
  • Horizon Sensitivity: While differences were less pronounced at 30 minutes, ELM showed dominant performance at 180 and 360 minutes.

• Probabilistic Performance:

  • Precision: ELM generated narrower prediction intervals (lower nMIL) than Quantile Regression (QR) while maintaining competitive coverage.
  • Accuracy: ELM achieved lower CRPS and Interval Scores (IS) than QR, indicating forecast distributions closer to actual observations.
  • Calibration: While QR showed slightly better calibration (PICP closer to nominal 80%), ELM's tighter intervals provided higher actionable precision.

• Computational Efficiency:

  • ELM training and inference were extremely fast (<30 seconds for 70 sites), making it highly suitable for real-time operational use compared to computationally heavy NWP or complex deep learning models.

5. Significance and Implications

• Operational Simplicity: By removing the dependency on atmospheric input data (AOD, water vapor) and complex Clearsky normalization, the proposed method is more robust in data-scarce regions and easier to deploy.
• Grid Integration: The ability to provide accurate, real-time probabilistic forecasts without synchronization errors improves grid management, energy storage optimization, and market participation.
• Paradigm Shift: The study suggests a shift away from physics-based pre-processing (Clearsky Index) toward purely data-driven, adaptive statistical learning for solar forecasting.
• Future Directions: The authors suggest that while Clearsky-Free is superior for raw GHI, hybrid approaches (using McClear as an input feature rather than a normalization step) could further enhance accuracy. Future work will focus on applying this to tilted surfaces (GTI) and integrating real-time aerosol data for extreme pollution events.

Conclusion: The paper establishes that Clearsky-Free ELM models are a superior, scalable, and robust alternative to traditional Clearsky-based methods for short-term solar radiation forecasting, offering higher accuracy with significantly reduced computational and data requirements.