A Controlled Visual-Backbone Benchmark for Multimodal Short-Term Solar Irradiance Forecasting
This paper presents a controlled benchmark for multimodal short-term solar irradiance forecasting that isolates the impact of visual backbones (including ConvNeXt, Swin Transformer, and Mamba variants) by keeping the entire forecasting pipeline fixed, demonstrating that while these architectures improve upon smart persistence on the Folsom dataset, they do not yet surpass it on the stricter NREL split.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine trying to predict the weather not by looking at a giant map of the whole world, but by staring at a single, tiny window in your own backyard. That's essentially what solar energy forecasters do. They use cameras pointed at the sky to snap pictures of clouds, hoping to guess how much sunlight will hit solar panels in the next few minutes. Why does this matter? Because solar power is a bit like a fickle friend; it's great when the sun is shining, but if a cloud rolls by unexpectedly, the power supply can drop instantly. To keep the lights on, engineers need to know exactly when that cloud will arrive.
For years, scientists have been trying to build "smart eyes" for these cameras using artificial intelligence. They've built many different types of AI "brains" (called backbones) to look at the sky pictures. Some brains are built like classic convoluted nets, others are like giant attention-grabbing transformers, and the newest ones are like state-space models that scan images like a snake slithering through a maze. The problem is, every time someone tests a new brain, they also change the camera settings, the way they feed in weather data, and the rules for how the AI learns. It's like testing a new race car engine but also swapping the tires, the driver, and the track at the same time. You can't tell if the engine is actually faster, or if you just got lucky with the tires. This paper steps in to fix that confusion by creating a perfectly controlled race.
The researchers, a team from the University of Peradeniya in Sri Lanka, decided to run a "fair play" tournament. Instead of letting every AI model change its own rules, they built one single, unchangeable racing track. They kept the camera settings, the weather history data, the way the AI combines the picture with the weather, and even the computer's random number generator exactly the same for every single test. The only thing they changed was the "visual backbone"—the specific type of AI brain looking at the sky. They tested five different families of brains: ConvNeXt, Swin Transformer, VMamba, Spatial Mamba, and MambaVision.
The results were a bit like a high-stakes game of "who can guess the cloud's path best?" On the main test track, called Folsom (which had over 224,000 test samples), every single visual AI model managed to beat the "smart persistence" baseline. Think of "smart persistence" as a simple rule: "If the sun was shining five minutes ago, it will probably shine five minutes from now." Surprisingly, this simple rule is very hard to beat. However, the AI models did edge it out. The winner of the race, by a hair's breadth, was VMamba Small, which predicted the sunlight with an error of 65.39 W m⁻². It was followed almost immediately by Swin Base, with an error of 65.50 W m⁻². The difference between them was so tiny—just 0.11 W m⁻²—that the authors suggest we shouldn't declare a single, absolute winner, but rather see them as a very close pack of top contenders.
But here is where the story gets interesting and a little humbling. The researchers also tested these models on a second track, the NREL dataset, which was much smaller and trickier. After all the filtering for matching data, this track only had 313 samples. On this tiny, difficult track, the simple "smart persistence" rule actually crushed all the fancy AI models. The best visual AI (Swin Tiny) had an error of 23.76 W m⁻², while the simple rule was only 17.48 W m⁻². This suggests that when there isn't enough data to learn from, the complex AI brains actually get confused, and the simple "guess it will stay the same" strategy is the safest bet.
The paper also looked at how fast these models could think. The VMamba Small model was the most accurate but was a bit slower, running at 168.5 FPS (frames per second). Meanwhile, the ConvNeXt Tiny and Swin Tiny models were much faster, zooming along at 561.5 FPS and 471.5 FPS respectively, while only losing a tiny bit of accuracy. This is like choosing between a super-fast sports car that gets slightly worse gas mileage versus a slightly slower sedan that is incredibly efficient. The authors found that making the AI "brains" bigger and more complex didn't always make them better; sometimes, the smaller, simpler models were just as good and much faster.
In the end, this paper doesn't claim to have found the "perfect" AI for solar forecasting. Instead, it provides a clear, controlled map of how different AI brains perform when they are all forced to play by the exact same rules. It suggests that while models like VMamba Small and Swin Base are currently the top performers on good data, the choice of model depends heavily on how much data you have and how fast you need the answer. The authors conclude that before we start building massive, complex systems, we should stick to these controlled comparisons, check our simple baselines, and make sure we aren't just chasing bigger models when a smaller, faster one might do the job just as well.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.