MPCM-Net: Multi-scale network integrates partial attention convolution with Mamba for ground-based cloud image segmentation

This paper proposes MPCM-Net, a novel multi-scale network integrating partial attention convolutions with Mamba architectures to address limitations in current ground-based cloud image segmentation for photovoltaic forecasting, while also introducing the CSRC dataset to benchmark superior accuracy and efficiency.

The Gist

Imagine you are trying to predict how much sunlight will hit a solar panel farm tomorrow. To do this accurately, you need to know exactly where the clouds are, how thick they are, and what color they are. If a cloud is thin and white, it lets some sun through; if it's thick and gray, it blocks most of it.

The problem is that looking at the sky from the ground is tricky. Clouds change shape, they stretch out like long ribbons near the horizon, and the sun's glare can make the edges of clouds look blurry or even turn them red.

This paper introduces a new AI system called MPCM-Net designed to solve these specific problems. Think of it as a super-smart, high-speed camera operator who never blinks and can instantly draw perfect outlines around every cloud, no matter how weird the shape or lighting is.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Blurry Edge" and the "Big Picture"

Current AI systems are like a student trying to draw a map of the world.

• The Issue: They are good at seeing the big continents (large clouds) but often miss the tiny islands (small cloud fragments).
• The Glare: When the sun is in the picture, the AI gets confused. The bright light makes the edge of a cloud look like part of the sky, so the AI draws the line in the wrong place.
• The Speed: Some AI systems are so complex they take too long to think. For solar power, we need answers in milliseconds, not minutes.

2. The Solution: MPCM-Net's "Smart Team"

The authors built a new AI with two main teams: an Encoder (the observer) and a Decoder (the painter).

The Encoder: The "Selective Observer"

Instead of looking at every single pixel of the sky (which is slow and wasteful), this team uses a technique called Partial Attention.

• The Analogy: Imagine you are at a crowded party. A normal person tries to listen to every conversation at once and gets overwhelmed. MPCM-Net is like a spy who only listens to the most important conversations (the cloud edges) and ignores the background noise.
• The "Strip" Trick: Clouds near the horizon often look like long, stretched-out strips because of how the camera lens works. Standard AI uses square "magnifying glasses" to look at them, which is inefficient. MPCM-Net uses long, thin "strip" magnifying glasses that fit the shape of the clouds perfectly, saving time and energy.
• The "Color" Insight: The authors realized that existing AI didn't care enough about color. A white cloud near the sun looks different than a gray cloud in the shade. MPCM-Net was trained to notice these subtle color shifts to know exactly where the cloud ends and the sky begins.

The Decoder: The "Mamba Painter"

Once the observer finds the clouds, the painter has to draw them back onto the full-size image.

• The Issue: Usually, when you zoom back in, the edges get fuzzy.
• The Mamba Solution: They used a new type of AI architecture called Mamba. Think of Mamba as a painter who doesn't just look at the spot they are painting; they can "remember" the whole picture at once without getting tired.
• The Hybrid Domain: This painter uses a special technique called SSHD. It's like having two brushes: one that paints the fine details (the texture of the cloud) and one that paints the big picture (the overall shape). It mixes them together so the final image is sharp and accurate, even in the tricky areas near the sun.

3. The New "Textbook": The CSRC Dataset

You can't teach a student well if you give them a textbook with blurry pictures. The authors realized that all the existing "textbooks" (datasets) for cloud AI were too simple. They mostly just said "Cloud" or "No Cloud."

So, the team created a brand new, super-detailed dataset called CSRC.

• What's new? It doesn't just say "Cloud." It says: "This is a white, thin cloud," "This is a thick gray cloud," and "This is the sun."
• Why it matters: This teaches the AI to understand that a thin white cloud lets sun through, while a thick gray one blocks it. This is crucial for predicting how much electricity the solar panels will make.

4. The Results: Fast, Accurate, and Ready for the Real World

When they tested MPCM-Net against other top AI systems:

• Accuracy: It drew the cloud outlines much more precisely, especially in the tricky, blurry areas near the sun.
• Speed: It was incredibly fast. While other complex systems were like a slow turtle, MPCM-Net was a hare. It could process images fast enough to be used in real-time solar power plants.
• Efficiency: It achieved this high speed not by cutting corners, but by being smarter about what it looked at (the partial attention) and how it remembered the image (the Mamba architecture).

Summary

In short, the authors built a new AI that acts like a super-observant, fast-thinking meteorologist. It ignores the noise, focuses on the important shapes and colors of clouds, and draws perfect boundaries even when the sun is blinding. They also built a better "textbook" to train it, ensuring that solar farms can predict their energy output with much higher confidence.

Technical Summary

1. Problem Statement

Ground-based cloud image segmentation is critical for photovoltaic (PV) power forecasting, as it enables real-time identification of cloud formations and irradiance fluctuations. However, existing Deep Learning (DL) approaches face three primary limitations:

• Inefficient Multi-scale Extraction: Current methods rely heavily on dilated convolutions which often fail to leverage inter-channel interoperability and partial feature efficacy, leading to suboptimal global information interaction.
• Accuracy-Throughput Trade-off: Attention-based feature enhancement mechanisms frequently introduce high computational overhead, compromising inference speed required for real-time deployment.
• Decoder Limitations: Standard decoder modifications often fail to re-establish global interdependencies among hierarchical local features, resulting in the loss of fine-grained boundary details and "ghosting effects" during motion extrapolation.
• Dataset Deficiencies: Existing public datasets are limited to binary segmentation (cloud vs. sky) or lack fine-grained categorization based on radiative properties and color attributes, which are essential for accurate PV power prediction.

2. Methodology: MPCM-Net

The authors propose MPCM-Net, a novel encoder-decoder architecture designed to balance segmentation accuracy with inference efficiency.

A. Encoder: Multi-scale Partial Attention Convolution (MPAC)

The encoder replaces standard convolutions with a specialized MPAC block to adaptively extract multi-scale context while reducing redundancy. It consists of:

1. Multi-scale Partial Convolution (MPC) Block:
   • Partial Channel Module (ParCM): Splits input channels into a primary subset (processed via 3×3 convolution for global context) and a secondary subset (processed via Coordinate Attention to model spatial interactions). This reduces computational redundancy while preserving feature discriminability.
   • Partial Spatial Module (ParSM): Uses a split-and-process strategy with spatial selectivity mechanisms to focus on relevant spatial regions, mitigating boundary ambiguity caused by downsampling.
2. Multi-scale Partial Attention (MPA) Block:
   • Combines a Partial Attention Module (ParAM) with ParSM.
   • Introduces Softmax-like Linear Attention (SLA): A novel attention mechanism that approximates the amplitude propagation of standard Softmax attention but maintains linear computational complexity ($O(N)$). It addresses the performance degradation of standard linear attention by introducing an offset term to amplify weights for salient regions.

B. Decoder: Multi-scale Mamba Block (M2B)

The decoder utilizes a Multi-scale Mamba Block (M2B) to recover spatial details and establish global dependencies without the quadratic complexity of Transformers.

• Spatial-Semantic Hybrid Domain (SSHD): This is the core innovation within M2B. It processes features through two parallel branches:
  1. 2D-SSM Branch: Uses a 2D Selective State Space Model to capture global contextual information and long-range dependencies via selective scanning in four directions.
  2. Hybrid Attention (HA) Branch: Refines fine-grained information using a Gaussian-SE module (incorporating mean and variance statistics) and spatial attention.
• The SSHD effectively bridges the gap between local details and global semantics, mitigating contextual loss during upsampling.

C. Loss Function

The model employs a compound loss function combining Focal Loss (to handle class imbalance and focus on hard boundary pixels) and Dice Loss (to optimize structural integrity and Intersection-over-Union).

3. Key Contributions

1. MPCM-Net Architecture: A novel network integrating partial attention convolutions in the encoder and Mamba-based decoding in the decoder, achieving an optimal balance between accuracy and speed.
2. Novel Modules:
   • MPAC: Introduces ParCM, ParSM, and ParAM to facilitate global information exchange and reduce computational costs.
   • SLA: A new attention mechanism that retains the representational power of Softmax while maintaining linear complexity.
   • SSHD: A hybrid domain in the decoder that captures hierarchical global correlations to recover boundary details.
3. CSRC Dataset: The authors introduce and release the Complex-Scale, Radiative, and Color (CSRC) dataset. Unlike previous benchmarks, CSRC is a fine-grained dataset (2,330 samples, 1260×1260 resolution) that categorizes clouds based on radiative sources (Sun), color attributes (White, Gray, Blue background), and complex scales, addressing critical gaps in existing public data.

4. Experimental Results

Extensive experiments were conducted on the CSRC dataset, comparing MPCM-Net against state-of-the-art (SOTA) methods including U-Net, DeepLabv3+, SegFormer, CloudFU-Net, and FA-CloudSeg.

• Quantitative Performance: MPCM-Net achieved the highest Mean Intersection over Union (MIoU) of 64.8%, outperforming the second-best method (FA-CloudSeg at 63.1%) by 1.7%. It also achieved the highest Precision (90.6%) and Recall (91.1%).
• Efficiency: Despite having more parameters than lightweight models like BSANet, MPCM-Net achieved the lowest inference latency (24.6 ms) among high-performing models and significantly lower FLOPs than Transformer-based approaches.
• Qualitative Analysis: Visual results demonstrated superior handling of:
  • Multi-scale variations: Accurate segmentation of both large stratiform clouds and small cumulus fragments.
  • Circumsolar regions: Effective handling of high-radiation zones where spectral boundaries are blurred.
  • Boundary Integrity: Sharper edges and reduced "ghosting" compared to baseline methods.
• Ablation Studies: Confirmed that removing MPAC or MMD modules caused significant MIoU drops (3.6% and 3.2% respectively), validating the necessity of partial attention mechanisms and the Mamba-based decoder.

5. Significance

This work represents a significant advancement in ground-based cloud segmentation for renewable energy applications.

• Practical Impact: By achieving high accuracy with low latency, MPCM-Net is suitable for ultra-short-term PV power forecasting, enabling grid dispatch operations to react quickly to cloud-induced irradiance fluctuations.
• Data Contribution: The release of the CSRC dataset sets a new standard for the research community, providing a realistic, fine-grained benchmark that incorporates radiative and color properties previously ignored.
• Architectural Innovation: The successful integration of Mamba (State Space Models) with Partial Attention mechanisms demonstrates a viable path forward for high-performance, real-time semantic segmentation tasks that require both global context and fine-grained detail.