Spectral-Structured Diffusion for Single-Image Rain Removal

Imagine you are looking at a beautiful landscape photo, but someone has taken a marker and drawn hundreds of messy, diagonal lines all over it. That's what a "rainy image" looks like to a computer. The goal of rain removal is to erase those lines without smudging the trees, buildings, or faces underneath.

This paper introduces a new tool called SpectralDiff to solve this problem. Here is how it works, broken down into simple concepts and analogies.

1. The Problem: Rain is "Structured," Not Random

Most old rain-removal tools treat rain like static on an old TV—just random, messy noise. They try to smooth it out, which often makes the whole picture look blurry or washed out.

The authors realized something important: Rain isn't random.

Rain streaks have a specific direction (they fall diagonally).
They have specific thicknesses (some are thin, some are thick).
They overlap in layers.

Think of rain not as a chaotic mess, but as a stack of transparent sheets, where each sheet has a specific pattern of lines drawn on it. To fix the photo, you don't just want to "scrub" the image; you want to peel off these sheets one by one.

2. The Solution: A "Frequency" Filter (The Spectral Part)

Instead of looking at the rain as lines on a picture (the "spatial" view), the authors look at it through a special lens called the Frequency Domain.

The Analogy: The Piano Tuner
Imagine the image is a song played on a piano.

Rain streaks are like a specific, annoying hum or a specific note being played loudly and repeatedly.
The background (trees, sky) is the rest of the beautiful melody.

Standard tools try to mute the whole piano to get rid of the hum. SpectralDiff acts like a master piano tuner. It listens to the song, identifies the exact frequency of the annoying hum (the rain), and gently damps only that specific note, leaving the rest of the music perfectly clear.

By doing this in the "frequency" world, the computer knows exactly which "direction" and "thickness" of rain to remove, making the process much smarter and less likely to damage the original photo.

3. The Process: Peeling the Onion (Diffusion)

The paper uses a technique called Diffusion. Usually, diffusion models work by adding noise to an image and then teaching a computer to reverse the process (removing the noise step-by-step).

The Analogy: The Onion Peeler
Imagine the rainy image is an onion with many layers of skin (rain streaks).

Old methods tried to slice the whole onion off in one giant chop. This often cut into the onion flesh (the real image).
SpectralDiff peels the onion layer by layer.
- Step 1: Remove the thickest, most obvious rain.
- Step 2: Remove the medium rain.
- Step 3: Remove the fine, misty rain.

Because the computer knows the "frequency" of each layer, it peels them off perfectly without hurting the onion inside.

4. The Speed Boost: The "Full-Product" U-Net

Doing all this math usually takes a long time and requires a supercomputer. The authors wanted to make it fast enough for regular use.

The Analogy: The Shortcut Chef
Usually, to mix ingredients (convolution), a chef has to chop, stir, and mix every single item individually. This is slow.
The authors invented a new kitchen tool called the Full-Product U-Net.

Instead of chopping and mixing, this tool uses a magic "multiplication" trick.
It realizes that if you know the recipe (the rain pattern), you don't need to stir the whole pot. You just multiply the ingredients by the right amount instantly.

This makes the computer 18 times faster at the heavy lifting, allowing it to clean a photo in just 10 steps instead of 100, without losing any quality.

Summary: Why is this a big deal?

Smarter: It understands that rain has a specific shape and direction, so it doesn't blur the photo.
Faster: It uses a mathematical shortcut (the Full-Product U-Net) to work 18x more efficiently.
Better Results: It works great on both fake rain (computer-generated) and real rain (from actual cameras), handling complex weather better than previous tools.

In short, SpectralDiff is like a smart, fast, and gentle eraser that knows exactly how to wipe away rain without smearing the masterpiece underneath.

Here is a detailed technical summary of the paper "SPECTRAL-STRUCTURED DIFFUSION FOR SINGLE-IMAGE RAIN REMOVAL" by Yucheng Xing and Xin Wang.

1. Problem Statement

Single-image rain removal is a challenging computer vision task due to the ill-posed nature of separating rain streaks from background details. Key difficulties include:

Structural Complexity: Rain streaks exhibit strong directional and frequency-concentrated characteristics that overlap across multiple scales.
Limitations of Existing Methods:
- Traditional/Model-driven: Often rely on handcrafted priors (e.g., sparsity, rank) that fail in complex real-world conditions, leading to over-smoothing.
- Deep Learning (CNN/Transformer): While effective, many do not explicitly encode the spectral structure of rain, often treating rain as generic noise.
- Standard Diffusion Models: Existing diffusion-based restoration methods typically use isotropic Gaussian noise in the spatial domain. This fails to capture the directional and scale-specific nature of rain streaks, which are highly structured in the frequency domain.

2. Methodology: SpectralDiff

The authors propose SpectralDiff, a framework that integrates spectral-structured perturbations into the diffusion process and optimizes the underlying architecture for efficiency.

A. Theoretical Foundation: Layered Rain Structure

The paper models a rainy image $O$ as a superposition of a clean background $B$ and multiple rain layers $R_d$ ( $d=1 \dots D$ ), where each layer shares similar physical properties (scale, thickness, direction).

Analogy: The authors draw a conceptual parallel between the layered accumulation of rain and the iterative denoising steps of diffusion models.
Insight: While rain is complex in the spatial domain, its dominant frequency content is compact in the spectral domain. Straight, elongated rain streaks correspond to concentrated energy ridges in the frequency domain, oriented orthogonally to their spatial direction.

B. Spectral-Structured Diffusion Process

Instead of injecting standard isotropic Gaussian noise, SpectralDiff introduces structured spectral perturbations:

Frequency-Domain Masking: The noise injection occurs in the frequency domain. A complex Gaussian noise $\epsilon_f$ $ϵ_{f}$ is modulated by a direction- and scale-aware mask $M_d$ $M_{d}$ .
- Radial Mask ( $M_{d,r}$ ): A Gaussian band-pass filter controlling the scale/density of the rain.
- Angular Mask ( $M_{d,\theta}$ ): A von Mises distribution controlling the orientation of the rain streaks.
Forward Process: The clean image spectrum is progressively corrupted by adding masked noise: $x_{f,d} = \sqrt{1-\beta_d}x_{f,d-1} + \sqrt{\beta_d}(M_d \odot \epsilon_f)$ .
Reverse Process: The neural network is trained to predict the induced spatial perturbation ( $\epsilon_s$ $ϵ_{s}$ ) corresponding to the masked spectral noise.
- Crucially, the model operates in the spatial domain during inference. The training process samples noise in the frequency domain, transforms it to the spatial domain via Inverse FFT (IFFT), and trains the denoiser to predict this specific spatial noise pattern. This avoids the computational cost of running the entire diffusion loop in the frequency domain.

C. Full-Product U-Net Architecture

To address the computational overhead of diffusion models (which are inherently slow), the authors propose a Full-Product U-Net:

Mechanism: Leveraging the Convolution Theorem (convolution in space $\leftrightarrow$ element-wise multiplication in frequency), the architecture replaces standard convolutional layers with element-wise product layers.
Implementation: A modulation weight $w$ is dynamically generated based on input features using a bottlenecked $1\times1 $convolution. The output is$ h = x_{input} \odot G(x_{input})$.
Benefit: This reduces the computational complexity (FLOPs) significantly (theoretically by a factor of $\approx 18$ for standard layers) while preserving modeling capacity, as the product operation approximates frequency-selective filtering.

3. Key Contributions

SpectralDiff Framework: A novel diffusion-based approach tailored for rain removal that explicitly incorporates rain-specific spectral characteristics (direction and scale) into the perturbation design.
Structured Spectral Perturbations: A mechanism that modulates noise in the frequency domain to match the physical structure of rain streaks, improving robustness over standard isotropic noise.
Full-Product U-Net: An operator-level redesign of the U-Net backbone that replaces convolutions with element-wise products, drastically improving inference efficiency without sacrificing performance.
Efficient Inference Strategy: A non-uniform DDIM sampling strategy that allocates more inference steps to dominant rain directions identified in the input image's frequency spectrum.

4. Experimental Results

The model was evaluated on synthetic (Rain1400, RainCityscapes) and real-world (SPA-Data) benchmarks.

Performance:
- Real-World Robustness: SpectralDiff achieved superior performance on the real-world SPA-Data dataset (PSNR: 38.03, SSIM: 0.895), outperforming state-of-the-art diffusion models (e.g., WeatherDiff: 36.86 PSNR) and CNN-based methods.
- Synthetic Data: It achieved competitive results on Rain1400 (PSNR: 30.39), comparable to or slightly better than existing diffusion baselines.
Efficiency:
- Speed: SpectralDiff converges in 10 steps, whereas diffusion baselines typically require 100 steps.
- Latency: It achieves a significantly lower inference time (0.114s on Rain1400) compared to WeatherDiff (3.117s) and RainDiff (11.159s).
- Model Size: The Full-Product U-Net reduced parameters from ~16.66M (Conv U-Net) to 3.15M and FLOPs from ~1269M to 199M.
Ablation Studies:
- Removing the structured mask (using raw spectral noise) caused performance to collapse, proving the necessity of the spectral structure.
- The Product U-Net maintained performance while drastically reducing computational cost.

5. Significance

This work bridges the gap between physical signal properties (spectral structure of rain) and generative modeling (diffusion processes).

Scientific Impact: It demonstrates that explicitly modeling the frequency-domain characteristics of degradation leads to more robust restoration, particularly for directional artifacts like rain.
Practical Impact: By combining spectral-aware training with a computationally efficient architecture, SpectralDiff solves the common trade-off in diffusion models where high quality comes at the cost of extreme latency. It makes diffusion-based rain removal viable for real-time or resource-constrained applications.