Spectral-Domain Local Statistics with Missing-Data Support for Cartesian and Polar Grids

This paper introduces a boundary-aware spectral method using DCT and RFFT to compute stable local statistics (such as mean and variance) on incomplete Cartesian and polar grids, effectively mitigating wrap-around artifacts and handling missing data.

The Gist

Imagine you are trying to take a beautiful, high-resolution photo of a crowded festival, but there’s a problem: some parts of the photo are blurry, some are blocked by pillars, and some parts are just pure black because the camera failed.

If you tried to "average out" the colors to fix the photo using standard math, the black spots would make the whole image look muddy and dark. Even worse, if you used a standard mathematical trick called "periodic wrapping," the people on the far left edge of your photo might accidentally "bleed" into the people on the far right edge, creating a ghostly, nonsensical mess.

This scientific paper describes a smarter way to "clean up" and understand messy, incomplete data—specifically for things like weather radar and satellite imagery.

Here is the breakdown of how they do it, using everyday analogies:

1. The "Smart Spotlight" (Normalized Convolution)

Imagine you are walking through a dark room with a flashlight. You want to know the average brightness of the area around you.

• The Old Way: You just shine the light and take an average. But if a large part of the area is a black hole (missing data), your average brightness will look much lower than it actually is.
• The Paper’s Way: They use a "Smart Spotlight." The math doesn't just look at the brightness; it also looks at how much of the floor is actually visible. If the flashlight only hits 20% of the floor because the rest is a hole, the math says, "Hey, only count that 20%! Don't let the darkness trick you into thinking the whole room is dim." This is what they call "support-aware" statistics.

2. The "Mirror vs. The Carousel" (Boundary Conditions)

When computers process grids of data, they often get confused at the edges. The researchers solved this by giving the computer two different sets of rules depending on what it’s looking at:

• The Mirror (Cartesian Grids): For standard maps (like a city grid), they use the DCT (Discrete Cosine Transform). Imagine standing at the edge of a lake; instead of the world suddenly ending, the math treats the edge like a mirror. It reflects the data back inward so the math doesn't "fall off the cliff" or create weird artifacts.
• The Carousel (Polar Grids): For weather radar, which spins in a circle, they use the RFFT (Real Fast Fourier Transform). This treats the data like a carousel. The "end" of the circle (360 degrees) is seamlessly connected to the "beginning" (0 degrees), so the wind patterns don't look broken where the circle closes.

3. The "Bouncer" (Outlier Identification)

The researchers tested this by creating a fake "wind storm" (a 3D cyclone) and throwing in some "fake" wind gusts (outliers) to see if their system could catch them.

• They used the Local Standard Deviation as a bouncer at a club. The bouncer knows what a "normal" person looks like in a specific neighborhood. If a gust of wind shows up that is wildly different from its immediate neighbors, the bouncer flags it as an "outlier."
• Because the math is "local," it doesn't get confused by the massive, natural changes in a giant storm; it only cares if a specific tiny spot is acting weird compared to the people standing right next to it.

Why does this matter?

In the real world, meteorologists need to know exactly where a storm is intensifying or where a radar sensor is failing. If the math is "dumb," the errors in the sensor look like actual weather patterns, which could lead to bad forecasts.

This paper provides a "mathematical toolkit" that ensures when we look at messy, hole-filled weather data, we are seeing the actual weather, not just the mathematical glitches caused by the holes.

Technical Summary

Technical Summary: Spectral-Domain Local Statistics with Missing-Data Support

1. Problem Statement

Scientific imaging and remote sensing datasets (e.g., meteorology, radar, Earth observation) frequently contain "missing" or invalid data due to sensor limitations, beam blockage, or quality-control masking. Traditional methods for computing local statistics (mean, variance, standard deviation) face two primary challenges:

• Computational Efficiency: Spatial-domain convolution scales poorly ($O(N \cdot K)$) for large support windows compared to spectral-domain methods ($O(N \log N)$).
• Boundary and Data Artifacts: Standard Fast Fourier Transform (FFT) approaches assume periodic boundaries, which causes "wrap-around" artifacts in non-periodic Cartesian grids. Furthermore, treating missing values as zeros in spectral transforms conflates "no data" with "zero value," leading to biased statistics.

2. Methodology

The authors propose a framework for computing normalized local statistics using boundary-aware spectral operators. The core approach involves applying a smoothing operator $S_g$ to both the data and a certainty mask to ensure that statistics are weighted by local support.

Key components include:

• Boundary-Aware Transforms:
  • Reflective (DCT-II): Used for Cartesian axes and radar range to enforce half-sample symmetric extension, preventing wrap-around artifacts.
  • Periodic (RFFT): Used for the azimuth axis in polar geometry to respect the circular nature of the scan.
• Normalized Convolution: To handle missing data, the local mean ($\mu$) is calculated as the ratio of the smoothed masked data to the smoothed mask: $\mu = S_g\{x \cdot m\} / S_g\{m\}$. This ensures the estimate is unbiased by the presence of NaNs.
• Stability Safeguards: To prevent numerical instability in regions with low data support, the framework implements:
  • A denominator floor ($\epsilon$) to prevent division by zero.
  • A prefill fallback where under-supported regions are replaced by a prefilled field (iterative spectral smoothing).
  • A variance clamp to prevent negative variance caused by floating-point errors.
• Adaptive Polar Smoothing: For polar grids, the azimuth kernel width is adaptively scaled based on the range ($r$) to maintain a constant physical smoothing width.

3. Key Contributions

1. Unified Framework: Generalizes previous work to provide a single mathematical framework for both Cartesian and polar grids with explicit boundary mode controls.
2. Multi-Statistic Support: Extends beyond simple mean smoothing to provide local variance, standard deviation, and effective sample count ($N_{eff}$).
3. Outlier Identification Logic: Formulates a method for 3D outlier detection using local z-scores ($|v - \mu| > k\sigma$), which is more robust in complex, sign-changing flows (like cyclones) than global statistics.
4. Open-Source Implementation: Provides the dct_toolkit for reproducible scientific workflows.

4. Results and Validation

The framework was validated through three distinct scenarios:

• Cartesian Boundary Check: Demonstrated that using DCT (reflective) instead of FFT (periodic) significantly reduces RMSE in non-periodic signals. Periodic treatment showed a 258% increase in all-domain RMSE and a 561% increase in edge-only RMSE due to wrap-around artifacts.
• 3D Synthetic Outlier Test: In a simulated 3D cyclone wind field, the method successfully identified sparse anomalies. The optimal balance between precision and recall (F1 score) was found at a threshold of $k=3.0$, proving that local normalization effectively adapts to structured, high-variance environments.
• Real-Radar Application: Applied to X-band radar data with significant beam blockage. The results showed that the mixed-boundary polar operator (periodic azimuth/reflective range) produced coherent, stable local-mean estimates without the distortions typically caused by large data gaps.

5. Significance

This paper provides a robust, computationally efficient solution for high-dimensional scientific data processing. By integrating boundary-condition modeling directly into the spectral transform, the authors allow researchers to compute reliable local diagnostics in "messy" real-world datasets. The method is particularly significant for radar meteorology and geophysical imaging, where the transition between Cartesian and polar geometries and the presence of large missing-data sectors are standard operational challenges.