Physically-Informed Fuzzy Clustering of Vertical Sounding Ionograms

This paper introduces a physically-informed fuzzy clustering method using an expectation-maximization algorithm and modified Bayesian information criterion to automatically determine the optimal number of tracks and separate vertical sounding ionograms, even under disturbed ionospheric conditions, by incorporating adaptive noise filtering and extraordinary mode removal.

The Gist

Imagine the Earth's upper atmosphere, the ionosphere, as a giant, invisible mirror floating high above us. Scientists use a device called an ionosonde to "ping" this mirror with radio waves. The result is a picture called an ionogram.

Think of an ionogram like a sonar map of the ocean floor, but instead of water depth, it shows how high radio waves bounce back. In a perfect, calm world, this map would show a few clean, smooth lines (tracks) representing different layers of the atmosphere.

However, the real world is messy. The ionosphere is often turbulent, disturbed by solar storms or weather, creating a chaotic "fog" of dots on the map. Some dots are real signals bouncing off different layers, some are signals bouncing off the same layer multiple times, and many are just random static (noise).

The Problem:
Traditionally, computers tried to read these maps using rigid rules, assuming there was always a fixed number of layers (like "there are always three layers"). But when the ionosphere gets messy, those rules break down. The computer gets confused, unable to tell where one signal ends and another begins, or how many layers are actually there.

The Solution: A "Smart Detective" Approach
The authors of this paper created a new method called Physically-Informed Fuzzy Clustering. Here is how it works, using simple analogies:

1. Cleaning the Mess (Noise Filtering)

Before trying to find the lines, the computer first acts like a janitor. It looks at the scattered dots on the map.

• The Analogy: Imagine a room full of people. Some are standing in tight groups (the real signals), and others are wandering around alone or in tiny, random pairs (noise).
• The Method: The computer uses a technique called DBSCAN (a smart way to spot crowds) combined with a statistical guesser (Gaussian Mixture). It automatically decides: "These dots are too far apart to be a group; they are just noise. Let's throw them away." This leaves only the dense, meaningful clusters.

2. The "Flexible Snake" Model (The Track Shape)

Once the noise is gone, the computer tries to fit a line through the remaining dots. But it doesn't use a straight ruler or a simple curve.

• The Analogy: Imagine trying to trace the path of a snake that can stretch, shrink, and bend. The computer uses a mathematical "snake" model based on how the atmosphere physically behaves (specifically, how it acts like a parabolic layer).
• The Twist: This snake has six adjustable knobs (parameters). Three are standard (like the snake's height and width), and three are special "helper" knobs. These helpers allow the snake to wiggle and account for weird effects, like a signal bouncing off a lower layer before hitting a higher one. This makes the model flexible enough to handle the messy, real-world data.

3. The "Guess and Check" Game (Fuzzy Clustering)

The computer doesn't know how many snakes (tracks) are on the map. It has to figure that out.

• The Analogy: Imagine you are looking at a pile of mixed-up colored yarn. You don't know how many balls of yarn are in the pile. You start by guessing there are 2 balls. You try to sort the yarn. Then you guess 3, then 4, and so on.
• The Method: The computer runs a "trial and error" loop (called the Expectation-Maximization algorithm). It tries different numbers of tracks. For each guess, it asks: "Does this number of tracks explain the dots better than the last guess?"
• The "Fuzzy" Part: Unlike old methods that forced a dot to belong to only one line, this method is "fuzzy." It allows a dot to belong to two lines at once with a certain probability. This is crucial because in the real ionosphere, signals often cross or overlap. The computer says, "This dot is 60% likely to be Line A and 40% likely to be Line B," which helps untangle the mess.

4. Finding the "Goldilocks" Number

How does the computer know when to stop guessing?

• The Analogy: Imagine you are packing a suitcase. If you pack too little, you miss things. If you pack too much, you have empty space and wasted effort. You want the perfect amount.
• The Method: The computer uses a mathematical rule called the Bayesian Information Criterion (BIC). It's like a scorecard that penalizes the computer for being too complicated (guessing too many tracks) or too simple (missing tracks). The computer keeps increasing the number of tracks until it finds the "Goldilocks" number—the one that fits the data perfectly without being unnecessarily complex.

5. The Result

The final output is a clean map where the messy dots are organized into distinct, colored tracks.

• What it achieves: It can separate signals that are touching or crossing. It can tell the difference between a signal bouncing once and one bouncing twice. It works even when the number of layers is unknown.
• Speed: It takes about 3.7 minutes to process one map on a standard computer, which is fast enough for real-time monitoring.

Limitations (What the paper admits)

• One-sided view: The method currently works best if you only look at one type of radio wave (the "Ordinary" wave). If you try to mix in the other type (the "Extraordinary" wave) without special hardware to separate them, the computer gets confused.
• Randomness: Because the computer uses a "guess and check" method that involves some randomness, running the same data twice might give slightly different results, though they will be very similar.
• Shape limits: It assumes the atmospheric layers look somewhat like smooth, curved hills (parabolas). If the atmosphere is shaped in a way that defies this model, the method might struggle.

In Summary:
This paper presents a smart, flexible computer program that acts like a detective. It cleans up the static, uses a flexible "snake" model to trace the paths of radio waves, and automatically figures out how many layers of the atmosphere are present, even when the sky is chaotic and the signals are crossing over each other.

Technical Summary

1. Problem Statement

Automatic processing of vertical sounding ionograms is a critical challenge in ground-based ionospheric diagnostics. Traditional methods often rely on simplified assumptions:

• Fixed Models: They assume the ionosphere is a 1.5D inhomogeneous medium described by a fixed number of layers (e.g., simple F-layer models).
• Limited Complexity: They struggle with disturbed ionospheric conditions where multiple reflections, sporadic E-layers (Es), horizontal inhomogeneities, and intersecting tracks create complex patterns that cannot be mapped to a simple $f \to r_o, r_x$ function.
• Unknown Track Count: Existing algorithms often require the number of tracks to be known a priori, which is rarely the case in disturbed conditions.
• Noise and Artifacts: Ionograms contain noise, interference, and overlapping ordinary (O) and extraordinary (X) wave components that complicate automated separation.

The paper addresses the need for an algorithm that can automatically detect an unknown number of physically interpretable tracks, separate intersecting or contacting tracks, and handle complex, disturbed ionospheric conditions without prior knowledge of the track count.

2. Methodology

The proposed solution is a Physically-Informed Fuzzy Clustering framework based on the Expectation-Maximization (EM) algorithm. The methodology consists of four main stages:

A. Adaptive Noise Filtration

Before clustering, the algorithm removes noise and interference:

• Pre-processing: Concentrated interference is removed via block processing; low-intensity noise is removed via thresholding.
• DBSCAN + Gaussian Mixture (GM): The algorithm uses DBSCAN to identify clusters based on point density. To optimize the DBSCAN hyperparameter $\epsilon$ (distance threshold), it employs a Gaussian Mixture model on the distribution of distances to the 10 nearest neighbors. This statistically separates "signal" (dense clusters) from "noise" (isolated points).
• X-Component Removal: Since the model focuses on the Ordinary (O) mode, a preliminary heuristic removes points likely belonging to the Extraordinary (X) mode to prevent clustering artifacts.

B. Physically-Informed Track Model

The core of the method is a parametric model for ionogram tracks derived from a parabolic ionospheric layer model but extended for flexibility.

• Base Model: A standard parabolic layer model describes the relationship between frequency ($f$) and effective range ($R$).
• Extended 6-Parameter Model: To handle underlying layers and complex shapes, the model is expanded to six parameters:
  1. $h_1$: Lower boundary of the layer.
  2. $y_m$: Half-width of the layer.
  3. $f_0$: Critical frequency (maximum plasma frequency).
  4. $A, B, C$: Additional parameters to account for underlying layer effects, low-frequency track shifts, and deviations from a perfect parabola near the peak.
• Optimization: The parameters for a single track are found using the Sequential Least Squares Quadratic Programming (SLSQP) algorithm, which minimizes a weighted Mean Absolute Error (WMAE) loss function. SLSQP was selected for its superior speed-to-accuracy ratio compared to other optimizers (e.g., Trust-constr, COBYQA).

C. Fuzzy Clustering via EM Algorithm

The clustering process treats the ionogram as a mixture of distributions where each distribution corresponds to a track model.

• E-Step (Expectation): Calculates the probability of each point belonging to each track. Unlike standard EM, the probability is based on the Euclidean distance of a point from the parametric curve (normalized by standard deviations), assuming a half-normal distribution.
  • Fuzzy Logic: The algorithm uses probabilistic Top-2 sampling. Instead of assigning a point to the single most probable track, it allows points to belong to multiple tracks (up to two), which is crucial for handling intersecting or contacting tracks.
  • Uniformity Penalty: A factor is introduced to penalize tracks with non-uniform point distributions, improving the separation of valid tracks from noise.
• M-Step (Maximization): Updates the track parameters ($\theta_m$) and standard deviations ($\sigma_m$) using the SLSQP algorithm on the points assigned to that track.
• Smoothing: Exponential smoothing is applied to parameters between iterations to stabilize convergence.

D. Optimal Number of Tracks Selection

The algorithm does not assume a fixed number of tracks ($T$). Instead, it searches for the optimal $T_{opt}$:

• Iterative Search: The algorithm runs the EM process for $T$ ranging from 2 to a maximum (e.g., 28).
• Modified BIC: It uses a modified Bayesian Information Criterion (BIC) to select the best $T$. The penalty term is adjusted to heavily penalize models that result in empty clusters, preventing overfitting.
• Stopping Criterion: The search stops when the BIC value increases for 10 consecutive iterations (Early Stopping).

3. Key Contributions

1. Physically-Informed Fuzzy Clustering: A novel approach that combines physical ionospheric models with fuzzy clustering, allowing for the separation of intersecting and contacting tracks without requiring them to be disjoint.
2. Automatic Track Count Determination: The method autonomously determines the optimal number of tracks using statistical criteria (BIC), making it suitable for highly disturbed ionospheric conditions where the track count is unknown.
3. Robust Noise Handling: A hybrid DBSCAN-Gaussian Mixture approach for adaptive noise filtering that effectively distinguishes between ionospheric signals and random noise.
4. Extended Parametric Model: A 6-parameter track model that accounts for underlying layers and complex shapes, offering greater flexibility than traditional 3-parameter parabolic models.
5. Top-2 Sampling Strategy: A probabilistic assignment mechanism that allows a single data point to contribute to multiple tracks, effectively resolving ambiguities in overlapping regions.

4. Results and Performance

• Accuracy: The method successfully separates complex ionograms containing multiple reflections, sporadic E-layers, and horizontal inhomogeneities. It outperforms non-physical clustering methods (like standard GMsDB) in cases where tracks intersect.
• Speed: On a 32-core processor, the average processing time is 3.7 minutes per ionogram. The algorithm typically converges within 66 EM iterations.
• Scalability: The execution time scales linearly with the number of clusters and quadratically with the maximum search range for $T$. The authors suggest limiting the search range ($T \le 18$) and iterations ($\approx 55$) for real-time applications.
• Limitations:
  • The method currently processes only the O-component; X-component removal is heuristic and can introduce artifacts if not perfectly separated.
  • It relies on the assumption that tracks can be approximated by the specific 6-parameter model; extremely complex or non-parabolic structures may require further model refinement.
  • The stochastic nature of the EM algorithm can lead to slight variations in results for the same input.

5. Significance

This paper presents a significant advancement in ionospheric diagnostics by moving beyond simple, fixed-parameter models.

• Disturbed Ionosphere Analysis: It enables the automated analysis of ionograms in disturbed conditions (e.g., during geomagnetic storms) where traditional methods fail due to the complexity and unpredictability of track structures.
• Data-Driven Insights: By automatically separating tracks, the method allows for the statistical analysis of layer characteristics (e.g., sporadic E frequency, F-layer hops) without manual intervention.
• Foundation for Inversion: The separated tracks provide a robust input for subsequent electron density profile inversion algorithms, potentially allowing for the reconstruction of profiles in complex, multi-layered ionospheres that were previously uninterpretable.

In summary, the proposed algorithm bridges the gap between physical modeling and data-driven clustering, offering a robust, adaptive tool for modern ionospheric research.