📊 statistics

A Robust Framework for Model Order Selection in Correlated Large-Dimensional CES Noise

This paper proposes a robust two-stage framework for model order selection in large-dimensional correlated non-Gaussian CES noise that combines a Toeplitz-rectified $M$ -estimator for noise whitening with Random Matrix Theory-based subspace rank inference, demonstrating superior performance over existing methods across synthetic and real-world datasets.

Original authors: Eugénie Terreaux, Emmanuelle Jay, Frédéric Pascal, Jean-Philippe Ovarlez

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Eugénie Terreaux, Emmanuelle Jay, Frédéric Pascal, Jean-Philippe Ovarlez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to listen to a few specific musicians playing in a large, noisy concert hall. The problem isn't just that the hall is loud; it's that the noise itself is "sticky" and "bumpy." It echoes in a specific pattern (correlated), and sometimes the volume spikes wildly in unpredictable bursts (non-Gaussian/heavy-tailed noise).

In the world of signal processing, this is the challenge of Model Order Selection: figuring out exactly how many musicians (sources) are actually playing, distinct from the chaotic background noise.

This paper proposes a new, robust "two-stage" framework to solve this problem, specifically when the noise is messy, correlated, and the amount of data is huge. Here is how it works, broken down into simple concepts:

1. The Problem: The "Sticky" Noise

Traditional methods for counting sources often assume the background noise is like white static on an old TV—random, uniform, and easy to ignore. But in real life (like in radar, medical brain scans, or stock markets), noise is often correlated (it has a pattern, like a ripple in a pond) and impulsive (it has sudden, massive spikes).

If you try to count the musicians using old methods in this environment, you get confused. You might think the noise ripples are new musicians, or you might miss the real musicians because the noise is drowning them out.

2. The Solution: A Two-Stage "Cleaning" Process

The authors propose a framework that acts like a two-step cleaning crew to separate the signal from the noise.

Stage 1: Flattening the Floor (Whitening)

Before you can count the musicians, you have to fix the floor. The noise in this scenario is "tilted" and "bumpy."

The Analogy: Imagine trying to hear a whisper while standing on a floor that is sloped and covered in uneven rocks. You can't hear clearly.
The Paper's Fix: The authors use a mathematical tool called a Toeplitz-rectified M-estimator. Think of this as a smart leveler that flattens the floor and removes the rocks. It estimates the shape of the noise and "whitens" the data, effectively making the noise look like uniform, flat static again.
The Innovation: They offer three different "levelers" (estimators) to handle different types of messiness:
1. SCM-based: A standard approach (good for mild mess).
2. Maronna-based: A robust approach that handles heavy spikes well.
3. Tyler-based: A "distribution-free" approach that works even if you have no idea what kind of noise you are dealing with.

Stage 2: The Magic Threshold (Counting the Musicians)

Once the floor is flat, the real musicians stand out clearly against the background.

The Analogy: Now that the floor is level, you can see who is standing on a raised platform (the signal) versus who is standing on the flat ground (the noise).
The Paper's Fix: They use Random Matrix Theory (RMT). This is a branch of math that predicts exactly how high the "noise" will jump just by chance.
The Result: They derived a specific "height limit" (a threshold). Any musician standing above this line is a real source. Any noise that jumps up but stays below the line is just background chatter.
- For the robust methods (Maronna), the line is calculated based on the specific noise type.
- For the universal method (Tyler), the line is a famous mathematical constant known as the Marcenko-Pastur upper edge.

3. Why This Matters (The Results)

The authors tested this framework in four very different "concert halls":

Synthetic Data: Fake noise they created to test the math.
Hyperspectral Images: Images that capture light in hundreds of colors (used in remote sensing).
EEG Brain Scans: Recording electrical activity from the brain, which is notoriously noisy and correlated.
Financial Data: Stock market returns, which often have sudden, massive spikes (crashes or booms).

The Findings:

Old methods (like AIC) failed miserably in these messy environments, often counting hundreds of "sources" when there were only a few, because they couldn't distinguish the noise spikes from real signals.
The new framework successfully identified the correct number of sources in all four scenarios.
Robustness: Even when the noise behaved in ways the math didn't perfectly predict (like unexpected heavy spikes), the "Maronna" and "Tyler" versions of their method kept working, whereas the standard method struggled.

Summary

In short, this paper gives us a new, robust way to count hidden signals in a chaotic world. Instead of ignoring the messy, correlated, and spiky nature of real-world noise, it builds a mathematical "leveler" to flatten the noise first, and then uses a precise mathematical ruler to count only the things that truly stand out. This works for everything from finding hidden objects in radar to understanding brain waves or managing investment portfolios.

Technical Summary: A Robust Framework for Model Order Selection in Correlated Large-Dimensional CES Noise

Problem Statement
The paper addresses the fundamental signal processing challenge of estimating the number of statistically significant sources (model order) in noisy observations. While classical methods like the Akaike Information Criterion (AIC) and subspace techniques perform well in white, Gaussian noise, they become unreliable in modern high-dimensional settings characterized by two specific complexities:

Correlated Noise: The noise possesses a known or unknown correlation structure (specifically modeled here as a Toeplitz-structured scatter matrix).
Non-Gaussianity: The noise follows a Complex Elliptically Symmetric (CES) distribution, which includes heavy-tailed and impulsive behaviors (e.g., K-distributed or Student-t textures) that violate Gaussian assumptions.

In the large-dimensional regime where the observation dimension $m$ and sample size $N$ grow proportionally ( $m/N \to c > 0$ ), the Sample Covariance Matrix (SCM) fails to provide consistent estimates, and standard eigenvalue-based detection thresholds become invalid due to the interaction between correlation and heavy tails.

Methodology
The authors propose a unified, two-stage robust framework applicable to three distinct estimation branches. The core philosophy is to first "whiten" the data to remove the unknown correlation structure and then apply Random Matrix Theory (RMT) to separate signal eigenvalues from the noise bulk.

Stage 1: Robust Whitening via Toeplitz Rectification
The framework estimates the unknown Toeplitz scatter matrix $C$ of the noise and inverts it to whiten the observations. Three specific estimators are developed for this step:
- SCM-based: Uses the Sample Covariance Matrix rectified to a Toeplitz structure.
- Maronna-based: Uses Maronna's M-estimator (a robust parametric approach) rectified to Toeplitz structure.
- Tyler-based: Uses Tyler's M-estimator (a distribution-free approach) rectified to Toeplitz structure.
A key theoretical innovation is the "Toeplitz rectification" operator, which projects the estimated scatter matrix onto the space of Toeplitz matrices. The authors prove that this step asymptotically suppresses the low-rank signal contribution, ensuring the estimator remains consistent even in the presence of sources.
Stage 2: RMT-Based Eigenvalue Thresholding
Once the data is whitened, a robust scatter matrix is estimated from the whitened observations. The model order is determined by counting eigenvalues that exceed a specific theoretical threshold derived from RMT:
- For SCM and Maronna branches: The threshold is an explicit closed-form expression $t = \frac{\Phi_\infty(1 + \sqrt{c})^2}{\xi (1 - c \Phi_\infty)}$ , where $\Phi_\infty$ and $\xi$ depend on the weight function and the texture distribution.
- For the Tyler branch: The threshold simplifies to the upper edge of the Marčenko–Pastur law, $(1 + \sqrt{c})^2$ , making it distribution-free.

Key Contributions
The paper establishes the following theoretical and practical contributions:

Almost Sure Consistency: The authors prove that the Toeplitz-rectified estimators (SCM, Maronna, and Tyler) consistently estimate the true scatter matrix $C$ in the presence of correlated CES noise and sources (Theorems 1, 3, and 5).
Explicit RMT Thresholds: They derive explicit, closed-form eigenvalue thresholds that separate signal from noise components in the proportional growth regime. This includes a novel threshold for the Maronna estimator under correlated noise (Theorems 2 and 4) and the adaptation of the Marčenko–Pastur bound for the Tyler estimator (Theorem 6).
Three-Branch Pipeline: The framework offers a flexible pipeline:
- SCM Branch: A baseline approach.
- Maronna Branch: A robust parametric approach that offers improved finite-sample performance when the texture distribution is known or well-approximated.
- Tyler Branch: A distribution-free approach that requires no prior knowledge of the noise texture distribution, utilizing the universal Marčenko–Pastur threshold.
Data-Driven Parameter Estimation: The paper provides practical algorithms to estimate unknown scaling factors (such as $\xi$ and the texture expectation) directly from the data using leave-one-out approximations, enabling the method's application without prior distributional knowledge.

Results
The methodology was validated through extensive simulations and real-world data applications:

Synthetic Data: Monte Carlo simulations demonstrate that the proposed methods (M1 and M2) significantly outperform unwhitened methods (M3) and AIC, particularly in low Signal-to-Noise Ratio (SNR) regimes and when noise textures deviate from Gaussian assumptions. The Maronna branch showed superior robustness to texture mismatch compared to the SCM branch.
Hyperspectral Imaging: Applied to Indian Pines, SalinasA, PaviaU, and Cars datasets. The proposed methods consistently identified a physically plausible number of sources, whereas AIC and unwhitened methods vastly overestimated the model order (e.g., estimating >200 sources instead of ~16).
EEG Source Detection: On the MAMEM EEG SSVEP dataset ( $m=256, N=500$ ), classical criteria (AIC, MDL, KN) failed, estimating model orders near the dimensionality of the data ( $\approx 255$ ). The robust framework provided stable estimates between 6.7 and 7.8 sources with low variance.
Financial Data: Applied to 100 Fama–French portfolios ( $m=100, N=150$ ). The robust methods identified between 2 and 5 dominant factors, aligning with known financial factor structures, while AIC and MDL again overestimated the factor count significantly.

Significance and Claims
The paper claims that its primary significance lies in providing the first rigorous framework for model order selection that simultaneously handles large dimensionality, unknown correlation structures (Toeplitz), and non-Gaussian heavy-tailed noise without requiring prior knowledge of the noise distribution (in the Tyler branch).

The authors emphasize that existing methods either assume white noise, known correlation, or Gaussianity, rendering them ineffective for practical applications like hyperspectral imaging, EEG analysis, and financial portfolio optimization where these assumptions are routinely violated. By establishing almost sure consistency and deriving explicit thresholds, the work bridges the gap between theoretical Random Matrix Theory and robust practical signal processing, offering a principled tool for factor selection and covariance estimation in challenging high-dimensional regimes.