📊 statistics

Precision spectral estimation at sub-Hz frequencies: closed-form posteriors and Bayesian noise projection

This paper presents a Bayesian framework for precision spectral estimation in low-frequency regimes where Gaussian approximations fail, deriving closed-form posteriors for cross-spectral quantities and noise projection parameters, and demonstrating its effectiveness in analyzing LISA Pathfinder data to decorrelate temperature-induced acceleration noise.

Original authors: Lorenzo Sala, Stefano Vitale

Published 2026-04-16

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Lorenzo Sala, Stefano Vitale

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Listening to a Whisper in a Storm

Imagine you are trying to listen to a very quiet whisper (a specific signal) in a room that is incredibly noisy (background noise). In physics, specifically for missions like LISA Pathfinder (which tests technology for detecting gravitational waves), scientists need to measure the "noise" of their instruments with extreme precision. They aren't looking for a signal; they are trying to understand the noise itself to make sure their instruments are perfect.

The problem? They can only listen for a limited amount of time. In statistics, to get a clear picture of noise, you usually need to listen for a long time and take thousands of samples, then average them out. This is like taking a photo of a moving car: if you take 1,000 photos and average them, you get a clear, sharp image.

But in space, time is precious. Sometimes, scientists only have one or two "photos" (samples) to work with. If you try to use standard math (which assumes you have thousands of photos) on just one or two samples, the results are garbage. It's like trying to guess the average height of a whole country by measuring just one person. The standard math breaks down, leading to impossible results (like negative noise levels).

This paper provides a new, smarter way to do the math when you have very few samples.

The Core Problem: The "Gaussian" Trap

Standard statistics rely on something called the Central Limit Theorem. Think of this as the "Law of Large Numbers." It says: "If you take enough samples, the average will look like a perfect bell curve."

The Trap: When you have thousands of samples, the bell curve is perfect.
The Reality: When you only have 1 or 2 samples (common in low-frequency space experiments), the data looks nothing like a bell curve. It's jagged, unpredictable, and "skewed."
The Consequence: Using the old "bell curve" math on small data sets gives you false confidence. You might think you know the noise level, but you're actually guessing wildly.

The Solution: A Bayesian "Sherlock Holmes"

The authors propose a Bayesian approach. Instead of assuming the data must fit a bell curve, they use a method that asks: "Given this tiny bit of data I have, and what I know about how noise behaves, what is the most likely truth?"

They treat the problem like a detective solving a case with very little evidence. They don't force the evidence to fit a preconceived theory; they let the evidence shape the theory.

The Magic Tool: The "Wishart" Distribution

In the paper, they use a complex statistical shape called the Wishart distribution.

Analogy: Imagine you are trying to guess the shape of a cloud. Standard math assumes clouds are always perfect spheres. But in reality, clouds are weird shapes. The Wishart distribution is a flexible mold that can stretch and twist to fit the weird, jagged shape of your actual data, even if you only have a tiny piece of the cloud to look at.

The Two Main Tricks

The paper solves two specific problems using this new math:

1. Measuring the Noise Itself (Spectral Estimation)

The Goal: Figure out exactly how loud the background noise is at a specific frequency.
The Old Way: Average your samples. If you only have one sample, you just guess.
The New Way: They use a "Jeffreys Prior." Think of this as a neutral referee. It doesn't assume the noise is loud or quiet; it just says, "I know nothing about the magnitude, so I'll treat all possibilities equally until the data speaks."
The Result: Even with just one sample, they can calculate a "confidence interval" (a range where the truth likely lies) that is mathematically honest. It admits, "I'm not 100% sure, but here is the most probable range," rather than pretending to be precise when they aren't.

2. Noise Projection (The "Noise Cancellation" Trick)

The Goal: Sometimes, the noise isn't random. It's caused by something else. For example, in the LISA mission, temperature changes make the test masses wiggle, creating "fake" acceleration noise.
The Setup: You have the main sensor (measuring acceleration) and a thermometer (measuring temperature). You suspect the thermometer is the culprit.
The Old Way: Try to subtract the temperature noise. But with few samples, the subtraction is messy and often makes things worse.
The New Way: They use a technique called Schur Complement (don't worry about the name).
- Analogy: Imagine you are trying to hear a singer in a band. You have a microphone on the singer and a microphone on the drummer. You want to hear only the singer.
- The authors' method calculates exactly how much of the singer's sound is actually just the drummer's sound leaking in. It mathematically "subtracts" the drummer's contribution to reveal the pure singer.
- Crucially, it does this while accounting for the fact that you only have a few seconds of audio. It tells you: "We are 95% sure the temperature caused 80% of the noise, and here is the exact range of error."

Why This Matters for Space

The LISA Pathfinder mission was a testbed for future gravitational wave detectors. These detectors need to be incredibly sensitive—measuring movements smaller than the width of an atom.

The Challenge: At very low frequencies (like the slow rumble of a distant engine), you can't wait long enough to get thousands of data samples.
The Breakthrough: This paper proved that even with just a handful of samples, you can still accurately separate the "temperature noise" from the "real physics."
The Result: They successfully "decorrelated" (cleaned up) the data, proving that the instrument was working perfectly and that the remaining noise was indeed the tiny, real gravitational effects they were looking for.

Summary in One Sentence

This paper gives scientists a new, robust mathematical toolkit that allows them to accurately measure and clean up extremely faint signals in space, even when they only have a tiny amount of data to work with, by replacing rigid "bell curve" assumptions with flexible, honest probability models.

1. Problem Statement

The paper addresses the challenge of estimating cross-spectral quantities (Power Spectral Densities - PSD, Cross-Power Spectral Densities - CPSD, and coherences) in the low-frequency regime (sub-Hz to $\mu$ Hz).

The Limitation of Standard Methods: Conventional spectral analysis (e.g., Welch's method) relies on averaging a large number of periodograms ( $M$ ) to invoke the Central Limit Theorem, thereby justifying Gaussian statistics for uncertainty estimation.
The Low-Frequency Constraint: In precision experiments like LISA Pathfinder and the future LISA mission, observation times are long, but the frequency resolution required forces the number of available independent periodograms ( $M$ ) to be very small (sometimes $M=1$ or $M=2$ ).
Consequences: When $M$ $M$ is small, the Gaussian approximation fails. This leads to:
- Paradoxical results (e.g., non-negligible probabilities for negative PSD values).
- Inapplicable frequentist confidence intervals.
- Significant bias in estimators like coherence.
The Goal: Develop a rigorous Bayesian framework that provides closed-form posterior distributions for spectral quantities and noise projection parameters, valid even when $M$ is small and far from the Gaussian limit.

2. Methodology

The authors develop a Bayesian inference method based on Complex Wishart statistics.

A. Statistical Foundation

Data Model: The input is a multivariate, stationary, zero-mean Gaussian time series.
Likelihood: The periodogram matrix $\Pi$ $Π$ (averaged over $M$ $M$ segments) follows a Complex Wishart distribution ($CW$) conditional on the true CPSD matrix $\Sigma$ $Σ$ .
- $W = M\Pi \sim CW(\Sigma, M)$ .
Priors: The authors derive appropriate non-informative priors to ensure unbiased estimation:
- Single PSD ( $p=1$ ): The Jeffreys prior ( $p(S) \propto 1/S$ ) is identified as the only unbiased choice that remains defined for $M=1$ .
- Full CPSD Matrix ( $p>1$ ): A specific prior $p(\Sigma) \propto |\Sigma|^{-2p+1}$ (equivalent to $p(Q) \propto |Q|^{-1}$ for the inverse matrix) is recommended to avoid bias that scales with the number of channels $p$ .
- Coherence: Flat priors are shown to be effective approximations for coherence estimation.

B. Closed-Form Posteriors

The paper derives analytical expressions for the posterior distributions of key quantities:

PSD: The posterior follows an Inverse Gamma distribution ( $S|\Pi \sim \text{inv}\Gamma(M, M\Pi)$ ).
CPSD Matrix: The posterior follows a Complex Inverse Wishart distribution.
Coherence (MSC & Multiple Coherence): Closed-form posteriors involving Gauss hypergeometric functions ( ${}_2F_1$ ) are derived for magnitude-squared coherence and multiple coherence.
Noise Projection (Decorrelation):
- Model: A "main" signal $x(t)$ is modeled as a linear combination of filtered "disturbance" signals $y_i(t)$ plus a residual background $x_0(t)$ : $x(t) = x_0(t) + \sum \alpha_i * y_i(t)$ .
- Susceptibilities ( $\alpha$ ): The coupling coefficients (transfer functions) are inferred.
- Residual PSD ( $S_{x_0x_0}$ ): The power of the decorrelated noise is estimated.
- Results: The joint posterior for the residual PSD and susceptibilities is derived. The marginal posterior for the residual PSD is an Inverse Gamma distribution, while the susceptibilities follow a Complex Multivariate Student-t distribution.

C. Handling Readout Noise

The paper analyzes the impact of readout noise in the disturbance sensors. It demonstrates that while the method remains unbiased for the Schur complement ( $1/\Sigma^{-1}_{11}$ ), significant readout noise in the disturbance channels leads to an overestimation of the residual PSD and an underestimation of susceptibility magnitude.

3. Key Contributions

Closed-Form Bayesian Posteriors: The paper provides explicit, closed-form probability density functions (PDFs) for spectral quantities (PSD, CPSD, Coherence) and noise projection parameters, eliminating the need for computationally expensive Monte Carlo sampling for standard cases.
Validity at Low $M$ : The methodology is rigorously defined for $M \ge 1$ , solving the breakdown of Gaussian statistics in low-frequency regimes where only a few periodograms are available.
Unbiased Estimation: By selecting specific non-informative priors (Jeffreys and its multivariate generalizations), the authors eliminate the bias inherent in standard estimators when $M$ is small.
Noise Projection Framework: A complete Bayesian treatment for "noise hunting" (decorrelation), providing joint posteriors for both the coupling coefficients (susceptibilities) and the residual noise spectrum.
Real-World Validation: Application to LISA Pathfinder data, successfully decorrelating temperature-induced acceleration noise.

4. Results

Simulations:
- Bias Analysis: Simulations confirm that the proposed Bayesian estimators are unbiased, whereas frequentist Gaussian-based methods significantly underestimate uncertainty and miss true values more often than predicted (especially for $M < 50$ ).
- Decorrelation: In simulated multivariate time series, the method successfully retrieved the true residual PSD (even when orders of magnitude smaller than the disturbance) and the true complex susceptibilities.
- Frequency-Independent Susceptibilities: A Markov Chain Monte Carlo (MCMC) approach was developed for cases where susceptibilities are real and frequency-independent, allowing the combination of data across frequencies to improve estimates even at very low frequencies where $M < p$ .
LISA Pathfinder Application:
- Applied to 18.5 days of data (sampling 0.1s).
- Successfully decorrelated temperature-induced forces from the acceleration noise.
- Achieved a reliable estimation of the coupling coefficient (susceptibility), showing the imaginary part is consistent with zero and precisely measuring the real part.
- Demonstrated effective noise reduction at $\mu$ Hz frequencies where temperature noise dominates.

5. Significance

Enabling Precision Gravitational Physics: This work is critical for space-based gravitational wave observatories (like LISA) where the signal of interest lies in the sub-Hz regime. Standard spectral analysis tools are insufficient for the required precision at these frequencies.
Robust Uncertainty Quantification: It provides a mathematically rigorous way to quantify uncertainties in spectral estimates without relying on the "large $M$ " assumption, preventing false positives/negatives in noise characterization.
General Applicability: While motivated by LISA, the method is applicable to any field requiring high-precision spectral analysis of multivariate time series with limited data segments (e.g., geophysics, medical imaging, or other precision metrology).
Software Implementation: The derived closed-form posteriors (summarized in Table I of the paper) are designed to be easily implemented in standard data analysis software, facilitating immediate adoption by the scientific community.

In summary, Sala and Vitale provide a necessary statistical toolkit for the next generation of precision experiments, replacing approximate Gaussian methods with exact Bayesian inference tailored for the data-scarce, low-frequency regime.