💻 computer science

The Signal in the Noise: OOD Detection Through Goodness-of-Fit Testing in Factorised Latent Spaces

This paper introduces Signal in the Noise (SITN), a novel out-of-distribution detection method that leverages the properties of continuous normalizing flows to identify atypical noise mappings in factorized latent spaces, thereby overcoming the unreliability of traditional likelihood-based indicators without requiring OOD data or incurring significant computational costs.

Original authors: Philipp Bomatter, Jack Geary, Henry Gouk

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Philipp Bomatter, Jack Geary, Henry Gouk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot that has spent years studying a specific type of art, say, Impressionist paintings. It knows exactly what a "normal" Impressionist painting looks like: the brushstrokes, the colors, the way light hits the canvas.

Now, you show the robot a picture of a modern graffiti wall. The robot needs to decide: "Is this an Impressionist painting (In-Distribution) or is this something completely different (Out-of-Distribution)?"

The Old Problem: The "Confident Mistake"

For a long time, scientists tried to solve this by asking the robot to calculate a "Likelihood Score." This score answers: "How likely is it that this image was created by my training process?"

The problem, as this paper points out, is that the robot often gets tricked.

If you show it a simple, blank white canvas (which is technically an Impressionist painting in the robot's eyes), it might give it a high score because it's "simple" and easy to explain.
If you show it a complex, chaotic graffiti wall, it might give it a low score.
But here's the catch: If you show it a picture of a cat (which is definitely not an Impressionist painting), the robot might accidentally give it a higher score than the blank canvas, simply because the cat has some visual features (like edges or colors) that look "simple" to the robot's math.

The robot is confused by complexity. It thinks, "This looks simple, so it must be real," even when it's actually a fake. This is called Complexity Bias.

The New Solution: "Signal in the Noise" (SITN)

The authors, Philipp Bomatter and his team from the University of Edinburgh, propose a new way to think about the problem. They use a special kind of AI called a Continuous Normalising Flow (CNF).

Think of this AI as a magic translator.

The Training: The AI learns to translate real Impressionist paintings into a specific type of "static noise" (like the static you see on an old TV when there's no signal). It learns that every real painting, no matter how complex, should translate into this specific, random static pattern.
The Reverse: Because the translation is perfect (mathematically speaking), the AI can also translate the static back into a painting.

The "Goodness-of-Fit" Test

Here is the clever part of their new method, SITN:

Instead of asking, "How likely is this painting?", they ask, "If I translate this image into static, does the static look like real TV static?"

Real Painting (In-Distribution): When the AI translates it back to static, the result looks like perfect, random TV static. The pixels are independent of each other, and they follow the rules of randomness.
Fake Image (Out-of-Distribution): When the AI tries to translate a graffiti wall or a cat into static, the result looks weird.
- Maybe the static has a strange pattern (like a checkerboard) that shouldn't be there.
- Maybe the "noise" is too smooth or too bumpy.
- The "static" doesn't fit the rules of randomness.

The paper calls this "Signal in the Noise." Even though the image has been turned into noise, the structure of the original fake image leaves a "signal" behind in the noise that gives it away.

How They Check the Noise

The authors use two simple tests to see if the "noise" is real:

The "Shape" Test: Does the noise look like a perfect bell curve (the standard shape of random noise)? If the noise is too flat or too spiky, it's a fake.
The "Friendship" Test: In real random noise, one pixel shouldn't care about its neighbor. They are total strangers. If the noise shows that pixels are "friends" (correlated) or following a pattern, it's a fake.

If the noise fails either of these tests, the AI says, "This isn't a real painting; it's Out-of-Distribution."

Why This is Better

The paper claims this method is superior for three main reasons:

It ignores complexity: It doesn't care if the image is simple or complex. It only cares if the "noise" looks right. This fixes the problem where the robot was fooled by simple fakes.
It's fast: It doesn't need to run extra, slow calculations or train a second robot to help it decide.
It's strict: It can mathematically guarantee that it won't cry "Fake!" too often when looking at real things. You can set a rule that says, "Only flag 1 in 100 real things as fake," and it will stick to that rule.

The Results

The team tested this on standard image datasets (like CIFAR-10, which has small pictures of cars, animals, and planes).

When they showed the robot pictures of SVHN (street numbers) or CelebA (celebrity faces) as "fakes," the old methods (Likelihood, Typicality) got confused and often called the fakes "real" or called real things "fakes."
SITN consistently spotted the fakes correctly.
They even tested it on images with artificial glitches (like blurring, snow, or motion blur). SITN was the most consistent at spotting these glitches, whereas other methods failed when the glitches changed the "complexity" of the image.

A Small Glitch

The authors are honest about one small weakness. Sometimes, if an image has a huge, solid block of color (like a big blue sky), the AI's translator struggles to turn that block into "random static." It leaves a faint pattern behind. This can sometimes trick the system into thinking a real image is fake. However, the authors note this is a rare edge case and the method works very well overall.

Summary

In short, the paper introduces SITN, a method that stops trying to guess "how likely" an image is. Instead, it translates the image into "noise" and checks if that noise looks like genuine, random static. If the noise has a hidden pattern (the "signal"), the image is a fake. It's a smarter, faster, and more reliable way to catch AI imposters.

Technical Summary: Signal in the Noise (SITN)

Problem Statement

Deep generative models, particularly Continuous Normalising Flows (CNFs), offer a theoretical foundation for Out-of-Distribution (OOD) detection by explicitly modeling data distributions. However, prior research has established that the likelihoods assigned by these models are unreliable indicators for distinguishing in-distribution (ID) from OOD data in high-dimensional spaces. A well-documented failure mode involves models trained on one dataset (e.g., CIFAR-10) assigning higher likelihoods to samples from a different dataset (e.g., MNIST or SVHN).

Existing corrections to this issue, such as typicality tests, complexity penalties, and likelihood ratios, often suffer from inherent biases linked to the likelihood itself (specifically "complexity bias," where simple images are favored) or impose practical constraints. These constraints include the need for domain-specific compression algorithms, access to OOD data for background models, or significant computational overhead (e.g., training ensembles or fine-tuning on test samples).

Methodology: Signal in the Noise (SITN)

The authors propose Signal in the Noise (SITN), a framework that reframes OOD detection as a single-sample goodness-of-fit test within the latent noise space of CNFs.

Core Insight

CNFs utilize diffeomorphic, mass-preserving mappings to transform a standard Gaussian noise prior ( $P_0 = \mathcal{N}(0, I)$ ) into the target data distribution ( $P_{ID}$ ). Because these mappings are reversible, ID data maps back to samples that conform to the standard Gaussian prior. Conversely, OOD samples, which possess low probability mass under the target distribution, map to noise latents that exhibit atypical empirical properties under the Gaussian prior. Crucially, this atypicality is independent of the vector field's divergence, which is the primary driver of likelihood values.

The Framework

SITN exploits the complete factorization of the Gaussian prior. A single high-dimensional latent vector $Z$ (mapped from an input $X$ via the inverse flow $\phi^{-1}$ ) is treated as a collection of $D$ independent and identically distributed (i.i.d.) samples from a univariate standard normal distribution. The method tests $Z$ against two necessary conditions for the null hypothesis ( $H_0$ ) that $Z \sim \mathcal{N}(0, I)$ :

Marginal Normality: Each element of $Z$ $Z$ must follow a standard univariate normal distribution.
- Statistic: The Anderson-Darling (AD) statistic ( $S_{AD}$ ) is used to quantify the discrepancy between the empirical cumulative distribution function (CDF) of the latent elements and the theoretical standard normal CDF.
Dimensional Independence: The elements of $Z$ $Z$ must be mutually independent.
- Statistic: The Coefficient of Variation (CV) of the empirical power spectrum ( $S_{CV}$ ) is used. For white Gaussian noise, the power spectrum follows an exponential distribution, yielding a theoretical CV of 1. Structural correlations (dependence) manifest as dominant frequencies, inflating the spectral variance and driving $S_{CV}$ significantly above 1.

Scoring and Calibration

To produce a single scalar OOD score ( $S_{SITN}$ ), the method employs a max-quantile strategy:
$S_{SITN} = \max(\hat{F}_{AD}(S_{AD}), \hat{F}_{CV}(S_{CV}))$
where $\hat{F}$ represents empirical CDFs estimated from an ID calibration set. This approach maps statistics to a uniform $[0, 1]$ range, allowing for meaningful comparison. By taking the maximum, the method rejects $H_0$ if the sample violates either marginal normality or dimensional independence.

Theoretical Properties

False Positive Control: The method allows for strict control of the Type I error rate (False Positive Rate). By calibrating the empirical CDFs on a held-out ID set, a threshold can be set to guarantee a specific false positive rate without requiring access to OOD data.
Computational Efficiency: The overhead is negligible compared to the inverse flow computation. Calculating $S_{AD}$ involves sorting ( $O(D \log D)$ ), and $S_{CV}$ involves a Fast Fourier Transform ( $O(D \log D)$ ).
Robustness: The framework remains applicable even if the CNF is an approximation of the true distribution, though this may increase Type II errors (missed detections) rather than False Positives.

Key Contributions

New Framework: A theoretically grounded approach casting OOD detection as single-sample goodness-of-fit testing against the factorized Gaussian noise prior of CNFs.
SITN Method: A specific instantiation requiring no OOD data, minimal computational overhead, and providing strict Type I error control.
Empirical Validation: Comprehensive evaluation demonstrating effectiveness across standard benchmarks and synthetic perturbations, highlighting the absence of the complexity bias inherent in likelihood-based methods.

Experimental Results

The authors evaluated SITN against baselines including raw Log-likelihood, Typicality, and Density of States Estimation (DoSE) across three datasets: CIFAR-10, SVHN, and CelebA (downsampled to 32x32).

Cross-Dataset Performance: SITN consistently outperformed all baselines. For example, when training on CIFAR-10 and testing on SVHN, SITN achieved an AUROC of 0.946, compared to 0.105 for Log-likelihood and 0.802 for DoSE. In the reverse setup (SVHN training, CIFAR-10 testing), SITN achieved 0.986 versus 0.980 for Log-likelihood.
Synthetic Perturbations: On CIFAR-10-C (19 corruption types), SITN achieved the highest average AUROC (0.85), significantly outperforming Log-likelihood (0.59), Typicality (0.69), and DoSE (0.78). SITN maintained robust performance even in conditions where other methods dropped to chance levels.
Complexity Bias Analysis: Visualizations and scatter plots revealed that Likelihood, Typicality, and DoSE exhibited strong correlations with image complexity (measured by PNG compression size), often flagging simple, constant-background images as OOD. SITN showed no such bias ( $R^2 = 0.052$ ).
Failure Modes: While highly effective, SITN showed a specific edge case where the UNet architecture struggled to map large, constant background patches to typical Gaussian noise, causing the AD statistic to flag them as atypical even when they were in-distribution.

Significance and Claims

The paper claims that SITN offers a significant practical and statistical advancement over likelihood-centric approaches. By shifting the detection mechanism from the data space (where likelihoods are unreliable) to the noise space (where OOD samples manifest as atypical noise), the method achieves:

Reliability: It avoids the catastrophic failure modes associated with likelihood-based OOD detection.
Practicality: It requires no access to OOD data for training or calibration and incurs minimal computational cost.
Statistical Rigor: It provides a formal mechanism for controlling the false positive rate, a feature often lacking in heuristic OOD detectors.

The authors conclude that while SITN is not a panacea (noting the specific edge case with constant backgrounds), it represents a critical step toward safer deployment of machine learning in high-risk domains by providing robust, theoretically grounded safeguards against out-of-distribution inputs.