🤖 machine learning

An Information-Theoretic Framework for Feature Construction in Out-of-Distribution Detection

This paper proposes a novel information-theoretic framework for constructing out-of-distribution detection features by optimizing a loss functional that combines KL divergence for distribution separation and the Information Bottleneck principle for compression, resulting in a new, superior shaping function that outperforms existing methods across diverse benchmarks.

Original authors: Sudeepta Mondal (Mary), Xinyi (Mary), Xie, Alex Wong, Ganesh Sundaramoorthi

Published 2026-08-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Sudeepta Mondal (Mary), Xinyi (Mary), Xie, Alex Wong, Ganesh Sundaramoorthi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to recognize different types of animals. You show it thousands of pictures of cats, dogs, and birds, and it learns to say, "That's a cat!" with great confidence. But what happens if you suddenly show the robot a picture of a toaster or a cloud? Since the robot has never seen these things, it might still confidently shout, "That's a cat!" because it's trying to force the new image into a category it already knows. This is a big problem for AI safety. We need a way for the robot to say, "Wait, I don't know what this is," instead of guessing wrong. This is the challenge of "Out-of-Distribution" (OOD) detection: figuring out when data is totally new and different from what the AI was trained on.

To solve this, scientists often look at the "features" the AI uses to make decisions. Think of these features as the robot's internal notes or sketches it makes of the image before giving an answer. Some researchers have tried to tweak these notes using fixed rules, like "if a number is too big, cut it down." But these rules are often just guesses based on what worked in specific tests, and they might fail when the robot sees something truly strange. The big question is: can we find a mathematical rule that tells us exactly how to adjust these notes for any kind of new data, not just the specific examples we've seen before?

This paper introduces a new way to think about that problem using a mix of math and information theory. The authors propose a theory that treats the adjustment of these internal notes not as a fixed rule, but as a flexible, random process. They use a mathematical "loss function" (a scorecard for how well the AI is doing) that tries to do two things at once: first, it pushes the notes for new, strange data as far away as possible from the notes for familiar data; and second, it uses a concept called the "Information Bottleneck" to make sure the notes don't get too messy or lose the important details needed to spot the weird stuff.

By running simulations with different assumptions about what "strange" data looks like (like assuming it follows a bell curve or has heavy tails), the authors found that the best way to adjust the notes changes depending on the type of weirdness. For example, if the strange data is very noisy, the best adjustment looks like "clipping" (cutting off extreme values), which explains why some existing methods work. If the strange data has rare, extreme outliers, the best adjustment looks like "zeroing out" small values.

Based on these findings, the authors suggest a new, flexible tool called a "Piecewise Linear Function" (PLF). You can think of this as a smart, adjustable filter that can bend and shape the AI's internal notes in just the right way to spot new data, no matter what kind of new data it is. When they tested this new filter on a wide variety of standard tests, including those with tricky image distortions and completely new categories, it performed better than almost every other method currently available. In fact, in a massive comparison of 21 different detection methods, their new approach ranked in the top three across all the different tests, suggesting it is a much more reliable and general solution for keeping AI honest when it encounters the unknown.

=== DRAFT ===
Technical Summary: An Information-Theoretic Framework for Feature Construction in Out-of-Distribution Detection

Problem Statement
Machine learning systems often fail when encountering out-of-distribution (OOD) data, producing high-confidence incorrect predictions. While numerous post-hoc OOD detection methods exist, they are largely empirically driven, rule-based approaches (e.g., ReAct, FS-OPT, VRA, ASH) that lack a unified theoretical foundation. Consequently, these methods often struggle to generalize across different datasets and types of distributional shifts (semantic and covariate). There is a need for a theoretical framework that can derive OOD features based on underlying statistical assumptions, explain the properties of existing methods, and systematically generate new, more robust features.

Methodology
The authors propose a novel information-theoretic framework that formulates OOD feature construction as a variational optimization problem.

Random Feature Formulation: Unlike existing deterministic feature shaping functions, the authors model OOD features ( $\tilde{Z}$ ) as random variables derived from network features ( $Z$ ) via a conditional distribution $p(\tilde{z}|z)$ . The mean of this distribution corresponds to the deterministic shaping functions found in literature.
Loss Functional: The optimization seeks a conditional distribution that minimizes a composite loss functional $L$ $L$ :
- KL Divergence Term: Maximizes the separation between the resulting ID and OOD feature distributions using the symmetrized Kullback-Leibler (KL) divergence.
- Information Bottleneck (IB) Regularization: Minimizes the mutual information between the input feature $Z$ and the OOD feature $\tilde{Z}$ while maximizing the mutual information between $\tilde{Z}$ and the OOD label $Y$ . This ensures the feature is compressed yet retains information specifically relevant to OOD detection, filtering out extraneous variations.
Optimization: Since the problem is defined over an infinite-dimensional space of probability measures, the authors employ the calculus of variations to derive gradients. They assume element-wise independence and Gaussianity for the conditional distribution $p(\tilde{z}|z) \sim \mathcal{N}(\mu(z), \sigma_c(z))$ , allowing for a computationally feasible gradient descent algorithm to solve for the optimal mean $\mu(z)$ and standard deviation $\sigma_c(z)$ .
Distributional Analysis: The framework analyzes optimal features under various assumed OOD distributions (Gaussian, Laplacian, and a proposed Inverse Gaussian variant). The study reveals that different distributional assumptions lead to distinct shaping behaviors:
- Gaussian OOD: Yields features with positive slopes for small values and negative slopes (or clipping) for large values, depending on the IB regularization weight.
- Heavy-tailed OOD (Laplacian/Inverse Gaussian): Yields features that suppress small values (zeroing them out) and clip large values, with linear regions for intermediate values.
Proposed Shaping Function (PLF): Motivated by the observation that optimal shapes across different distributions can be approximated by piecewise linear functions, the authors introduce a Piecewise Linear Function (PLF). This 7-parameter function is parameterized using quantiles of the in-distribution feature magnitudes to ensure adaptability across architectures and datasets.

Key Contributions

Theoretical Framework: Introduction of a variational, information-theoretic loss functional that derives OOD features as random variables, unifying the separation of distributions with information compression.
Theoretical Explanation of Existing Methods: The framework demonstrates that existing state-of-the-art (SoA) rule-based methods (e.g., ReAct, VRA, ASH) implicitly correspond to specific underlying OOD distributional assumptions and regularization strengths. For instance, clipping is linked to high IB regularization, while suppression of small values is linked to heavy-tailed OOD assumptions.
New Shaping Function (PLF): Derivation of a new, theoretically motivated piecewise linear shaping function that generalizes the properties of existing methods without committing to a single distributional assumption a priori.
Quantile-Based Parameterization: A novel formulation for the PLF breakpoints using empirical quantiles, allowing the method to generalize across different feature ranges without increasing parameter count.

Results
The authors validate the PLF method against SoA post-hoc OOD detectors on extensive benchmarks:

Semantic Shifts: On the OpenOOD benchmark (covering CIFAR-10, CIFAR-100, ImageNet-200, and ImageNet-1K), PLF ranks within the top three of all 21 evaluated methods across all datasets and architectures. It achieves the best overall ranking on the challenging ImageNet-1K benchmark and consistently outperforms other feature-shaping methods (Energy, ReAct, VRA-P, FS-OPT).
Covariate Shifts: On the BROAD dataset and OpenOOD covariate shift benchmarks (including ImageNet-C, ImageNet-R, ImageNet-V2, and adversarial attacks like AutoAttack and PGD), PLF achieves the highest average AUROC among feature-shaping methods and ranks second overall among all 21 methods. It demonstrates robustness across diverse shifts where other methods often specialize in only a subset. Notably, this work presents the first comprehensive benchmarking of post-hoc OOD detectors on covariate shifts specifically within the OpenOOD framework.
Efficiency: The inference cost is comparable to existing piecewise linear methods (microseconds per image on an A100 GPU).

Significance
The paper claims to provide the first general theoretical framework for constructing OOD features based on statistical distributions. Its significance lies in moving beyond empirical rule-based heuristics to a principled approach that explains why existing methods work and how to construct better ones. By demonstrating that a single, theoretically derived function (PLF) can outperform specialized methods across both semantic and covariate shifts, the work suggests that a unified, distribution-agnostic feature shaping strategy is achievable. The authors note that this is the first comprehensive benchmarking of post-hoc OOD detectors on covariate shifts within the OpenOOD framework, highlighting the method's broad generalization capabilities. Future work is identified as exploring the full random feature distribution (not just the mean), integrating score functions into the theory, and extending to general vector shaping functions.

More like this