🤖 AI

A Robust Out-of-Distribution Detection Framework via Synergistic Smoothing

This paper introduces ROSS, a robust post-hoc out-of-distribution detection framework that leverages median smoothing to quantify local score instability, thereby achieving state-of-the-art symmetric robustness against both score-minimizing and score-maximizing adversarial attacks.

Original authors: Maria Stoica, Abdelrahman Hekal, Alessio Lomuscio

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Maria Stoica, Abdelrahman Hekal, Alessio Lomuscio

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart security guard (an AI) standing at the door of a club. This guard is excellent at recognizing regular members (In-Distribution data) and usually knows when to let them in. However, the guard has a dangerous flaw: if a stranger (Out-of-Distribution data) puts on a convincing disguise or tricks the guard with a specific type of noise, the guard might confidently let them in, thinking they are a member. This is a safety risk.

The paper introduces a new system called ROSS (Robust OOD Detector via Synergistic Smoothing) to fix this. Here is how it works, using simple analogies:

1. The Problem: The "Fickle" Guard

Current security guards (AI detectors) are often "directional."

Some guards are trained to get nervous if someone looks too perfect. If a trickster makes the stranger look slightly more "member-like," the guard gets confused and lets them in.
Other guards are trained to get nervous if someone looks too messy. If a trickster makes the stranger look slightly more "stranger-like," the guard gets confused.
The Flaw: These guards are vulnerable to specific types of tricks. If you know how to trick Guard A, you can bypass the security.

2. The Solution: The "Stability Test"

ROSS changes the game. Instead of just looking at the stranger once, ROSS asks the guard to look at the stranger 25 times while slightly shaking the camera or adding a little bit of static noise to the image each time.

Think of it like this:

The Real Member (In-Distribution): If you take a photo of a real member 25 times with a shaky camera, they still look like the same person. Their face is stable. The guard's confidence doesn't waver much.
The Imposter (Out-of-Distribution): If you take a photo of an imposter 25 times with a shaky camera, the image might look like a cat in one shot, a rock in another, and a blur in the next. The guard's opinion swings wildly. They are unstable.

3. The Two-Step Check

ROSS uses two specific tools to make its decision:

A. The "Median" (The Middle Ground)
Instead of taking the average of the guard's 25 opinions (which can be skewed by one weird outlier), ROSS takes the middle opinion. This is like asking a crowd of people for a guess and ignoring the loudest, most extreme voices. This gives a "smoothed" score that is harder to trick.

B. The "Instability Meter" (The MAD)
ROSS measures how much the guard's opinion jumped around during those 25 shakes.

Low Jumping (Stable): "Okay, this looks like a member, and it looked like a member every time I shook the camera." -> High Trust.
High Jumping (Unstable): "This looked like a member once, but a dog the next time." -> Low Trust.

4. The "Confidence Gate" (The Safety Net)

Here is the clever part. The paper notes that sometimes, a total stranger might look very stable (e.g., a picture of a solid gray wall). If the guard just looked at stability, they might think, "Wow, that wall is very consistent, let them in!"

ROSS prevents this by adding a Confidence Gate:

It only gives a "Stability Bonus" if the stranger already looks like a member with high confidence.
If the stranger looks like a member but is unstable, they get rejected.
If the stranger looks like a member and is stable, they get a "super-boost" to their score.
If the stranger looks like a stranger, they get rejected regardless of stability.

5. Why It's "Symmetric" (The Best Part)

Previous methods were like a lock that only works if you push the key in one direction. If you pulled it the other way, it broke.

ROSS is a "Symmetric" lock. Because it doesn't care about pushing the score up or down, but rather looks at how wobbly the score is, it works against attackers who try to make the AI think the stranger is a member, AND against attackers who try to make the AI think a member is a stranger.

Summary of Results

The authors tested this on standard image datasets (like CIFAR-10 and ImageNet). They found that:

ROSS is much harder to trick than previous methods.
It improved the system's ability to spot fakes by up to 40% compared to older methods when under attack.
It works as a "plug-in" upgrade. You don't need to retrain the whole AI; you just add this "stability check" layer on top of existing systems.

In short, ROSS makes AI security guards less likely to be fooled by disguises by checking if the guard's confidence holds up when the world gets a little shaky.

Technical Summary: A Robust Out-of-Distribution Detection Framework via Synergistic Smoothing

Problem Statement

Deep Neural Networks (DNNs) are increasingly deployed in safety-critical applications, yet they remain vulnerable to adversarial perturbations and distribution shifts. While Out-of-Distribution (OOD) detection methods serve as a crucial safeguard to identify novel or malicious inputs, recent studies indicate that state-of-the-art OOD detectors themselves are susceptible to adversarial attacks. Specifically, an adversary can craft inputs that are OOD but are misclassified as in-distribution (ID), effectively disabling the system's first line of defense.

Existing perturbation-based detectors, such as ODIN and PRO, attempt to improve detection by applying directional optimizations (e.g., increasing or decreasing confidence scores). However, the authors identify a critical flaw in these approaches: they exhibit asymmetric vulnerability. A detector optimized to resist score-minimizing attacks often becomes vulnerable to score-maximizing attacks, and vice versa. This directional bias undermines the reliability of OOD detection in dynamic, adversarial environments.

Methodology: The ROSS Framework

The authors propose ROSS (Robust OOD Detector via Synergistic Smoothing), a model-agnostic, post-hoc framework designed to achieve symmetric robustness against both score-minimizing and score-maximizing attacks. ROSS does not require retraining the underlying DNN and can be applied to any existing OOD scoring technique (e.g., MSP, Energy, GEN, fDBD).

The core of ROSS involves generating $N$ noisy variants of an input $x$ by adding random perturbations. From the resulting "score stack," ROSS computes two robust statistics grounded in robust statistics theory:

Median Score ( $S_{med}$ ): The median of the score stack, representing a smoothed central tendency of the OOD score.
Score Stability ( $\sigma_{med}$ ): The Median Absolute Deviation (MAD) of the score stack, quantifying the local instability of the score landscape.

Key Insight: Instability as a Signal

The authors observe that ID samples typically reside in smooth regions of the score landscape, yielding low-variance scores under perturbation (high stability). Conversely, OOD samples inhabit irregular regions where small perturbations induce large score fluctuations (high instability).

The Scoring Function

ROSS combines these metrics into a final score ( $S_{ROSS}$ ) using a confidence-aware gating mechanism. The final score is defined as:
$S_{ROSS} = \min(S_{95}, S_{med}) + \Delta_{score} \cdot \left(1 + \frac{\lambda}{\sigma_{med}}\right)$
Where:

$S_{95}$ is a confidence threshold (the 5th percentile of the ID validation distribution).
$\Delta_{score} = \max(0, S_{med} - S_{95})$ ensures the stability bonus is applied only when the median score exceeds the confidence threshold.
$\lambda$ is a hyperparameter controlling the magnitude of the stability bonus.

This formulation ensures that stability is rewarded only for high-confidence, ID-like samples. It prevents "far-OOD" samples, which might be stable but have low confidence, from receiving an artificial score boost. By using the median and MAD, the method is inherently resistant to gradient-based manipulation and outliers.

Key Contributions

Identification of Asymmetric Vulnerability: The paper is the first to systematically analyze and demonstrate that existing perturbation-based OOD detectors (like PRO and ODIN) suffer from asymmetric robustness, defending against one attack direction while remaining vulnerable to the opposite.
Proposal of ROSS: A novel, post-hoc framework that leverages local score instability as a confidence multiplier. Unlike prior work, ROSS is non-directional and gradient-free, achieving symmetric robustness against both PGD-min and PGD-max attacks.
Decomposition of Stability and Confidence: Through extensive ablation studies, the authors isolate the contributions of score stability and confidence thresholding, demonstrating that both are critical for effective ID/OOD separation.
Empirical Validation: Comprehensive evaluation across CIFAR-10, CIFAR-100, and ImageNet, showing that ROSS consistently outperforms baseline detectors and prior perturbation-based methods.

Experimental Results

The authors evaluated ROSS on standard benchmarks using pre-trained ResNet-18 and ResNet-50 models.

Baseline Performance: On clean data, ROSS achieves state-of-the-art performance. For instance, on CIFAR-10 with the GEN base score, ROSS achieved an average AUROC of 84.72% and an FPR95 of 45.77%, outperforming the median-only baseline ( $S_{med}$ ) and the naive ratio ( $S_{med}/\sigma_{med}$ ).
Adversarial Robustness: Under PGD attacks ( $\epsilon = 8/255$ $ϵ = 8/255$ ), ROSS demonstrates significant resilience.
- On CIFAR-10, while the MSP baseline's AUROC collapsed to 3.76% under PGD-max attacks, ROSS-MSP maintained an AUROC of 52.65%.
- On CIFAR-100, ROSS-fDBD improved the AUROC from 11.02% (baseline) to 44.72% under PGD-max attacks.
Symmetric Defense: Unlike PRO and ODIN, which degrade significantly when attacked from the "wrong" direction, ROSS maintains consistent performance against both PGD-min and PGD-max attacks. The authors report improvements of up to 40 AUROC points over prior perturbation-based methods under strong attacks.
Efficiency: Despite requiring $N=25$ forward passes, the parallelizable nature of the inference on modern GPUs results in only a 1.2x latency increase compared to a single forward pass, making it practical for deployment.

Significance and Claims

The paper claims that ROSS provides a practical, lightweight, and easily deployable first-line defense for OOD detection. Its primary significance lies in shifting the paradigm from directional optimization (which creates asymmetric weaknesses) to synergistic smoothing (which leverages statistical stability).

The authors assert that ROSS successfully balances clean accuracy and adversarial robustness without the computational cost of adversarial retraining. By integrating seamlessly with existing OOD scoring approaches, it enhances the trustworthiness of automated systems in open-world environments where unknown or malicious inputs are inevitable. The work suggests that quantifying local instability is a more robust proxy for OOD detection than simply attempting to push scores in a specific direction.

Future work, as noted by the authors, aims to extend this framework beyond gradient-based attacks to encompass distributional shifts induced by generative models and real-world corruptions.