⚡ electrical engineering

Anticipating the Optimism Gap: Predicting Distribution-Shift Degradation of RF-Impairment Detectors from In-Distribution Statistics

This paper demonstrates that the performance degradation of GNSS impairment detectors under distribution shifts can be predicted solely from in-distribution statistics, revealing that the "optimism gap" is driven by the number of observables used rather than the learning method, a finding validated across both synthetic benchmarks and real-world field data.

Original authors: Chakshu Baweja

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chakshu Baweja

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Practice Field" vs. The "Real Game"

Imagine you are a coach trying to pick a new player for your team. You look at their stats from a practice game. They scored a perfect 10/10. You think, "Great! This player is a star."

But then, you send them to the real game, where the wind is blowing, the lights are dim, and the other team is playing dirty. Suddenly, that same player only scores a 4/10.

This is the "Optimism Gap." In the world of GPS security (specifically detecting jamming or spoofing), engineers build detectors to spot bad signals. They test these detectors on clean, perfect data (the practice field) and get a high score. But when they take those detectors out into the real world (the real game), their performance often crashes. The problem is, nobody knows how much the score will drop until it's too late, because real-world data is hard to get.

What This Paper Did: The "Simulated Storm"

The authors built a simulated testbed (a digital playground) to study this gap. Instead of waiting for real bad weather, they created a "tunable storm."

The Setup: They created a digital GPS environment with 13 different "detectives" (algorithms) trying to spot four types of trouble: jamming, time spoofing, position spoofing, and multipath (signals bouncing off buildings).
The Twist: They started with perfect conditions (In-Distribution). Then, they slowly made the conditions worse (Distribution Shift), making the bad signals harder to see, just like turning down the volume on a radio until it's static.
The Goal: They wanted to see if they could predict before the storm hit exactly how much each detective's score would drop, using only the stats from the calm, sunny day.

The Four Big Discoveries

1. The Gap Grows with the Storm
As the conditions got worse (the "severity" of the interference increased), the drop in performance got bigger. This seems obvious, but they proved it happens consistently across almost all detectors.

2. It's About "Eyes," Not "Brain"
This is a crucial finding. The authors compared two types of detectives:

The "Physics" Detective: Uses a simple, pre-programmed rule (e.g., "If signal drops, it's bad").
The "Learning" Detective: Uses AI to learn the pattern from data.

They found that learning doesn't save you. If a simple rule-based detective and a complex AI detective both look at only one piece of evidence (like just the signal strength), they both fail by the exact same amount when the storm hits.

The Winner: The detectives that looked at five different clues at once (signal strength, power levels, timing, etc.) held up much better.
The Lesson: It's not about being "smart" (AI); it's about having more eyes (using more data sources). A narrow view is fragile; a wide view is sturdy.

3. Some Bad Guys Are Harder to Spot Than Others
Not all types of interference are equally tricky.

Position Spoofing: Hard to detect, big drop in performance.
Time Spoofing (Matched Power): This is the "ninja" attack. It's designed to look exactly like a normal signal. Surprisingly, this had the smallest drop in performance. Why? Because even in the perfect practice conditions, it was already hard to spot. There was no "optimism" to lose. The detectors that looked amazing in practice were the ones that crashed the hardest in the real world.

4. The Crystal Ball (The Main Result)
The most exciting part: The authors built a simple "crystal ball" model.

How it works: They fed the model only the stats from the calm, sunny day (the In-Distribution data). They didn't show it any storm data.
What it did: The model successfully predicted how much each detective's score would drop when the storm hit.
The Magic: It worked even for detectives it had never seen before, and for types of attacks it had never seen before.
The Secret Sauce: The model looked at the "shape" of the data. If a detective had a high score in calm weather but the "good" and "bad" signals were very mixed up (overlapping), the model knew: "This detective is lucky right now, but they are going to crash hard when things get tough."

Did It Work in the Real World?

The authors were honest: their main results were from a simulation. To check if it held up in reality, they tested their "crystal ball" on three real-world datasets (real GPS recordings from universities and campaigns).

The Result: The mechanism survived! The model correctly predicted the direction of the drop (that performance would get worse) and the ranking of which detectors would fail most.
The Catch: The drop wasn't as huge as in the simulation. Real-world conditions are messy and uncontrolled, so the "crystal ball" wasn't as precise as in the perfect digital lab. But the core idea—that you can predict the drop using only calm-day stats—held true.

The Takeaway

If you are building a GPS security system, don't just trust the high score you get in the lab.

Check the "Shape": If your detector looks great in the lab but the data is messy and overlapping, it's a ticking time bomb.
Use More Clues: A detector that looks at five different things will survive a storm much better than one that looks at just one thing, even if the single-thing detector uses fancy AI.
Predict the Drop: You don't need to wait for a disaster to know your system is fragile. You can use simple math on your current data to forecast how much it will fail when conditions change.

In short: A high score in perfect conditions is often a trap. This paper gives you a way to see the trap before you walk into it.

Technical Summary: Anticipating the Optimism Gap in RF-Impairment Detectors

Problem Statement
In Global Navigation Satellite System (GNSS) security, detectors for radio-frequency impairments (jamming, spoofing, multipath) are typically evaluated using a single Area Under the ROC Curve (AUC) metric derived from in-distribution (ID) data. Practitioners understand that this ID performance often overestimates real-world effectiveness when operating conditions shift (e.g., changes in signal severity or environment). However, the magnitude of this "optimism gap"—the difference between ID AUC and out-of-distribution (OOD) AUC—is rarely predictable in advance due to the scarcity of labeled field data. Existing machine learning literature on predicting OOD accuracy focuses on fixed tasks and models across input distributions, often predicting classification accuracy rather than AUC, and rarely addressing transfer across different detector architectures or impairment classes.

Methodology
The authors address this gap using a controlled, parameter-grounded synthetic testbed built upon the open-source Kshana PNT-resilience simulator.

Testbed Construction: The corpus consists of 13 detectors evaluated across four impairment classes (jamming, time spoofing, position spoofing, multipath) and a nominal class. Impairment severity is controlled by a scalar scale $(0, 1]$ , where $1.0$ represents the in-distribution state and lower values represent increasingly subtle impairments.
Detector Panel: The study evaluates five physics-based baselines (single-observable detectors and a combiner), five full-feature learned detectors (logistic regression and multilayer perceptrons of varying widths), and three single-feature learned controls.
Metric Definition: The "optimism gap" ( $\Delta$ ) is defined as the difference between the ID AUC and the AUC at a shifted severity level: $\Delta(d, c, s) = \text{AUC}_c(d, 1.0) - \text{AUC}_c(d, s)$ .
Prediction Framework: The core hypothesis is that the gap can be predicted using only ID statistics. The authors extract six ID-only features (including ID AUC, score separation $d'$ , distribution overlap, variance ratio, tail margin, and detection rate at a fixed false-alarm rate) and a seventh ablatable feature (self-perturbation slope). A ridge regression model is trained to predict the realized gap for held-out detectors and held-out impairment classes.
Validation: The study employs leave-one-detector-out and leave-one-class-out cross-validation, significance testing against a 2000-fold label-permutation null, and ablation studies to ensure the prediction is not a definitional artifact of the ID AUC feature.

Key Findings

Monotonic Growth of the Gap: The optimism gap grows monotonically as the distribution shift deepens (severity scale decreases). The mean Spearman correlation between shift magnitude and the gap is 0.50 across detector-class pairs.
Evidence Breadth vs. Learning: The gap is driven by the breadth of evidence a detector integrates, not by whether the detector is learned. Single-feature learned detectors degrade by the same amount as their matched physics baselines. Conversely, detectors combining all five observables degrade significantly less. This suggests that narrow detectors are inherently fragile to shift, regardless of their training method.
Class Heterogeneity: The gap varies systematically by impairment class. The "matched-power" time spoofing case (the hardest to separate even in ID) exhibits the smallest gap, while classes with high ID performance (like position spoofing) show the largest drops.
Predictability of the Gap: A simple ridge model built solely from ID statistics successfully predicts the gap for detectors and impairment classes it has never seen.
- Cross-Detector Prediction: $R^2 = 0.47$ (RMSE = 0.028).
- Cross-Class Prediction: $R^2 = 0.46$ (RMSE = 0.029).
- Both results are statistically significant ( $p < 0.001$ ) against label permutation and survive the removal of the ID AUC feature, confirming that the shape of the score distribution (specifically overlap and tail margin) carries predictive power.

Real-Data Validation
The authors pre-registered a protocol to test these findings on three open field corpora: Yunnan University captures, the Jammertest 2024 campaign, and SatGrid.

Jammertest 2024: The cross-detector prediction held on real data ( $R^2 = 0.11, p = 0.009$ ), though the cross-class prediction failed ( $R^2 = -0.25$ ), likely due to the small number of classes and mechanistic differences.
SatGrid: This dataset provided a calibrated severity axis (spoofer power sweep). It confirmed that ID AUC overstates performance at higher severities, with gaps reaching up to 0.22. The direction of the gap was perfectly rank-correlated with ID AUC ( $\rho = 1.0$ ), though the quantitative predictor was underpowered due to the limited number of detectors and classes.
Conclusion on Real Data: The mechanism survives contact with real data but at a smaller magnitude than in simulation, consistent with uncontrolled field conditions.

Significance and Claims
The paper claims to provide the first demonstration that the evaluation optimism gap in RF-impaired detection can be predicted for unseen detectors and unseen impairment classes using only in-distribution statistics.

Mechanism over Magnitude: The authors explicitly state that their synthetic testbed isolates the mechanism of degradation rather than predicting exact field magnitudes.
Practical Warning: The central practical takeaway is that a high ID AUC achieved on a "thin, heavily overlapping score margin" is a signature of a detector that will degrade significantly under shift. This warning can be read before deployment without requiring OOD labels.
Reproducibility: The study releases the full testbed, software-receiver front ends for converting raw IQ to correlator features, ingest adapters for real data, and a pre-registered protocol, ensuring all claims are falsifiable on real data.

The work does not claim to solve the field-data scarcity problem entirely but offers a method to quantify and anticipate the risk of optimism in detector selection, shifting the focus from "what is the best ID score?" to "how robust is this score likely to be?"