💻 computer science

How Stable Is a PNT Resilience Score? Decision-Instability of Single-Number Resilience Ratings under Framework-Aligned Weighting

This paper demonstrates that single-number PNT resilience ratings are decision-unstable and susceptible to gaming, as re-weighting framework dimensions frequently alters rankings and declared techniques can artificially inflate scores, thereby arguing for the use of traceable per-dimension sub-scores and rank ranges instead of composite maturity levels.

Original authors: Chakshu Baweja

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chakshu Baweja

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to buy a car, and every dealership gives you a single "Safety Score" out of 100 to tell you how safe the vehicle is. You see a score of 95 and think, "Great, that's the safest one!" But what if that score changes to a 40 just because the person grading it decided to care more about "brakes" than "airbags"? Or what if a car that looks like a tank on paper (because the owner checked every safety box) actually falls apart in a real crash, while a car with fewer checkboxes survives?

This paper is about Positioning, Navigation, and Timing (PNT) systems—think of them as the GPS clocks that guide our phones, ships, and power grids. The authors are asking: Is a single "Resilience Score" actually a reliable way to make decisions?

Here is the breakdown of their findings using simple analogies:

1. The Problem: The "One-Number" Trap

Right now, when experts check how "resilient" (tough and recoverable) a GPS system is, they usually give it a single number or a "Maturity Level" (like Level 1 to 4).

The Flaw: To get that one number, you have to combine many different factors (like how well it detects fake signals, how fast it recovers, and how many backup sources it has).
The Analogy: Imagine grading a student. If you weigh "Math" 90% and "Art" 10%, the math genius wins. If you weigh "Art" 90% and "Math" 10%, the artist wins. The paper shows that for GPS systems, changing the weights slightly can completely flip the winner. A system that looks like the "best" choice today might look like the "worst" choice tomorrow, just because the grader changed their mind about what matters most.

2. The "Paper Tiger" vs. The Real Deal

The authors built a simulation to test this. They created a "Paper Tiger" system: a simple, weak GPS receiver that simply checks every single box on the safety checklist (claiming to have all the advanced features).

The Result: Because the scoring system rewards "checking the boxes" (declaring you have a feature) rather than "proving you have it," the Paper Tiger got a high score.
The Reality: When they actually simulated a jamming attack (like a hacker blocking the signal), the Paper Tiger failed miserably. Meanwhile, a genuinely tough system that didn't check every box but actually had diverse, independent backups survived.
The Lesson: You can "game" the system by just claiming you have the features. A single score hides the difference between saying you are tough and being tough.

3. The "Fake Redundancy" Illusion

Imagine you have four backup generators. You think, "Wow, I have four times the power!"

The Reality: If all four generators are in the same building and the power grid goes down, they all fail at the exact same time. You don't have four backups; you have one backup that is just four times bigger.
The Paper's Finding: Many GPS systems look like they have huge redundancy (four different signals), but if they all rely on the same satellite network, a single jammer can knock them all out. The paper shows that once you account for this "shared failure," the "redundancy" score drops from 4 down to 1. A single score often misses this critical detail.

4. The "Context" Problem (The Weather Analogy)

The authors found that a system's "Level" (its grade) changes depending on the threat it faces.

The Analogy: A raincoat is a "Level 5" item in a storm, but a "Level 0" item in a desert.
The Finding: A GPS system might get a high grade when there is no jamming (because it recovers quickly from minor glitches), but drop to a low grade when a heavy jamming attack happens.
The Issue: If you just say "This system is Level 3," you are lying because it's only Level 3 in this specific weather. In a different storm, it might be Level 0.

The Conclusion: Stop the Single Number

The authors argue that giving a single number (like "Score: 85") is dangerous because it is unstable and can be faked.

Instead, they propose a new way of reporting:

Don't give one number. Give a report card with separate scores for each skill (e.g., "Detection: Good," "Recovery: Bad," "Diversity: Excellent").
Show the range. Instead of saying "Rank #1," say "This system could be Rank #1 or Rank #5 depending on what you value most."
Be honest about the source. Clearly label which scores are based on real tests and which are just claims made by the manufacturer.

In short: A single "Resilience Score" is like a weather forecast that only says "It's nice out." It's safe when the sun is shining, but useless when a storm hits. To make real decisions, you need to see the full forecast, not just the summary.

Technical Summary: Decision-Instability of Single-Number PNT Resilience Ratings

Problem Statement
Current Positioning, Navigation, and Timing (PNT) resilience frameworks—specifically the DHS Resilient PNT Conformance Framework (RPCF) v2.0, the Resist-Detect-Respond-Recover (RDRR) model, and Yang's criteria—define resilience conceptually but lack a quantitative measurement engine. They rely on self-attestation, checklists, or qualitative maturity levels (0–4). This creates a gap where system owners declare conformance, but the resulting single-number scores or maturity levels are neither reproducible nor evidence-based. The paper identifies a specific hazard in collapsing multi-dimensional, physically trade-offed realities (e.g., precision vs. robustness) into a single composite score or level, noting that such collapses are sensitive to arbitrary weighting choices and can be gamed by declaration rather than measured behavior.

Methodology
The authors constructed an open, deterministic scoring engine aligned with RPCF v2.0, RDRR, and Yang's criteria to generate a "missing measurement layer."

Architecture Panel: Seven distinct PNT architectures were modeled, ranging from single-band GNSS and multi-band GNSS to GNSS-inertial hybrids, a spoof-hardened variant, a "paper-tiger" single-band receiver (which declares all seven RPCF techniques without the underlying capability), and a genuinely diverse architecture (GNSS, inertial, clock, eLoran).
Threat Ensemble: Five threat scenarios were simulated: nominal, wideband jamming, spoofing, meaconing, and a combined attack.
Scoring Mechanism:
- Sub-scores: Each architecture was reduced to behavior summaries (holdover, availability, detector AUC, integrity, bounded-degradation flag). These drove sub-scores for the seven RPCF categories. Notably, procedural categories (Obfuscate, Limit, Isolate) were driven by declared presence, while functional categories (Verify, Diversify, Mitigate, Recover) were driven by measured simulation outcomes.
- Composite Score: A weighted mean of the seven RPCF sub-scores.
- Maturity Level: An operationalized "weakest-link" ladder (minimum sub-score) with a bounded-degradation gate, distinct from the qualitative RPCF definition.
Stability Analysis: The study sampled dimension weightings from a symmetric Dirichlet distribution (varying concentration $\alpha$ from 1 to 20) and evaluated the stability of the top-ranked architecture and the assigned maturity level. Metrics included top-1 flip rates, Kendall's tau for rank agreement, and rank ranges. Robustness was tested via $\pm 20\%$ perturbations of all behavior drivers.

Key Contributions

Measurement Engine: An open, deterministic engine that maps simulated PNT behavior to traceable, per-dimension sub-scores with provenance (validated vs. modelled).
Stability Quantification: A rigorous quantification of decision-instability for single-number PNT resilience ratings, applying composite-indicator sensitivity analysis to the PNT domain.
Reporting Protocol: A proposal to replace single scores with per-dimension sub-scores, rank ranges, and scenario-specific levels.

Results
The study yields three primary findings regarding the stability of resilience ratings:

Weighting Instability (H1a): The stability of a single composite winner depends heavily on the weighting prior.
- Under active denial (jamming, spoofing), the genuinely diverse architecture dominates regardless of weighting (stable in ~99% of draws).
- Under nominal conditions (no denial), re-weighting alone flips the winner in 22% of draws. In this regime, a checklist-compliant "paper-tiger" receiver competes with robust systems, and the winner depends entirely on which dimensions are weighted.
- The instability is concentrated in the "contested middle" of the ranking; the top and bottom architectures remain relatively stable.
Level Threat-Dependence (H1b): A single maturity Level is not a property of the architecture but of the assumed threat.
- Using the "weakest-link" operationalization, the same diverse architecture was rated Level 3 under jamming/spoofing, Level 2 under meaconing, and Level 0 under nominal conditions (where no recovery is demonstrated).
- This instability is invariant to weighting; it arises because the minimum-over-categories rule is driven by the threat scenario, not the aggregation method.
Declaration vs. Measurement (H2): Self-attestation can be gamed.
- A "paper-tiger" receiver declaring all seven RPCF techniques scored 0.69 (second place) under nominal conditions, outscoring a genuinely resilient GNSS-inertial system (0.34).
- However, under denial scenarios, the paper-tiger failed completely (Level 0, unbounded degradation), while the diverse system maintained high resilience. This demonstrates that composite scores rewarding declared techniques can mask a total lack of measured resilience.
Apparent vs. Effective Redundancy (H3): Apparent redundancy is illusory under common-mode failure.
- A quad-GNSS array with four independent-looking sources collapsed to an effective diversity of 1.0 when assigned to a single GNSS common-mode failure domain.
- In contrast, the genuinely diverse architecture retained an effective diversity of 3.9. The scoring engine successfully separated raw source count from effective diversity based on failure domains.

Significance and Claims
The paper argues that a single-number PNT resilience score is a "phantom" metric that is safe only when one design clearly dominates (a scenario where the metric is unnecessary) but is unstable and misleading in contested cases where decision-makers actually need guidance.

The authors claim that:

Single numbers are insufficient: They obscure critical trade-offs, threat-dependencies, and the gap between declared posture and measured behavior.
Reporting discipline is required: Stakeholders should report per-dimension sub-scores with provenance (validated vs. modelled), rank ranges across defensible weightings, and maturity levels specific to threat scenarios.
Honesty over Certification: The proposed engine is a "simulation-derived self-assessment," not a certification. Its value lies in its transparency (open source, oracle-checked, versioned) and its refusal to claim authority, providing an "honest account of uncertainty" for skeptical buyers and insurers.

The conclusions hold under $\pm 20\%$ perturbations of model drivers, though the authors note that robustness to the functional form and specific decision thresholds (e.g., detection AUC cutoffs) remains an open question. The work aligns with but does not replace existing frameworks like RPCF v2.0 or IEEE P1952, offering a measurement layer to support them.