⚡ electrical engineering

Prior-matched evaluation of operational Earth-observation classifiers: a three-number reporting method demonstrated on Sentinel-1 internal-wave detection

This paper addresses the systematic overestimation of precision in rare-event Earth-observation classifiers caused by reporting at an incorrect class prior, proposing a three-number reporting method and a precision-first development cycle that successfully improved the operational detection of internal waves in Sentinel-1 data from 0.192 to 0.927 precision.

Original authors: Joao Pinelo, Joao Goncalves, Arun Shukla, Adriana Santos-Ferreira

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Joao Pinelo, Joao Goncalves, Arun Shukla, Adriana Santos-Ferreira

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a very busy security checkpoint at an airport. Your job is to find a specific, rare type of suspicious package (let's call them "Wave Packages") hidden among millions of ordinary, harmless boxes.

The Problem: The "Perfect" Test vs. The Real World
In the past, the team trained their security scanner (an AI model) using a special test set. To make the math easy, they gave the scanner an equal number of "Wave Packages" and "Harmless Boxes" to practice on. On this balanced test, the scanner looked like a genius, scoring a 79% success rate.

But in the real world, the situation is very different. For every 20 boxes the scanner checks, only one is actually a "Wave Package." The other 19 are harmless.

Because the scanner was trained expecting a 50/50 split, it got confused by the real world's 5/95 split. It started screaming "SUSPICIOUS!" at almost every harmless box.

The Result: In the real world, the scanner's success rate dropped from a "great" 79% down to a "terrible" 19%.
The Cost: Every time the scanner falsely alarms on a harmless box, a human expert has to stop what they are doing, walk over, and inspect it. This wastes the most valuable resource: human attention.

The Mistake: Trying to Fix the Wrong Thing
The team initially thought the problem was that the scanner needed to be retrained to expect fewer "Wave Packages." They tried adjusting the training data to match the real world's ratio.

The Discovery:
They realized this was like trying to fix a broken thermometer by changing the room temperature. The scanner's ability to tell the difference between a wave and a non-wave was actually fine; the problem was just how they reported the score.

The Analogy: Imagine a metal detector that beeps loudly for gold. If you test it in a room full of gold coins, it looks perfect. If you test it in a room full of sand with only one gold coin, it will beep constantly at the sand. The metal detector didn't break; the context changed. The old way of reporting scores (based on the gold-filled room) was lying about how the detector would perform in the sand.

The Solution: The "Three-Number" Report
Instead of giving just one score, the authors propose a new way to report results using three numbers to tell the full, honest story:

The "Classroom" Score: How the model did on the balanced, easy test (the gold-filled room). This shows the model's raw potential.
The "Real World" Score: How the model does on a test that matches the real ratio (the sand-filled room). This predicts what the human experts will actually face.
The "Live" Score: The actual results after the model is deployed and humans have checked the real alerts.

By showing all three, you can see the gap between the "Classroom" and the "Real World." This gap isn't a mistake in the model; it's a mathematical reality of rare events.

The Fix: Turning the Dial
The team didn't need to rebuild the whole scanner. They just needed to turn a "sensitivity dial" (a threshold).

They made the scanner stricter. It now only screams "SUSPICIOUS!" when it is very sure.
The Trade-off: It misses a few more actual waves (which is okay because the satellite will fly over the same spot again in 12 days to catch them), but it stops screaming at harmless boxes.
The Result: The number of false alarms dropped by 76%. The human experts are no longer overwhelmed.

The "Secret Weapon": Better Eyes, Not Bigger Brains
The team tried making the AI "smarter" by giving it more computing power (adding more layers), but that didn't help much. The real breakthrough came from changing how the AI looks at the image.

The Analogy: Imagine looking at a crowded room. A "lazy" AI (average pooling) looks at the whole room and says, "There's a lot of movement." A "sharp" AI (Generalized Mean pooling) focuses on the loudest movement.
There was a specific type of harmless pattern (like ripples on ice) that looked like a wave. The "lazy" AI got fooled by the average texture. The "sharp" AI learned to ignore the average and focus on the specific peaks, successfully ignoring the fake waves.

The Conclusion
This paper teaches us that for rare events (like finding a needle in a haystack), you cannot trust a single "average" score. You must report how the system performs in the real, skewed environment.

The team proved that:

Honesty is better than hype: Reporting the "Real World" score prevents false hope.
Don't over-engineer: Sometimes, you don't need a bigger, more complex AI; you just need to tune the settings and improve how it looks at the data.
The limit is data, not code: The biggest barrier to making the system even better isn't the computer code; it's having enough verified examples of the rare event to teach the AI what to look for.

In short: They stopped lying about the model's performance by showing the full picture, and they fixed the system by making it stricter and sharper, not by making it bigger.

Technical Summary: Prior-Matched Evaluation of Operational Earth-Observation Classifiers

Problem Statement
Operational Earth-observation classifiers often suffer from a systematic reporting artifact caused by a mismatch between the class prior used during training/evaluation and the true prior encountered in deployment. The Internal Waves Service, which screens the Sentinel-1 Wave-mode archive for internal solitary waves, exemplifies this issue. The service's classifier was trained and evaluated on a balanced dataset (1:1 wave-to-non-wave ratio), a convention inherited from the service's early "recall-first" phase when the operational positive rate was unknown. However, the actual operational stream exhibits a heavily skewed prior of approximately 0.05 (one wave scene in twenty).

This mismatch creates a deceptive evaluation landscape. A model reporting a high balanced-test precision (e.g., 0.794) may deliver a drastically lower precision in real operation (e.g., 0.192) at the same decision threshold. The paper argues that standard metrics like balanced-test precision are "optimistic" and invisible to the systematic error they introduce. Furthermore, common remedies such as prior correction or probability calibration are shown to be ineffective for improving precision at a fixed recall; these methods are monotone transformations of the score that merely relocate the operating threshold without shifting the underlying precision-recall curve. The core problem is identified not as a training defect, but as an evaluation problem requiring a reporting method that matches the deployment prior.

Methodology
The paper proposes a "prior-matched reporting method" based on a three-number axis to characterize an operational classifier, alongside a rigorous development cycle designed to isolate performance gains.

The Three-Number Reporting Axis:
- Balanced-Test Performance: The conventional metric (precision, recall, F1) on a held-out set with equal class proportions. This serves as a cautionary baseline, exposing the gap between conventional reporting and reality.
- Operational-Prior Performance: The model's performance on a frozen test set constructed specifically at the operational positive rate (0.05). This figure is designed to predict the precision a validator will actually face in the field.
- Real Operational Performance: The ground truth obtained post-deployment through expert adjudication of the model's detections on the live stream. This is a forward commitment, pre-registered to close the evaluation loop.
Evaluation Instrument Design:
- Frozen, Leakage-Controlled Dataset: The dataset was pinned before splitting to prevent the growing archive from leaking into evaluation. Spatial leakage is controlled at the "hotspot" level (defined by true image-footprint overlap) rather than by centroid radius, ensuring that near-identical scenes from the same orbit repeat do not straddle the train/dev/test splits.
- Sealed Lockbox: A test set (1,537 positives, 29,203 negatives) is sealed off-site with checksums and read exactly once at a pre-registered threshold.
- Operating Discipline: The service operates under a "precision-first" discipline with recall held at a floor of 0.80. This is justified because missed waves are recoverable via orbit repeats and archive reprocessing, whereas false positives consume scarce expert attention.
Development Cycle:
- Improvements were pursued via single-lever interventions, each isolated and promoted only if it exceeded a pre-registered margin (0.021).
- Levers Tested:
  - Threshold Adjustment: Moving the operating point along the existing curve (immediate relief, but no curve lift).
  - Data Variety: Increasing negative variety (widening the negative class) improved precision, whereas shifting the training ratio toward the operational prior did not (confirming the inertness of prior correction at fixed recall).
  - Capacity: Increasing network width showed diminishing returns; beyond a certain point, added parameters did not clear the promotion margin.
  - Aggregation Head: Replacing global-average pooling with Generalized-Mean (GeM) pooling significantly improved precision by better suppressing specific geophysical confusers (banded surface textures mimicking waves) that average pooling failed to distinguish.

Key Results

The Prior Effect: The incumbent model demonstrated the systematic artifact: a balanced-test precision of 0.794 versus a real operational precision of 0.192.
Model Improvement: The promoted model (using GeM pooling and varied negative data) achieved a precision of 0.927 at the operational prior (recall 0.827) on the sealed lockbox. In contrast, the same model on a balanced set reported a precision of 0.996, illustrating that the balanced figure crosses performance bars (e.g., F1 > 0.90) that the honest operational figure misses.
Out-of-Time Robustness: An independent check on data collected after the dataset freeze showed that while the model's discrimination (AUC) remained stable (0.993 $\to$ 0.987), the fixed operating point decayed. Precision dropped from 0.927 to 0.810, and the false-positive rate tripled, highlighting that a fixed threshold is not a "set-and-forget" quantity and requires re-validation against current data.
Constraint Shift: The development cycle revealed that the binding constraint has moved from model architecture to the size of the reducible false-positive mass. The aggregation lever was spent, and further capacity increases yielded no significant gain, indicating that future improvements require more validated data rather than larger networks.

Significance and Claims
The paper claims that the mismatch between training priors and deployment realities is an evaluation problem, not a training one. Its primary contribution is a prior-matched reporting method that makes the "prior effect" visible rather than hidden. By reporting the contrast between balanced-test, operational-prior, and real operational figures, the method provides an "honest measure" of classifier skill.

The authors emphasize that this approach is not specific to internal waves or Sentinel-1 but is a portable convention for any operational Earth-observation service bootstrapping a rare-event detector under an unknown or shifting prior. The work also establishes a discipline of reproducibility through pinned datasets, leakage-controlled splits, and sealed, single-read evaluations. Finally, the paper concludes that the next cycle of improvement for this service will not come from larger models, but from expanding the validated dataset to address the irreducible false-positive mass, and from validating operating points against current catalogue slices rather than relying on development-set thresholds.

Technical Summary: Prior-Matched Evaluation of Operational Earth-Observation Classifiers

More like this