📊 statistics

Moving beyond spatial and random cross-validation in environmental modelling: a call for prediction-domain adaptive evaluation

This paper advocates for "prediction-domain adaptive evaluation" as a new category of cross-validation methods to provide more reliable accuracy estimates for spatial environmental models in real-world scenarios that fall between the extremes of random and spatial data distributions.

Original authors: Jan Linnenbrink, Jakub Nowosad, Hanna Meyer

Published 2026-05-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jan Linnenbrink, Jakub Nowosad, Hanna Meyer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: How Do We Know Our Maps Are Good?

Imagine you are a cartographer trying to draw a map of where a specific type of tree grows in a forest. You have a bunch of data points (samples) showing where you found these trees. You feed this data into a computer program (a machine learning model) to teach it what the trees look like. Once the computer learns, it draws a map predicting where the trees are everywhere else.

But here is the problem: How do you know if your map is actually accurate?

To check the map, you need to test it. But you can't use the same data you used to teach the computer, or the computer will just "cheat" and give you a perfect score because it already knows the answers. You need to hold back some data to test it with. This is called Cross-Validation.

The paper argues that the way scientists currently split their data to test these maps is often wrong, leading to maps that look great on paper but fail in the real world.

The Three Ways to Test the Map

The paper discusses three main ways to split your data to test the model. Think of these as three different ways to take a driving test.

1. Random Cross-Validation (The "Pop Quiz" Approach)

How it works: You take your data points and throw them in a hat, pulling out random ones to be the "test" questions.
The Flaw: In the real world, nature isn't random. If you found a tree in one spot, there's a good chance there's another tree right next to it. If you randomly pick a test point that is right next to a training point, the computer isn't really being tested on its ability to predict; it's just being tested on its ability to memorize.
The Result: This gives you a fake high score. It's like a student taking a driving test where the instructor asks, "What do you do at a stop sign?" and the student answers correctly because they just saw the sign five seconds ago. They haven't proven they can drive in a new neighborhood.

2. Spatial Cross-Validation (The "Hard Mode" Approach)

How it works: To fix the cheating problem, scientists started separating the test points far away from the training points. They make sure the test data is in a completely different part of the map.
The Flaw: While this stops the cheating, it creates a new problem. If your training data is scattered all over the map, but you force the test data to be in a totally different, far-away region, you are testing the computer on a situation it will never face.
The Result: This gives you a fake low score. It's like testing a driver who has only driven in sunny California by suddenly dropping them into a blizzard in Alaska. They will fail the test, not because they are bad drivers, but because the test was unfair.

3. Prediction-Domain Adaptive Evaluation (The "Real-World Simulation" Approach)

The New Idea: The authors propose a third way. Instead of forcing the test to be random or forcing it to be far away, you should mimic the actual situation where the map will be used.
How it works: You look at how the data is actually distributed in the real world.
- If your real-world data is scattered randomly, your test should be scattered randomly.
- If your real-world data is clumped together in groups (leaving gaps), your test should also be clumped, leaving similar gaps.
- If you are trying to predict a brand new area you've never seen, your test should reflect that difficulty.
The Result: This gives you a realistic score. It's like giving the driver a test that matches the exact road conditions they will face tomorrow. If they drive well on the test, you can trust them on the road.

The "Extrapolation Continuum" (The Spectrum of Difficulty)

The paper argues that we shouldn't just think in black and white (Interpolation vs. Extrapolation). Instead, think of it as a slider.

Left side (Easy): You have data everywhere. Predicting the middle is easy.
Right side (Hard): You have data in one corner, and you are trying to predict a totally different continent. This is very hard.
The Middle: Most real-world problems fall somewhere in between. You might have data in a few dense clusters with big empty spaces in between.

The paper says: Your testing method must slide along with the difficulty.

If the job is easy, use a "Random" style test.
If the job is hard, use a "Spatial" style test.
If the job is in the middle, use the new Adaptive method that matches the specific gaps in your data.

The "Area of Applicability" (The Safety Zone)

There is one final warning in the paper. Even with the best testing method, you cannot trust your map everywhere.

Imagine you trained your computer only on trees in a tropical rainforest. If you ask it to predict trees in a desert, it will guess, but it will be guessing blindly. The paper calls the safe zone where the computer has actually seen similar data the "Area of Applicability."

The Lesson: You should only trust the accuracy numbers for the parts of the map that look like the data you used to train the model. If you try to predict outside that zone, the accuracy numbers are meaningless, no matter how good your testing method was.

Summary of the Authors' Main Points

Random testing is too easy for clustered data (it overestimates accuracy).
Spatial testing is often too hard for random data (it underestimates accuracy).
The solution is "Adaptive Testing": Match your testing method to the specific pattern of your data. If your data has gaps, your test should have gaps.
Know your limits: Only trust the map's accuracy in the areas where the model was actually trained. Don't trust it in completely new territories.

In short: Stop using a "one-size-fits-all" test. Tailor your evaluation to the specific challenges of your map to get a true picture of how good it really is.

Technical Summary: Moving Beyond Spatial and Random Cross-Validation in Environmental Modelling

1. Problem Statement

The application of spatial predictive modeling in ecology and geosciences faces a critical challenge: how to appropriately evaluate map accuracy when a formal probability sample of the prediction area is unavailable. While consensus exists that independent probability sampling is the gold standard for accuracy estimation, such samples are often infeasible with large, opportunistic ecological databases (e.g., GBIF, sPlotOpen).

Consequently, researchers rely on data splitting strategies, primarily random cross-validation (CV) and spatial cross-validation. However, a significant debate persists regarding which method yields unbiased estimates:

Random CV assumes data are independent and identically distributed (iid). Critics argue this ignores spatial autocorrelation, leading to data leakage and overly optimistic accuracy estimates when training points are clustered.
Spatial CV (e.g., block CV) enforces spatial independence between training and test folds. Critics argue this creates artificial extrapolation scenarios for randomly distributed data, leading to overly pessimistic error estimates.

The core problem is that the literature often treats "interpolation" (training and prediction areas overlap) and "extrapolation" (predicting to new areas) as binary opposites. In practice, most ecological datasets fall on a continuum between these extremes, characterized by varying degrees of sampling bias and clustering. Existing methods fail to flexibly adapt to this continuum, resulting in unreliable accuracy estimates that do not reflect the actual prediction difficulty.

2. Methodology

To address this gap, the authors reproduce and extend a simulation study originally presented by Wadoux et al. (2021), focusing on the prediction of forest above-ground biomass (AGB) in the Amazon basin using Random Forest models and 28 environmental predictors.

Simulation Design

The study generated four distinct sampling designs to represent the extrapolation continuum:

Random: Training points distributed randomly across the prediction area.
Weakly Clustered: Moderate clustering of training points.
Strongly Clustered: High density of training points in specific areas, leaving large gaps.
Extrapolation: Training points in one region, prediction area in a completely distinct geographic region.

Evaluation Strategies Compared

The study compared three cross-validation strategies against a design-based inference baseline (using independent random test samples):

Random CV: Standard random splitting.
Spatial CV: Splitting using spatial blocks to enforce distance between folds.
Prediction-Domain Adaptive CV: A new category of methods implemented using k-Nearest Neighbour Distance Matching (kNNDM). This approach explicitly aims to match the distribution of nearest-neighbor distances between training and test folds to the distribution of distances between the prediction domain and the training data.

Quantitative Metrics

Wasserstein Statistic ( $W$ ): Used to quantify the degree of extrapolation by measuring the difference between the density of spatial nearest-neighbor distances within the training set versus the distance between prediction locations and the training set.
Root Mean Square Error (RMSE): The primary metric for evaluating accuracy. The study calculated the difference between the estimated RMSE (from CV) and the true RMSE (calculated against simulated truth across the whole area).
Area of Applicability (AoA): A dissimilarity index used to define the geographic extent where the model was trained and validated, limiting accuracy estimates to regions where the prediction difficulty resembles the training data.

3. Key Contributions

Conceptual: The Extrapolation Continuum

The paper argues against a binary view of interpolation vs. extrapolation. Instead, it proposes that prediction scenarios exist on a continuum defined by the density of spatial nearest-neighbor distances. The difficulty of prediction increases as the gap between training and prediction locations widens, quantified by the Wasserstein statistic.

Methodological: Prediction-Domain Adaptive Evaluation

The authors introduce prediction-domain adaptive evaluation as a new category of cross-validation. Unlike random or spatial CV, which enforce fixed rules (randomness or independence), adaptive methods (specifically kNNDM) flexibly adjust the resampling strategy to resemble the prediction situation.

In random sampling scenarios, it mimics random CV (small distances).
In clustered or extrapolation scenarios, it mimics spatial CV (large distances).
The goal is not to enforce independence per se, but to ensure the "prediction difficulty" in the validation folds matches the difficulty in the actual prediction domain.

Empirical: Resolving the Bias Dilemma

The study demonstrates that:

Random CV consistently overestimates accuracy (underestimates error) in clustered and extrapolation scenarios due to data leakage.
Spatial CV consistently underestimates accuracy (overestimates error) in random or weakly clustered scenarios because it forces extrapolation where none exists.
Prediction-Domain Adaptive CV provides the most reliable estimates across the continuum, consistently matching the true RMSE in random, weakly clustered, and strongly clustered scenarios.

Limitation of Global Accuracy

The paper highlights that even the best CV method fails in pure extrapolation scenarios (predicting to a completely new region). In these cases, CV estimates remain overly optimistic because the model encounters predictor values outside its training distribution. The authors emphasize that accuracy estimates are only valid within the Area of Applicability (AoA), where the prediction domain resembles the training domain.

4. Results

The simulation results (visualized in Figure 2 of the paper) show:

Random CV: Produced negative bias (underestimation of error) for clustered and extrapolation designs.
Spatial CV: Produced positive bias (overestimation of error) for random and weakly clustered designs.
Prediction-Domain Adaptive CV: Provided estimates closest to zero difference (perfect estimation) for random, weakly clustered, and strongly clustered designs.
Extrapolation Scenario: Even adaptive CV struggled to estimate true error accurately for the whole area, confirming that performance statistics do not generalize to regions fundamentally different from the training data. However, when restricting the evaluation to the Area of Applicability, the estimates improved significantly.

5. Significance and Claims

The authors position this work as a consolidation of the ongoing debate between proponents of random and spatial cross-validation. They claim:

Unbiased Estimation: Only a probability sample provides a truly unbiased estimate of map accuracy; all CV methods are proxies.
Contextual Suitability: There is no single "best" CV method. Random CV is suitable for random data; spatial methods are suitable for extrapolation. However, most real-world cases fall in between.
Adaptive Solution: Prediction-domain adaptive evaluation offers a unified framework that adapts to the specific prediction context (the extrapolation continuum), yielding more realistic accuracy estimates than rigid random or spatial approaches.
Scope of Validity: Accuracy estimates derived from CV are only meaningful for new prediction situations that resemble the validation folds in terms of prediction difficulty. Therefore, predictions and their associated accuracy assessments should be restricted to the Area of Applicability.

The paper concludes that while cross-validation cannot replace design-based inference, prediction-domain adaptive evaluation provides a balanced, flexible approach to model tuning and accuracy estimation in the absence of probability samples, effectively resolving the trade-off between data leakage and artificial extrapolation.