Spatially honest validation of machine-learning prospectivity maps for karst bauxite in Kazakhstan

Technical Summary: Spatially Honest Validation of Machine-Learning Prospectivity Maps for Karst Bauxite in Kazakhstan

Problem Statement
Mineral prospectivity mapping (MPM) increasingly relies on machine learning (ML) to integrate geological, geochemical, and remote sensing data. However, a critical gap exists in the validation of these models. Standard random cross-validation (CV) often yields deceptively high performance metrics (e.g., ROC-AUC > 0.95) because it fails to account for the spatial clustering of mineral deposits and the inherent autocorrelation of raster data. In such designs, test pixels are often spatially adjacent to training pixels, allowing models to "memorize" local textures rather than learning transferable mineralization criteria. This leads to "spatial optimism," where models perform well on known terrain but fail when applied to genuinely unseen, clustered deposit systems. This issue is particularly acute for karst bauxite, which occurs in sparse, elongated, and discontinuous lenses controlled by complex paleotopography and cover, making the distinction between apparent pixel-level discrimination and transferable deposit-scale skill essential.

Methodology
The study proposes a "split-first, audit-first" workflow designed to separate apparent discrimination from transferable skill. The authors mapped karst bauxite prospectivity across ~6,500 km² in Kazakhstan using Sentinel-2 surface reflectance (2019–2024), nine spectral indices, and three terrain derivatives (elevation, slope, aspect) on a 30 m grid. To protect sensitive exploration data, absolute coordinates were withheld; analyses were conducted in a locally shifted UTM system preserving topology and distances.

The core innovation is a four-rung validation ladder applied to an operational Random Forest model (700 trees), alongside Logistic Regression, XGBoost, and LightGBM baselines:

1. Random Cross-Validation: Serves as a baseline for apparent pixel-level discrimination.
2. Spatial-Block Cross-Validation: Separates training and testing data geographically (default 5 km blocks) to mitigate spatial leakage.
3. k-Fold Nearest-Neighbour Distance Matching (kNNDM): Aligns the distance distribution between training/test folds with the actual prediction setting to test realism.
4. Leave-One-Deposit-Cluster-Out (LOCO-CV): The strictest test, where entire clusters of known deposits (defined via DBSCAN) and their surrounding background are withheld during training to evaluate the model's ability to find new deposits.

Additional methodological components include:

• Positive-Unlabelled (PU) Sensitivity: Acknowledging that "background" pixels are unverified terrain rather than confirmed barren ground.
• Conformal Prediction: Used to quantify asymmetric uncertainty, distinguishing between reliable background rejection and heterogeneous bauxite confirmation.
• Area-of-Applicability (AOA): A mask identifying regions where predictor combinations fall outside the training distribution (extrapolation).
• Residual Autocorrelation Analysis: Using variograms and Moran's I to determine the optimal block size for validation (approx. 20 km).

Key Results
The validation ladder revealed a substantial gap between apparent and transferable skill:

• Random CV: The model achieved an ROC-AUC of 0.984 and PR-AUC of 0.941, reflecting the "optimism" of spatial leakage.
• Spatial-Block CV: Performance dropped to ROC-AUC ≈ 0.84 (5 km blocks). As block size increased toward the residual autocorrelation range (~20 km), performance stabilized at ROC-AUC ≈ 0.78.
• Strict Prediction-Oriented Schemes: Both kNNDM and LOCO-CV converged to ROC-AUC ≈ 0.77–0.78, with LOCO-CV yielding a PR-AUC of 0.507.
• Deposit-Level Skill: Bootstrapping across 17 held-out deposit clusters showed a mean ROC-AUC of 0.72 (95% CI: 0.64–0.80), indicating that while the model learns a real signal, it is heterogeneous across different deposit systems.

Uncertainty and Applicability

• Conformal Prediction: Showed asymmetric uncertainty. The model reliably rejected background-like terrain (90% coverage) but struggled to confirm bauxite-like terrain in unseen clusters (41% coverage).
• AOA Screening: Of 69 high-score candidate targets (>0.1 km²) identified in the top 1% of the prospectivity surface, 20 were flagged as extrapolative (outside the AOA), meaning their predictor combinations were poorly represented in the training data.
• Predictor Importance: Elevation was the dominant predictor, followed by moisture/SWIR and vegetation indices. The authors interpret this as the model capturing surface expression and landscape position (residual relief, preservation) rather than direct subsurface mineralization.

Significance and Claims
The paper does not claim to have produced a perfect classifier or a new algorithm. Instead, its primary contribution is an auditable validation and deployment protocol that rigorously separates apparent pixel-level discrimination from transferable deposit-scale skill.

The authors argue that for mineral systems characterized by sparse, clustered labels and unverified backgrounds, reporting a single cross-validation score is misleading. The study demonstrates that "honest" validation (LOCO-CV and spatial blocking) reduces performance estimates by approximately 20 percentage points compared to random CV. Consequently, the resulting prospectivity map should not be interpreted as a probability-of-discovery surface. Instead, it is a calibrated, uncertainty-aware, and applicability-screened exploration triage product. It serves as a decision-support tool to rank targets for field verification, explicitly flagging where the model is extrapolating and where uncertainty is high, thereby preventing overconfidence in exploration targeting.