Rejections Based on Predictive Uncertainty Enable Reliable Routine Soil Spectroscopy

Technical Summary: Rejections Based on Predictive Uncertainty Enable Reliable Routine Soil Spectroscopy

Problem Statement
Conventional soil analysis relies on elaborate, costly, and time-consuming laboratory methods involving physical and chemical processing. While highly accurate, these constraints limit sampling density and monitoring frequency, hindering precision agriculture. Visible–near–infrared spectroscopy (VNIRS) paired with machine learning offers a rapid, low-cost alternative. However, its routine adoption is hindered by skepticism regarding reliability; predictive accuracy varies significantly across soil properties and individual samples. For instance, while organic matter and clay fractions are predicted with high accuracy, essential nutrients like potassium (K) and phosphorus (P) often lack distinct spectral signals, leading to unreliable predictions. Current discourse often presents a false dilemma between the high cost of conventional methods and the perceived unreliability of VNIRS, forcing a binary choice rather than a nuanced integration.

Methodology: The "Reject-to-Remeasure" Framework
The authors propose a decision-theoretic framework termed reject-to-remeasure, which operationalizes predictive uncertainty to balance cost and reliability. The workflow proceeds as follows:

1. Initial Measurement: Soil samples are analyzed via VNIRS.
2. Probabilistic Prediction: A probabilistic machine learning model generates a predictive distribution for the target soil property, rather than a single point estimate.
3. Uncertainty-Guided Rejection: A "rejector" function evaluates the predictive uncertainty against user-defined constraints. Specifically, it calculates the probability that the absolute prediction error exceeds a threshold $\delta(Y)$.
   • If $Pr(|Y - \hat{y}(x)| \ge \delta(Y) | x) \le \alpha$ (where $\alpha$ is a risk tolerance, e.g., 0.05), the prediction is accepted.
   • If the probability exceeds $\alpha$, the prediction is rejected, and the sample is sent for conventional laboratory remeasurement.
4. Model Selection: The study benchmarks four probabilistic models: TabPFN (a transformer-based foundation model), TabICL (an in-context learning foundation model), Quantile Regression Forests (QRF), and Partial Least Squares Quantile Regression (PLSQR).

Key Contributions

• Framework Formalization: The paper formalizes the integration of probabilistic modeling with explicit rejection options for soil spectroscopy, moving beyond simple outlier detection to conditional reliability based on defined error thresholds.
• Model Benchmarking: The authors demonstrate that modern tabular foundation models (TabPFN and TabICL) outperform traditional methods (QRF, PLSQR) in both point prediction accuracy and, crucially, probabilistic performance (sharpness and reliability of predictive distributions).
• Operational Strategy: The study provides a concrete strategy for integrating VNIRS into routine workflows by accepting high-confidence predictions and reserving conventional analysis for uncertain cases, thereby reducing overall costs without compromising quality constraints.

Results
Evaluated on a regional soil library from Québec (793 samples across 272 sites), the framework yielded the following results:

• Uncertainty Quantification: The rejector successfully distinguished between high-error and low-error samples. For properties with strong spectral signals (Clay and Soil Organic Matter - SOM), acceptance rates were high (72% and 65.8%, respectively). For properties with weak signals (Potassium and Phosphorus), acceptance rates were low (15.1% and 7.1%, respectively), reflecting the inherent difficulty of predicting these nutrients via VNIRS.
• Risk Control: The framework effectively controlled the risk of incorrect acceptances. The frequency of accepted predictions that exceeded the error threshold was reduced to less than the target risk level of 5% ($\alpha = 0.05$).
• Cost Efficiency: Economic analysis suggests that for Clay and SOM, the reject-to-remeasure strategy significantly reduces measurement costs due to high acceptance rates and the large price gap between VNIRS and conventional particle size analysis. Conversely, for K and P, the strategy may not be economically viable in isolation due to high rejection rates and the relatively low cost of conventional nutrient extraction (Mehlich-3). However, the authors note that since a single VNIRS scan predicts multiple properties, the cost of a rejected nutrient prediction can be amortized if other properties from the same scan are accepted.

Significance and Claims
The paper claims that "reject-to-remeasure" resolves the false dilemma between cost efficiency and reliability in soil spectroscopy. By treating spectroscopy as the initial measurement method and reverting to conventional analysis only when predictive uncertainty exceeds acceptable levels, the framework enables the operational integration of VNIRS into routine agricultural and environmental workflows.

The authors emphasize that this approach acknowledges the fundamental limitations of VNIRS for certain chemical properties (as noted by critics like Baveye) but provides a mechanism to navigate these limitations. The study concludes that leveraging modern foundation models allows for reliable uncertainty quantification, making it possible to reduce laboratory measurement efforts while strictly adhering to user-defined accuracy requirements. The authors position this as a concrete step toward routine deployment, while noting that future work is needed to evaluate the framework's performance on out-of-distribution data and its potential for self-improving AI through iterative retraining on remeasured samples.