Technical Summary: Evaluating Machine Learning Models for Post-Wildfire Debris-Flow Prediction

Problem Statement

Predicting post-wildfire debris flows is critical for mitigating hazards to communities and infrastructure in recently burned areas during intense rainfall. However, identifying reliable machine learning (ML) models for this task is complicated by three primary challenges:

1. Feature Space Overlap: Debris-flow and non-debris-flow events exhibit significant overlap in their rainfall, burn-severity, and terrain signatures, preventing clean separation by simple thresholds.
2. Interpretability and Trust: When classes overlap, different model families establish decision boundaries differently. Understanding which features high-performing models rely on is non-trivial but essential for diagnosing whether predictions reflect robust signals or spurious patterns.
3. Data Scarcity and Imbalance: Available datasets are typically small and class-imbalanced (debris flows are the minority outcome), constraining a model's ability to learn the decision boundary and disproportionately limiting the identification of rare positive events.

Current operational approaches, such as those by the USGS, rely on logistic regression models. While improved threat scores have been reported for other models (e.g., Random Forest, XGBoost), prior studies often compare only a handful of models using single train-test splits, leading to variable performance estimates. Furthermore, emerging models for tabular data, such as foundation models, remain unevaluated in this specific domain.

Methodology

The study presents a rigorous evaluation framework comparing 15 models across five distinct families using a publicly available USGS dataset comprising 1,550 basin-storm observations from 34 fires across the western United States (2000–2013).

Data and Features

The dataset includes binary response labels (debris flow vs. no debris flow) and ten numeric features:

• Rainfall: Storm accumulation, duration, average intensity, and peak intensities at 15, 30, and 60 minutes.
• Fire & Terrain: Differenced Normalized Burn Ratio (dNBR), proportion of watershed burned at high/moderate severity on steep slopes (PropHM23), and contributing area.
• Soil: USLE soil erodibility factor for fine fragments (KF).

Experimental Design

• Cross-Validation: The authors employed repeated stratified 5-fold cross-validation (10 repeats, 50 total splits) to ensure stable performance estimates and avoid sensitivity to specific data splits.
• Preprocessing: A consistent protocol was applied, including fold-specific imputation (using physical relationships for rainfall and polynomial regression for dNBR) and standardization for distance/kernel-based models. Tree-based models operated on raw features.
• Hyperparameter Tuning: Bayesian optimization with 100 trials was used to maximize the threat score for most models, with separate tuning for unaugmented and augmented training conditions.
• Models Evaluated:
  • Logistic: Staley17 (operational baseline, 4 features) and standard Logistic Regression (10 features).
  • Distance/Kernel: K-Nearest Neighbors (KNN), Support Vector Classifier (SVC).
  • Deep Learning: Multi-Layer Perceptron (MLP), 1D CNN, LSTM, Transformer, and Mamba.
  • Tree-Based: Random Forest, ExtraTrees, XGBoost, CatBoost, and Recursive Feature Machines (xRFM).
  • Foundation Model: Tabular Prior-Data Fitted Network (TabPFN), a pre-trained Transformer that performs in-context learning without dataset-specific parameter updates.

Evaluation Metrics

The primary metric was the Threat Score (TS) (Jaccard Score), chosen for its robustness in class-imbalanced hazard assessment as it disregards true negatives. Supplementary metrics included AUC, F1-score, precision, and recall.

Additional Analyses

1. SHAP Feature Importance: SHapley Additive exPlanations (SHAP) were computed to evaluate feature reliance. This included global rankings, fold-stability analysis, and basin-level spatial mapping to verify if models conditioned on physical feature variations rather than global biases.
2. Synthetic Data Augmentation: TabPFN's unsupervised extension was used to generate class-conditional synthetic training data. The fidelity of this data was assessed using Kolmogorov–Smirnov statistics and Wasserstein distances. The impact of augmentation was evaluated by re-training all 15 models on a dataset augmented with these synthetic rows.

Key Results

Model Performance (Unaugmented)

• Top Performers: TabPFN achieved the highest mean threat score (0.637), closely followed by a tight tier of tree-based models: ExtraTrees (0.636), CatBoost (0.622), XGBoost (0.621), and Random Forest (0.620). The performance gap between the top models was negligible (span of 0.017) and often within the standard deviation of fold-to-fold variability.
• Baseline Comparison: The top nonlinear models significantly outperformed the operational Staley17 logistic regression model (TS 0.152) and the standard 10-feature Logistic Regression (TS 0.412).
• Deep Learning: Sequential and image-oriented deep learning models (CNN, LSTM, Transformer, Mamba) clustered in the middle tier (TS 0.426–0.459). They exhibited high recall but low precision, suggesting systematic over-prediction, likely due to a mismatch between their high capacity and the small sample size of the tabular dataset.

Feature Importance

• Dominant Signals: Across all top-performing models (TabPFN, Random Forest, XGBoost, CatBoost), rainfall features (PeakI15, StormAccum, PeakI30) consistently ranked as the most important predictors.
• Divergence: Rankings for secondary features (burn severity, soil erodibility) varied significantly between models. For instance, soil erodibility (KF) ranked highly for XGBoost and CatBoost but last for Random Forest.
• Physical Consistency: SHAP analysis confirmed that top models assigned positive contributions to higher values of burn severity, terrain steepness, soil erodibility, and rainfall intensity, aligning with physical expectations. Basin-level spatial maps showed that SHAP values tracked the spatial variation of features, indicating models were conditioning on physical quantities rather than spurious correlations.

Synthetic Data Augmentation

• Impact on Restricted Models: Augmentation provided the most significant gains for models with restricted feature sets or lower capacity. The Staley17 model saw a massive improvement (+0.133 in threat score), and deep learning models like LSTM and MLP also saw notable gains.
• Impact on Top Models: The leading tree-based models and TabPFN saw marginal improvements (≤ 0.022). TabPFN itself gained only +0.004, suggesting it had already saturated the information available in the original dataset.
• Conclusion on Augmentation: Synthetic augmentation is not a universal enhancement; its utility is highest for models that are under-parameterized or feature-restricted relative to the dataset's complexity.

Significance and Claims

The paper claims to provide the first comprehensive benchmark for post-wildfire debris-flow prediction that includes the TabPFN foundation model, systematic SHAP-based interpretability, and an evaluation of synthetic data augmentation.

Key contributions include:

1. Establishing a Performance Ceiling: The study demonstrates that a "practical top tier" of models (TabPFN and leading tree-based ensembles) exists with threat scores between 0.62 and 0.64. The tight clustering suggests that performance is bounded by the data's inherent overlap and sample size rather than model architecture alone.
2. Operational Efficiency: TabPFN matched or exceeded the performance of heavily tuned tree-based models (like XGBoost and CatBoost) without requiring any dataset-specific hyperparameter tuning, offering a significant advantage for rapid-response operational scenarios.
3. Interpretability Validation: The study validates that modern nonlinear models do not rely on spurious patterns; they converge on rainfall as the primary signal and respect physical expectations for secondary features, supporting their use alongside operational frameworks.
4. Targeted Augmentation: The findings clarify that synthetic data augmentation is most effective for models with limited capacity or feature sets, rather than as a blanket solution for all model families.

The authors maintain a modest stance, noting that the dataset is geographically concentrated (61% from Southern California) and temporally limited (first two post-wildfire years). Consequently, while the model rankings are robust under repeated cross-validation, the study emphasizes that spatial transferability to different fire regions and temporal transferability to later recovery stages require further investigation.