Pre-training Assessment of Model Transferability via Distribution Shift in Petroleum Engineering

Technical Summary: Pre-training Assessment of Model Transferability via Distribution Shift in Petroleum Engineering

Problem Statement
Data-driven models are increasingly deployed in petroleum engineering for tasks such as reservoir characterization, well-log interpretation, and production prediction. However, a critical challenge arises when models trained on a source dataset (e.g., a specific well or field) are applied to a target dataset with different geological properties, operating conditions, or measurement systems. This "distribution shift" often leads to unpredictable degradation in predictive performance. While transfer learning and similarity-based strategies exist to mitigate these issues, they typically require model retraining or adaptation after the shift is identified. There is currently a lack of simple, model-independent metrics that can quantify the potential difficulty of transferring a model before the training process begins.

Methodology
The authors propose a pre-training framework designed to quantify source–target distribution shift and examine its relationship with subsequent zero-shot model transfer performance. The methodology consists of four primary steps:

1. Feature Extraction: The framework utilizes a fixed set of seven predefined engineering features derived from well-log data: Gamma Ray (GR), Bulk Density (RHOB), Neutron Porosity (NPHI), Photoelectric Factor (PEF), Deep Resistivity (RDEP), Compressional Sonic Transit Time (DTC), and Caliper (CALI). These features are extracted from both source and target datasets without relying on learned representations.
2. Distribution Shift Measurement: The statistical difference between source and target distributions is measured using Wasserstein-based distances.
   • Marginal Shift: The first-order Wasserstein distance ($W_1$) is calculated for each feature individually.
   • Multivariate Shift: The Sliced Wasserstein distance ($WS_1$) is employed to capture joint distribution differences by projecting multidimensional data onto one-dimensional directions.
   • A Distribution-Shift Index ($D_{S,T}$) is computed as the average of these distances. Crucially, this index is calculated using only the feature distributions, independent of any prediction model.
3. Zero-Shot Transfer Evaluation: A Random Forest classifier (200 trees) is trained exclusively on the source domain. It is then evaluated directly on the target domain without any retraining, fine-tuning, or domain adaptation.
4. Statistical Correlation: The study analyzes the relationship between the pre-training shift index ($D_{S,T}$) and the relative performance degradation ($\Delta M$), defined as the difference between source and target performance (Accuracy and Macro-F1) normalized by the source performance. Pearson and Spearman correlations are used to test for systematic relationships.

Experimental Setup and Results
Due to the unavailability of the real FORCE 2020 dataset in the execution environment, the authors validated the computational pipeline using a controlled synthetic dataset comprising 24 synthetic wells (900 depth samples each) and six lithology classes. The synthetic data was designed to induce covariate shift (changes in input distribution) while keeping the underlying relationship between logs and lithology constant.

• Pipeline Validation: The corrected pipeline successfully processed the data, calculating shift indices and zero-shot performance metrics. The source validation accuracy averaged 0.833, while target zero-shot accuracy averaged 0.830, with relative performance changes ranging from -3.9% to +3.6%.
• Correlation Analysis: The study tested the hypothesis that larger distribution shifts correlate with greater performance degradation.
  • Macro-F1: A positive correlation was observed between the shift indices and Macro-F1 degradation (Spearman $\rho \approx 0.34$ for Sliced Wasserstein), suggesting that wells with larger distributional differences tend to suffer greater performance loss in class-balanced metrics.
  • Accuracy: Correlations with overall accuracy were negative and weak.
  • Significance: None of the correlations reached the conventional statistical significance level ($p < 0.05$). The authors attribute this to the small sample size (24 cases) and the controlled nature of the synthetic data, which lacked "concept shift" (changes in the conditional relationship between inputs and labels).

Key Contributions

1. Model-Independent Pre-training Assessment: The study introduces a diagnostic framework that quantifies source–target compatibility using only raw feature distributions, allowing engineers to assess transfer risk before investing in model development.
2. Decoupling Data and Model Metrics: The framework explicitly separates the measurement of data-level distribution shift from model-level performance, providing a clear statistical link between the two.
3. Use of Wasserstein Distances: The application of Wasserstein and Sliced Wasserstein distances provides a robust, magnitude-aware measure of distributional discrepancy suitable for multivariate engineering data.
4. Reproducible Pipeline: The authors provide a complete, executable workflow (detailed in Appendices) that starts from raw LAS files, ensuring the methodology can be reproduced on the full FORCE 2020 dataset once accessible.

Significance and Claims
The paper modestly claims that the proposed framework provides a preliminary diagnostic tool rather than a definitive predictor of model failure. The authors emphasize that the current results, derived from synthetic data, offer "early, direction-consistent support" for the hypothesis that pre-training distribution shift predicts zero-shot performance degradation, particularly when measured via Macro-F1.

The significance lies in the potential for this index to serve as an early screening measure. A high distribution-shift value does not guarantee model failure but indicates that the target well possesses a statistical log distribution significantly different from the source population. This signals to engineers that the transfer setting may be challenging and warrants further investigation into geological settings, petrophysical properties, or measurement conditions before deployment. The authors explicitly state that the current findings are not conclusive and that the next critical step is to apply the same pipeline to the real, geologically diverse FORCE 2020 dataset to verify statistical significance and generalizability.