Decision-Driven Geosteering Under Uncertainty: A Unified Framework for Sequential Decision Optimization

Technical Summary: Decision-Driven Geosteering Under Uncertainty

Problem Formulation
Geosteering is defined in this work as a sequential decision-making problem under uncertainty, specifically modeled as a Partially Observable Markov Decision Process (POMDP). The objective is to navigate a well trajectory through an unknown geological configuration by maximizing reservoir contact while minimizing drilling costs and avoiding hazards. The core challenge lies in the fact that the subsurface state ahead of the drill bit is not directly observable; decisions must be made based on indirect, noisy measurements (e.g., gamma-ray logs) acquired during drilling. This creates a scenario where the decision-maker must maintain a probabilistic belief state over geological boundaries and select steering actions (inclination/azimuth adjustments) that respect physical constraints, such as maximum dogleg severity (DLS).

Methodology
The paper proposes a unified framework that tightly integrates probabilistic subsurface interpretation with value-based sequential decision optimization. The methodology consists of three primary components:

1. Uncertainty Quantification via Particle Filtering (PF):
   Geological uncertainty is explicitly represented using a Particle Filter. The PF maintains a weighted ensemble of particles, where each particle corresponds to a plausible boundary configuration (defined by vertical offset and local dip).

   • Transition Model: Particle states evolve stochastically. Crucially, the transition model for boundary inclination increments is learned offline using a Kernel Density Estimator (KDE) trained on empirical angle statistics from reference stratigraphic interpretations.
   • Update Mechanism: As new gamma-ray (GR) measurements are acquired, particle weights are updated based on the likelihood of the observation given the particle's implied boundary configuration. Resampling is performed to prevent degeneracy, resulting in a real-time probabilistic estimate of the reservoir boundaries.

2. Decision State Representation:
   Rather than feeding the full particle ensemble into the decision policy, a compact decision state is constructed. This state vector includes:

   • Current well inclination.
   • Posterior weights of the top five particle hypotheses.
   • Boundary-relative geometry descriptors (normalized signed distances) evaluated at 26 checkpoints spanning the recent decision interval.
     This feature mapping preserves interpretability while reducing dimensionality for learning algorithms.

3. Decision Optimization Strategies:
   The framework evaluates three distinct decision-making policies operating under the identical PF-based uncertainty representation:

   • Approximate Dynamic Programming (ADP): A "white-box," non-learning approach that uses a fixed parametric value function augmented with an optimism mechanism (adding a term proportional to belief-state uncertainty) to encourage informative exploration. It employs a one-step look-ahead policy.
   • Deep Q-Learning (DRL): A standard Deep Q-Network baseline that learns a state-action value function through trial-and-error interaction with the environment.
   • Dual Deep Reinforcement Learning (Dual DRL): The proposed architecture utilizes a Dueling DQN structure, decomposing the Q-value into a state-value estimator ($V$) and an advantage estimator ($A$). This design, combined with a target network and experience replay, aims to improve stability and robustness, particularly when many actions yield similar long-term outcomes.

Key Contributions

• Unified Framework: The integration of a KDE-based geological generator, a Particle Filter for probabilistic forecasting, and multiple decision optimization schemes into a single, API-integrated testing environment (StarSteer simulator).
• Stability-Oriented Evaluation: Moving beyond aggregate final placement metrics, the paper introduces the Root Mean Square (RMS) Jerk as a metric to quantify steering smoothness and control stability. This allows for the assessment of how policies react to evolving uncertainty over time.
• Comparative Analysis: A controlled comparison of ADP, standard DRL, and Dual DRL under identical geological realizations, noise models, and operational constraints, isolating the effects of the decision-making algorithms.

Experimental Results
Experiments were conducted using high-performance computing resources with geological uncertainty modeled via stochastic sampling.

• Learning Dynamics: The Dual DRL architecture demonstrated superior learning stability compared to standard DRL. It exhibited reduced variance across random seeds, smoother optimization dynamics (lower value-function loss), and faster convergence to performance thresholds. A replay buffer size of 50,000 transitions was identified as the optimal balance between stability and efficiency.
• Policy Performance: In terms of final placement (cumulative reward), Dual DRL achieved the highest median performance with the least dispersion across independent runs. Standard DRL showed higher variability, while ADP, though deterministic, yielded the lowest median performance due to its fixed approximation structure.
• Control Smoothness: Dual DRL significantly outperformed both ADP and standard DRL in terms of trajectory smoothness, achieving the lowest RMS jerk (0.8 vs. 1.32 for ADP and 1.38 for DRL). This indicates that Dual DRL produces fewer abrupt steering corrections.
• Behavioral Analysis: Visual analysis of the "PF unfolding" (the evolution of uncertainty ahead of the bit) revealed that ADP tends to oscillate reactively around the interpreted centerline, while standard DRL makes frequent local corrections. Dual DRL maintained a trajectory closely aligned with the interpreted structure, avoiding excessive oscillations even as predictive uncertainty increased.

Significance and Claims
The paper claims that the quality of geosteering policies cannot be assessed solely by final trajectory overlap or aggregate reward. While different methods may produce visually similar paths, their underlying decision dynamics and responses to uncertainty differ substantially.

• Stability as a Criterion: The authors argue that stability and robustness are essential for operational viability. The Dual DRL architecture is shown to effectively balance responsiveness with robustness, making it better suited for the sequential nature of geosteering under uncertainty.
• Limitations of Aggregate Metrics: The study highlights that differences in decision quality are expressed through how policies interpret and act on evolving belief states, a nuance captured by the proposed stability metrics and PF unfolding analysis but missed by final placement metrics alone.
• Future Work: The authors note that while the current framework is robust, future work will extend it to more complex geological settings and integrate sequence-level decision models (such as Decision Transformers) into the existing PF-based belief-state setup.

The work positions itself as a step toward bridging the gap between synthetic experimentation and deployment-oriented evaluation by validating these methods within a realistic industrial simulator under measurement noise and strict operational constraints.