Information-Guided Safe Reinforcement Learning for Autonomous Gas Source Localization using sUAS

Technical Summary: Information-Guided Safe Reinforcement Learning for Autonomous Gas Source Localization

Problem Statement
The autonomous localization of fugitive gas emissions using small Unmanned Aircraft Systems (sUAS) is fundamentally an ill-posed inverse problem. In turbulent atmospheric boundary layers, scalar concentration fields are highly intermittent, violating the assumptions required for classical gradient-based navigation (chemotaxis). Data-driven estimators, such as Maximum Likelihood Estimation (MLE), suffer from severe noise and spurious local minima due to signal sparsity and sensor bias. Furthermore, the authors identify a critical vulnerability in deterministic information-seeking planners: a "Gramian bias." When planners rely solely on current, potentially flawed parameter estimates to compute trajectories (e.g., via Empirical Observability Gramian, EMGR), agents act greedily toward the estimate rather than the true source. This behavior creates a self-reinforcing degeneracy where the agent stalls, starving the estimator of the spatial diversity required for self-correction.

Methodology
To address these challenges, the authors propose an Information-Guided Safe Reinforcement Learning (RL) framework evaluated within a custom, GPU-accelerated 3D simulation environment.

1. Simulation Environment:
   The authors developed a hybrid simulation using the NVIDIA Warp framework. It couples an Eulerian wind solver (based on Stable Fluids and Navier-Stokes equations with operator splitting) to model macroscopic wind dynamics and turbulence, with a Lagrangian Gaussian puff dispersion model. This approach preserves sub-grid scalar intermittency and sharp concentration gradients, which are critical for realistic training and sim-to-real transfer.

2. Estimation and Observability:

   • Estimators: The system employs a dual-estimator approach: a frequentist MLE (enhanced with fractional filtering for wind history) and a Bayesian Particle Filter (PF) to maintain hypothesis diversity.
   • Reliability Monitoring: A Kullback-Leibler (KL) divergence metric compares empirical observations against estimator predictions. High divergence signals an unreliable estimate, triggering exploration.
   • EMGR Planning: A classical planner uses the Empirical Observability Gramian to maximize trajectory sensitivity to parameter perturbations. However, the authors note this planner is susceptible to the aforementioned Gramian bias.

3. Safe RL Architecture:
   The core contribution is a Residual Safe RL architecture that envelops the classical EMGR planner within a learned policy, governed by a Deterministic Meta-Supervisor.

   • Meta-Supervisor: This module dynamically blends the deterministic EMGR velocity ($v_{emgr}$) with a learned exploratory velocity ($v_{RL}$) from a Soft Actor-Critic (SAC) policy. The blending weight ($\omega_{act}$) is determined by four signals: KL divergence (mismatch), proximity to the current estimate (to counter bias), void steps (time since last detection), and a minimum exploration floor.
   • Safety Layer: To ensure zero safety violations, the framework integrates Robust Control Barrier Functions (RCBF). A Quadratic Program (QP) solves for a minimal correction to the blended velocity command, strictly mapping actions into a provably safe space while accounting for worst-case disturbances.
   • Training Strategy: The agent is trained via a progressive three-stage curriculum: (1) Static source in a restricted domain, (2) Static source in the full domain, and (3) Stochastic mobile source. This approach mitigates reward sparsity and combinatorial complexity.

Key Contributions

• Identification of Gramian Bias: The paper formally identifies and characterizes a specific failure mode in deterministic information-seeking planners where agents become trapped in local minima due to flawed early estimates.
• Hybrid Control Architecture: The integration of a classical EMGR planner with an off-policy SAC exploratory policy, mediated by a KL-divergence-based meta-supervisor, effectively breaks the degeneracy of greedy planning.
• Robust Safety Guarantee: The formulation of a kinematic RCBF that structurally enforces collision avoidance independent of the learned policy, ensuring zero safety violations even during aggressive exploration.
• High-Fidelity Simulation: The development of a GPU-accelerated 3D simulation environment that mathematically decouples microscopic gas transport from macroscopic grid dynamics, providing a rigorous testbed for turbulent source-seeking policies.

Results
Experiments were conducted over 100 episodes in a complex 3D domain with obstacles and mobile sources.

• Localization Success: The proposed RL framework achieved a ~78–79% success rate in localizing mobile sources within a 5m radius. In contrast, classical baselines (MLE+EMGR and MLE+EMGR+PF+KL) stalled at approximately 30% success, often failing to converge within the 2000-step horizon.
• Convergence Speed: RL agents converged to the target median distance in roughly 276–310 steps, whereas baselines stagnated above 24 meters.
• Safety: The framework achieved zero safety violations across all episodes. While RL agents triggered the RCBF safety filter significantly more frequently (84–89% of steps) due to aggressive exploration near obstacles, the minimum barrier margin remained above 2.54 meters, confirming the structural integrity of the safety layer.

Significance
The paper claims that by systematically breaking the degeneracy of greedy information-seeking through learned exploration and real-time reliability monitoring, the proposed framework offers a robust solution to the ill-posed nature of gas source localization in turbulence. The work demonstrates that combining classical observability theory with Safe RL, underpinned by a rigorous simulation environment, can drastically outperform classical baselines while maintaining strict safety guarantees. The authors position this as a foundational step toward the real-world deployment of autonomous sUAS for environmental monitoring and industrial leak detection, with future work focused on sim-to-real hardware transfer and multi-agent scenarios.