AoI-MDP: An AoI Optimized Markov Decision Process (Student Abstract)

Technical Summary: AoI-MDP for Underwater Autonomous Tasks

Problem Statement
The application of Reinforcement Learning (RL) to Autonomous Underwater Vehicles (AUVs) for ocean exploration faces a critical challenge: observation delays. In underwater environments, information limitations and channel constraints often cause significant delays between when an AUV observes a state and when that information is received or processed. Standard Markov Decision Processes (MDPs) typically assume instantaneous state reception, an idealization that fails to account for these delays. This discrepancy leads to non-causal control policies, reduced information freshness, and diminished decision-making efficiency, often resulting in task failure.

Methodology: AoI-MDP
To address these limitations, the authors propose an Age of Information (AoI) optimized Markov Decision Process (AoI-MDP). This framework explicitly models observation delay as a signal delay and integrates it into the decision-making loop.

• State Space ($S$): The state is redefined to include not only the observed information ($s'_i$) but also the observation delay ($Y_i$). Thus, $s_i = (s'_i, Y_i)$. The delay and observed information are modeled using a Statistical Delay Modeling (SDM) approach based on sensor data.
  • Delay Estimation: The system estimates time delay ($Y_i$) using a known sequence $S[n]$ corrupted by Gaussian white noise $W[n]$, employing a correlation-based estimator to maximize the likelihood of the delay.
  • Heading Estimation: The system estimates the heading angle ($\beta$) relative to environmental objects by maximizing a spatial period graph derived from sensor signals.
• Action Space ($A$): The action space is expanded to include a wait time variable ($Z_i$). An action is defined as a tuple $a_i = (a'_i, Z_i)$, where $a'_i$ is the physical action taken by the AUV, and $Z_i$ represents the duration the agent waits between observing environmental information and making a decision. This allows the agent to strategically delay actions to align with information freshness.
• Reward Function ($R$): The reward function is augmented to balance task performance with information freshness. It is defined as a tuple $r_i = (r'_i, -\bar{\Delta})$, where $r'_i$ is the original task reward and $-\bar{\Delta}$ represents the negative of the time-averaged AoI. The AoI ($\Delta(t)$) is modeled as a sawtooth function representing the time elapsed since the last successful update.

Key Contributions

1. Novel Framework: The authors claim to be the first to model underwater tasks as an MDP that jointly optimizes information updating and decision-making strategies using RL, specifically incorporating observation delay and AoI.
2. Statistical Delay Modeling (SDM): They introduce a sensor-based statistical modeling approach to realistically simulate and estimate observation delays and heading angles, moving beyond theoretical assumptions.
3. Action Space Expansion: By introducing "wait time" into the action space, the framework enables the AUV to actively manage the trade-off between acting immediately on stale data and waiting for fresher information.

Experimental Results
The authors evaluated AoI-MDP using a multi-AUV data collection task, comparing it against a standard MDP under identical conditions.

• Performance Metrics: Simulations demonstrated that AoI-MDP outperformed the standard MDP across four key metrics:
  • Lower Average AoI: Achieved a time-averaged AoI of approximately 1.97 compared to higher values in standard MDPs.
  • Reduced Energy Consumption: Showed lower energy usage (approx. 33.83 vs. higher baselines).
  • Higher Sum Data Rate: Achieved a sum information rate of 11.99, significantly outperforming standard MDPs.
  • Greater Cumulative Reward: Demonstrated superior learning effectiveness and total reward accumulation.
• Generalization: The model was tested against common delay distributions (Poisson, Exponential, Geometric). The SDM-based AoI-MDP consistently achieved near-optimal results across these distributions, confirming its robustness and generalization capabilities compared to models relying on specific delay assumptions.

Significance and Claims
The paper asserts that AoI-MDP provides a feasible and effective solution for underwater tasks where observation delays are inevitable. By integrating AoI into the state and reward structures and allowing for strategic wait times, the framework successfully enhances both information freshness and decision-making quality. The authors highlight that their approach offers superior performance and generalization in balancing multi-objective optimization (energy, data rate, and freshness) for marine robotics. To facilitate further research, the authors have released the source code and supplementary materials as open-source.