⚡ electrical engineering

Enhancing Information Freshness: An AoI Optimized Markov Decision Process

This paper proposes an Age of Information (AoI) optimized Markov Decision Process (AoI-MDP) that integrates observation delay into the state space and wait time into the action space to jointly optimize information freshness and decision-making for autonomous underwater vehicles, demonstrating superior performance in multi-AUV data collection tasks through open-source simulations.

Original authors: Jingzehua Xu, Yimian Ding, Yiyuan Yang, Guanwen Xie, Shuai Zhang

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jingzehua Xu, Yimian Ding, Yiyuan Yang, Guanwen Xie, Shuai Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guide a team of underwater robots (AUVs) to collect data from sensors scattered across the ocean floor. The problem? The ocean is a noisy, difficult place. When a robot looks at a sensor, the information it gets is often "stale" by the time it arrives. It's like trying to navigate a maze while looking at a map that is 10 minutes old; you might make a decision based on where a wall was, not where it is now. This delay causes the robots to make bad decisions, waste energy, and fail their missions.

This paper introduces a new way to teach these robots how to think, called AoI-MDP. Here is how it works, broken down into simple concepts:

1. The Problem: The "Stale Coffee" Effect

In the old way of doing things (Standard MDP), the robots were taught to act immediately the moment they received a signal. But in the deep ocean, signals take time to travel. By the time the robot acts, the information is already "cold coffee"—it's no longer fresh. The paper argues that simply acting fast isn't enough; you need to know how old your information is.

2. The Solution: The "Freshness" Meter (AoI)

The authors introduce a concept called Age of Information (AoI). Think of this as a "freshness meter" on a food label.

Old way: The robot ignores the expiration date.
New way (AoI-MDP): The robot constantly checks the "expiration date" of its data. If the data is getting too old, the robot knows its view of the world is blurry.

3. The Three Big Changes

To fix the problem, the researchers changed the "brain" of the robot in three specific ways:

Change #1: The Robot's "Memory" (State Space)
Instead of just telling the robot what it sees, they also tell it how long ago it saw it.
- Analogy: Imagine a driver who not only sees a red light but also knows, "I saw this red light 5 seconds ago, and the light might have changed since then." The robot now carries a "delay tag" with every piece of information.
Change #2: The "Wait" Button (Action Space)
In the past, robots were forced to act the second they got a signal. Now, the robot is allowed to wait.
- Analogy: Imagine you are waiting for a bus. If you see the bus is 2 minutes away, you don't start running immediately; you wait a moment so you don't get tired. The robot learns that sometimes, the smartest move is to pause and wait for a fresher, more accurate signal before making a move.
Change #3: The "Scorecard" (Reward Function)
The robot is given a new way to earn points. It doesn't just get points for finishing the job; it gets penalized if it uses old, stale information.
- Analogy: It's like a video game where you lose points if you try to shoot a target using a map from yesterday. To get the high score, you must balance moving fast with making sure your data is fresh.

4. How They Tested It

The researchers didn't just guess; they ran a simulation of a "multi-robot data collection" mission.

The Setup: They pitted their new "Freshness-Aware" robots against the old "Instant-Action" robots.
The Result: The new robots were much better. They used less energy, collected more data, and made fewer mistakes.
The "Realism" Factor: Most studies pretend the delay is random (like rolling a die). This paper used a more realistic model based on how sound and signals actually travel underwater (Statistical Delay Formulation). It's like the difference between guessing how long a phone call will take versus actually measuring the signal travel time. The new method worked better even when they tested it with different types of "noise" or delays.

5. The Bottom Line

The paper concludes that by teaching robots to care about how fresh their information is and giving them the permission to wait for better data, we can make underwater exploration much more efficient. They have even made their simulation code public so other researchers can try it out.

In short: They stopped treating the robots like they have instant, perfect vision, and started treating them like real explorers who have to account for the time it takes to see the world clearly.

Technical Summary: Enhancing Information Freshness via AoI-Optimized Markov Decision Process

Problem Statement
The paper addresses the critical challenge of observation delay in underwater exploration tasks utilizing Autonomous Underwater Vehicles (AUVs) and Reinforcement Learning (RL). While AUVs rely on Information Updating Networks (IUNs) for collaboration, the harsh underwater environment often leads to significant information limitations. These limitations cause observation delays, resulting in non-causal control policies where decisions are made based on outdated data. Existing approaches often model these delays as simple random distributions or stationary stochastic variables, or they assume instantaneous state reception, which fails to capture the complexity of real-world underwater communication. Furthermore, standard strategies often assume agents act immediately upon receiving information (zero-wait strategy), which is shown to be suboptimal in high-variance delay scenarios. The core problem is the lack of a framework that jointly optimizes information freshness and decision-making while realistically modeling observation delays.

Methodology
The authors propose an AoI-optimized Markov Decision Process (AoI-MDP) designed specifically for underwater tasks. This framework integrates the Age of Information (AoI) metric directly into the MDP structure to manage delay and freshness. The methodology consists of three primary components:

State Space Reformulation: Unlike standard MDPs, the AoI-MDP state space ( $S$ ) explicitly includes the observation delay ( $Y_i$ ) alongside the AUV's observed information ( $s'_i$ ). The state is defined as $s_i = (s'_i, Y_i)$ . To model this delay realistically, the paper employs a Statistical Delay Formulation (SDF) rather than simple random distributions. The SDF utilizes sensor-based models where AUVs estimate the time delay ( $Y_i$ ) and heading ( $\beta$ ) of environmental objects by analyzing signal propagation, reflection, and noise characteristics (modeled via Gaussian white noise). This allows the AUV to be aware of the "timing delay" of its observations.
Action Space Expansion: The action space ( $A$ ) is expanded to include a wait time component ( $Z_i$ ). The action is defined as a tuple $a_i = (a'_i, Z_i)$ , where $a'_i$ is the physical action and $Z_i$ is the duration the AUV waits before executing the decision. This allows the agent to optimize the timing of information updates, avoiding the suboptimal zero-wait strategy in high-delay variability scenarios.
Reward Function Integration: The reward function ( $R$ ) is modified to include the time-averaged AoI ( $-\bar{\Delta}$ ) as a penalty term alongside standard task rewards (e.g., efficiency, collision avoidance). The time-averaged AoI is calculated based on the interplay between observation delays ( $Y_i$ ) and wait times ( $Z_i$ ). The total reward function is a weighted sum of these components, guiding the RL agent to minimize AoI while maximizing task performance.

The framework is trained using RL algorithms (specifically Soft Actor-Critic and Conservative Q-Learning in the experiments) to achieve joint optimization of information updating strategies and decision-making policies.

Key Contributions
The paper claims the following contributions:

First Formulation: To the authors' knowledge, this is the first work to formulate underwater tasks as an MDP that explicitly incorporates both observation delay and AoI, utilizing RL for joint optimization of information updating and decision-making.
Statistical Delay Modeling: Instead of treating delay as a simple random variable, the study introduces a Statistical Delay Formulation (SDF) based on sensor models. This approach models observation delay as "delayed observation timing delay," aiming to produce more realistic results that bridge the gap between simulation and real-world underwater tasks.
Joint Optimization: The framework successfully integrates wait time into the action space and AoI into the reward function, demonstrating the ability to balance multi-objective optimization (freshness vs. task efficiency).

Experimental Results
The proposed AoI-MDP was validated through extensive simulations in a multi-AUV data collection scenario. Key findings include:

Performance vs. Standard MDP: Compared to a standard MDP (without delay awareness or wait time), the AoI-MDP achieved lower time-averaged AoI, reduced energy consumption, higher sum information rates, and greater cumulative rewards.
Generalization: The framework demonstrated strong generalization capabilities across different delay models (Poisson, Exponential, Geometric), with the SDF-based approach yielding near-optimal results in AoI, information rate, and energy metrics.
Algorithm Compatibility: The approach was successfully applied to both online (SAC) and offline (CQL) RL algorithms, showing favorable training outcomes in both contexts.
Trajectory Efficiency: Visual analysis of AUV trajectories showed that policies trained under AoI-MDP resulted in more extensive coverage and more effective data collection compared to the erratic and less efficient trajectories generated by standard MDP policies.

Significance and Claims
The paper positions AoI-MDP as a necessary evolution of the standard MDP framework for underwater robotics. The authors argue that extending the MDP to incorporate observation delays and AoI is essential because the idealization of instantaneous state reception does not hold in practical scenarios with channel limitations and high update frequencies. By minimizing AoI, the framework enhances information freshness, which in turn facilitates more efficient decision-making processes despite observation delays. The authors conclude that the approach effectively minimizes AoI while showcasing superior performance in underwater tasks, and they have released the simulation code as open-source to accelerate further research in this field.