🤖 machine learning

Adaptive Reinforcement Learning for Unobservable Random Delays

This paper introduces a general interaction layer framework and a model-based Actor-Critic with Delay Adaptation (ACDA) algorithm that enable reinforcement learning agents to dynamically adapt to unobservable, time-varying delays and packet losses, significantly outperforming existing methods in locomotion benchmarks.

Original authors: John Wikman, Alexandre Proutiere, David Broman

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: John Wikman, Alexandre Proutiere, David Broman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a high-speed video game where you control a robot. In a perfect world, you see the screen, press a button, and the robot moves instantly. But in the real world, things are messy. Your signal might get stuck in traffic, the robot's brain might take a moment to think, or the message might get lost entirely. By the time your command finally arrives, the robot has already moved, and your instruction is now wrong.

This is the problem researchers John Wikman, Alexandre Proutiere, and David Broman tackled. They found that standard AI training assumes everything happens instantly, which breaks down when dealing with real-world delays that are random and invisible to the AI.

The "Old Way" vs. The Real Problem
Previous methods tried to solve this by playing it super safe. They assumed the worst-case scenario: "What if the delay is 10 seconds?" So, they made the AI wait exactly 10 seconds before acting, even if the delay was usually only 1 second. It's like a driver who always waits at a green light for 10 minutes just in case the light stays red longer, even though it usually turns green in 2 seconds. This is safe, but it's incredibly slow and inefficient.

Other methods tried to guess the delay, but they often failed because the AI didn't actually know how long the delay would be when it made its decision. The paper argues against these rigid, fixed-delay approaches, showing they are too conservative and don't adapt to the chaos of real networks.

The New Solution: The "Interaction Layer"
The team introduced a clever new framework called the Interaction Layer. Think of this as a smart "command center" sitting between the AI brain and the robot.

Instead of sending one single command like "Jump now," the AI sends a matrix of future actions—a giant grid of possibilities. It's like a chess player who doesn't just make one move, but writes down a whole book of moves: "If the opponent moves here, I do this; if they move there, I do that."

The AI fills out this grid for many different possible future delays. It doesn't know which row of the grid will actually be used, because it doesn't know how long the message will take to arrive. But the Interaction Layer acts as the referee. When the message finally arrives, the Layer checks the time, figures out exactly how long the delay was, and picks the correct row from the grid to execute. If a message gets lost or arrives out of order, the Layer has backup rows ready to go.

The "Actor-Critic with Delay Adaptation" (ACDA)
To make this work, they built a new AI algorithm called ACDA. This AI is a bit of a psychic. Because it can't see the future, it uses a "model" to imagine what the world will look like when its action finally lands.

Imagine you are throwing a ball to a friend who is running away. You don't just aim at where they are now; you aim at where you think they will be when the ball arrives. ACDA does this mathematically. It calculates a "distribution" of where the robot will be in the future and generates actions based on that prediction. It's not guessing blindly; it's using a learned model of how the robot moves to fill in the gaps of missing information.

What the Numbers Say
The researchers tested this in a simulated world using MuJoCo, a physics engine for robots, across five different environments: Ant-v4, Humanoid-v4, HalfCheetah-v4, Hopper-v4, and Walker2d-v4. They didn't just use fake delays; they used real-world data collected from WiFi networks in a university library and an office.

The results were clear:

In the Ant-v4 environment with a specific simulated delay (GE1,23), the old best method (BPQL) scored 2691.88, while ACDA scored 4112.78.
In the Humanoid-v4 environment with the same delay, BPQL scored 585.19, while ACDA scored 4608.76.
Even with real-world WiFi delays from the library (DLib), ACDA consistently outperformed the state-of-the-art methods.

The paper notes that in almost every scenario tested, ACDA achieved higher average returns than the best existing methods. There was one exception: in the Ant-v4 environment with a specific type of delay (MM1), BPQL performed slightly better. However, across 15 out of 25 benchmarks tested, ACDA was the top performer.

How Sure Are They?
The authors are very confident in these results based on their simulations and replayed real-world data. They ran their tests over 1 million steps of training. However, they are careful to note that while the results are strong, they haven't mathematically proved that the algorithm will always converge to the perfect solution in every possible situation. The success is measured by how well it performed in their specific tests, not by a theoretical guarantee that it works 100% of the time in every universe.

The Bottom Line
This paper suggests that by letting the AI send a "menu" of future actions and letting a smart middle-layer pick the right one based on when it actually arrives, we can handle messy, unpredictable delays much better than by just waiting around or guessing. It turns a chaotic, unpredictable problem into a manageable game of "what if," allowing robots to move smoothly even when their signals are stuck in traffic.

Technical Summary: Adaptive Reinforcement Learning for Unobservable Random Delays

Problem Statement

Standard Reinforcement Learning (RL) algorithms, such as Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC), typically assume a Markov Decision Process (MDP) where the agent observes the system state instantaneously and executes actions without delay. In real-world cyber-physical systems, this assumption frequently fails due to interaction delays arising from observation transmission, computation, and actuation.

Existing approaches to handle delays generally fall into two categories:

Partially Observable MDPs (POMDPs): Modeling delays as outdated observations, which limits the agent's access to information about the environment's evolution during the delay.
Fixed-Delay Assumptions: Enforcing a constant delay (often the worst-case upper bound) via action buffers. While this simplifies decision-making by providing perfect information about action timing, it is overly conservative. It forces the agent to operate under worst-case latency even when actual delays are minimal, leading to suboptimal performance.

A critical gap exists in handling unobservable, time-varying random delays. In many real-world scenarios (e.g., networked control), the agent does not know at decision time how long an action will take to execute, nor can it predict if packets will be lost or arrive out of order. Existing methods like DCAC allow for unobservable delays but lack control over when actions are applied, while other state-of-the-art algorithms (e.g., BPQL, VDPO) rely on fixed-delay paradigms.

Methodology

1. The Interaction Layer Framework

The authors introduce a novel framework called the Interaction Layer, which sits between the agent and the controlled system. This layer manages the uncertainty of unobservable delays through two main components: an Observer and an Action Buffer.

Observation Packets: The observer samples the system state and sends an observation packet to the agent. This packet includes the state, the current action buffer contents, the delay of the packet that updated the buffer, and a shift counter.
Action Packets (Matrix Generation): Instead of sending a single action, the agent generates an action packet containing a matrix of candidate actions.
- The matrix has $L$ rows (prediction length) and $h$ columns (action buffer horizon).
- The $i$ -th row corresponds to the sequence of actions to be executed if the packet arrives at time $t+i$ .
- The columns handle scenarios where no new packet arrives; if a packet is missing at a specific step, the system shifts to the next column in the buffer.
Execution Logic: When an action packet arrives at the interaction layer, the system calculates the round-trip delay ( $\delta$ ) using local clocks. It then selects the specific row from the matrix corresponding to that delay. If the delay exceeds the matrix rows ( $L$ ) or packets arrive out of order, the packet is discarded.

2. Actor-Critic with Delay Adaptation (ACDA)

Building on the interaction layer, the authors propose ACDA, a model-based RL algorithm designed to adapt dynamically to varying delays.

Heuristic for Assumed Previous Actions: Since the agent cannot know exactly which actions were applied to the system between generating a packet and its arrival, ACDA employs a heuristic. It assumes that if the current packet has a delay of $k$ , previous packets also had a delay of $k$ . This allows the agent to reconstruct the sequence of actions applied to the system, enabling it to predict the current state distribution.
Model-Based Distribution Agent: Standard neural networks (MLPs) require fixed-size inputs, but the sequence of "memorized" actions in ACDA varies in length. To solve this, ACDA uses a multi-step distribution model to embed the variable-length action history into a fixed-size latent representation.
- EMBED: Encodes the current state into a latent distribution.
- STEP: A Gated Recurrent Unit (GRU) that updates the latent distribution by simulating the effect of applying the sequence of memorized actions.
- EMIT: A decoder used during training to ensure the latent representation accurately predicts future states (minimizing KL-divergence).
- The policy $\pi_\theta$ then generates actions based on this fixed-size embedding of the future state distribution, rather than raw state-action histories.
Training: ACDA follows an Actor-Critic setup based on Soft Actor-Critic (SAC). The critic evaluates values on undelayed system states, requiring the algorithm to reconstruct the trajectory from the perspective of the undelayed MDP during training.

Key Contributions

Interaction Layer Framework: A general framework enabling agents to handle unobservable, stochastic delays and packet loss by generating a matrix of future actions. This allows for informed decision-making under uncertainty without assuming a fixed delay.
ACDA Algorithm: A model-based RL algorithm that leverages the interaction layer. It introduces a heuristic for estimating applied actions and a distribution-based embedding mechanism to handle variable-length action histories, allowing the policy to reason about future state distributions.
Empirical Validation: Comprehensive evaluation on MuJoCo locomotion tasks using both simulated delay processes (Gilbert-Elliot and M/M/1 queue models) and real-world measured WiFi delays.

Results

The paper evaluates ACDA against state-of-the-art baselines, including SAC (with and without Constant-Delay Augmentation), Dreamer, BPQL, VDPO, and DCAC.

Performance: ACDA consistently outperforms all baselines across a wide range of environments (Ant, Humanoid, HalfCheetah, Hopper, Walker2d) and delay processes.
Real-World Delays: In experiments using recorded WiFi network delays (DLib and DOff), ACDA demonstrated superior performance, particularly because real-world delays exhibit temporal correlations that ACDA is designed to exploit.
Robustness: While ACDA assumes temporal correlation in delays, it maintains acceptable performance even under independent and identically distributed (i.i.d.) delay processes, though with a smaller margin.
Comparison to Fixed-Delay Methods: Methods relying on worst-case fixed delays (like BPQL and VDPO with Constant-Delay Augmentation) performed significantly worse, highlighting the inefficiency of conservative assumptions in stochastic environments.

Significance and Claims

The authors claim that this work represents the first framework allowing agents to make informed, controlled decisions under random, unobservable delays in RL. By moving beyond the fixed-delay paradigm, the interaction layer and ACDA enable agents to adapt to the actual latency of the system rather than a conservative upper bound.

The paper emphasizes that while convergence guarantees for deep RL under random delays (POMDP formulation) remain theoretically open, the empirical results demonstrate that ACDA achieves significantly higher average returns than current state-of-the-art approaches. The work suggests that handling the uncertainty of delay directly, rather than enforcing artificial constancy, is a more effective strategy for deploying RL in real-world cyber-physical systems. Future work directions include extending the framework to handle more complex interaction behaviors and optimizing the action packet structure to reduce computational and communication overhead.