🤖 machine learning

OnDeFog: Online Decision Transformer under Frame Dropping

This paper proposes OnDeFog, an online reinforcement learning method that integrates Decision Transformer mechanisms for handling frame dropping with direct environmental interaction to outperform existing offline and online approaches in environments with high frame loss rates and low-reward data.

Original authors: Daiki Yotsufuji, Kenta Nishihara, Shoma Shimizu, Kento Uchida, Shinichi Shirakawa

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daiki Yotsufuji, Kenta Nishihara, Shoma Shimizu, Kento Uchida, Shinichi Shirakawa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to walk, drive a car, or play a video game. Usually, you give the robot a steady stream of information: "Here is where you are, here is what you did, and here is how well you did."

But in the real world, things go wrong. Sometimes the camera glitches, the internet lags, or a sensor fails. This is called frame dropping. It's like someone suddenly snatching pages out of your instruction manual while you are reading it. The robot doesn't know where it is or what happened a second ago, and it starts to stumble.

The Problem with Previous Solutions

Scientists have tried to fix this before, but they had two main problems:

The "Library Student" (DeFog): One method, called DeFog, is like a student who only studies from a giant, pre-written library of past games. They memorize how to handle missing pages if those missing pages appeared in the library. But if the robot encounters a brand-new situation that wasn't in the library, the "student" freezes because they've never seen it before. Also, collecting that library is expensive and hard.
The "Live Gamer" (ODT): Another method, called ODT, is like a gamer who learns by playing live. They can handle new situations because they are exploring in real-time. However, if the internet lags (frame dropping) while they are playing, they get confused and perform poorly because they weren't trained to deal with missing information.

The New Solution: OnDeFog

The authors of this paper created a new method called OnDeFog. Think of OnDeFog as a hybrid student who combines the best of both worlds:

The Training: First, they study the "library" (offline data) just like DeFog. But here's the trick: while studying, they pretend pages are missing. They practice reading the manual with holes in it, so they learn how to guess what's missing based on what came before.
The Live Practice: Then, they go out and play the game live (online learning) just like ODT. Because they practiced with "missing pages" during their study phase, they don't panic when the internet lags during the live game. They keep moving forward smoothly.

How It Works (The Metaphor)

Imagine you are driving a car with a GPS that sometimes loses signal.

Old ODT: You rely entirely on the GPS. If the signal drops, you stop or drive in circles because you don't know where you are.
Old DeFog: You memorized a map of every possible route. If the GPS drops, you look at your map. But if you take a shortcut the map doesn't show, you get lost.
OnDeFog: You memorized the map and you practiced driving with the GPS turned off randomly. So, when the signal drops, you instinctively know to rely on landmarks and your memory of the last few seconds to keep driving safely, even on new roads.

What the Experiments Showed

The researchers tested this new robot driver in three different "simulated worlds" (Hopping, Walking, and Running tasks) with different levels of "signal loss" (frame dropping).

High Signal Loss: When the signal was terrible (lots of dropped frames), OnDeFog was the clear winner. It kept performing well, while the "Live Gamer" (ODT) crashed and burned.
Bad Data Sets: When they used a "library" full of bad, low-reward examples (like a practice book full of mistakes), OnDeFog did better than the "Library Student" (DeFog). This is because OnDeFog went out and practiced live, finding better ways to move that the bad library didn't teach.
The Catch: OnDeFog isn't perfect everywhere. If the library was already very good (full of high-reward examples), OnDeFog sometimes struggled to find even better paths than what was already in the book. It's like a student who is so busy exploring new shortcuts that they miss the fact that the original map was actually the best route.

The Bottom Line

OnDeFog is a smarter way to train AI agents for messy, real-world environments. By teaching the AI to expect and handle missing information while it learns from live experience, it becomes much more robust than previous methods. It's not just about memorizing the past; it's about learning how to keep going when the future is uncertain.

Technical Summary: OnDeFog: Online Decision Transformer under Frame Dropping

Problem Statement
In real-world reinforcement learning (RL) applications, such as autonomous driving or drone control, agents frequently encounter incomplete observations due to communication delays or sensor failures, a phenomenon known as frame dropping. This results in the agent missing states and associated rewards, leading to significant performance degradation. While the Decision Transformer under Random Frame Dropping (DeFog) was developed to address this by incorporating specific mechanisms into the Decision Transformer (DT), it is an offline RL method. Consequently, DeFog struggles to generalize to novel states not adequately represented in its pre-collected training datasets and relies on high-quality, extensive datasets, incurring high collection costs. Conversely, Online Decision Transformer (ODT) enables learning through direct environmental interaction but lacks inherent mechanisms to handle frame dropping, making it susceptible to performance drops when observations are lost.

Methodology: OnDeFog
The authors propose OnDeFog (Online DeFog), a novel online RL method that integrates DeFog's robustness mechanisms into the ODT framework. OnDeFog aims to achieve stable training in frame-dropping environments while retaining the ability to explore and learn from new interactions.

The proposed method employs a two-stage training approach:

Offline Pre-training: The policy is initially trained using a pre-collected dataset via offline RL.
Online Fine-tuning: The policy is further improved through direct interaction with the environment.

To handle frame dropping during training, OnDeFog incorporates two key techniques from DeFog:

Train-time Frame Dropping: During training, inputs (states and returns) are intentionally dropped at specific time steps with a defined probability ( $p$ ). For dropped frames, the state and return of the preceding frame are carried over. This simulates the dropping environment, allowing the model to learn robustness against missing data.
Drop Span Embedding: The method calculates the number of consecutively dropped frames ( $D$ ) at each time step. This count is encoded into a vector by a drop span encoder and added to the state and return tokens. This allows the model to explicitly learn to handle missing patterns.

Unlike DeFog, OnDeFog does not utilize "freeze-trunk fine-tuning," as experiments indicated this did not yield substantial results in the online setting. Instead, OnDeFog utilizes a replay buffer where trajectories collected during online interaction are stored. During online learning, the target return is scaled (set to twice the optimal return) to encourage aggressive exploration, and the actual measured return is used for learning (hindsight return relabeling) to correct discrepancies between target and observed returns.

Key Contributions

Integration of Robustness and Online Learning: OnDeFog successfully bridges the gap between offline robustness and online adaptability by embedding DeFog's frame-dropping handling mechanisms into the ODT architecture.
Mitigation of Dataset Dependency: By leveraging online interaction, OnDeFog reduces the reliance on high-quality, extensive offline datasets, addressing a primary limitation of DeFog.
Two-Stage Training Framework: The method establishes a protocol for pre-training on offline data followed by online fine-tuning, specifically adapted to handle frame dropping via train-time simulation and drop span embeddings.

Experimental Results
The authors evaluated OnDeFog on the D4RL benchmark (Gym-MuJoCo tasks: Hopper, Walker2d, and HalfCheetah) across varying frame drop rates (0 to 0.9).

Vs. ODT: In environments with high frame drop rates, OnDeFog demonstrated superior performance and a more gradual degradation curve compared to ODT, which suffered sharp performance declines. In low drop-rate environments, OnDeFog performed comparably or slightly inferior to ODT, a result the authors attribute to the high drop rates used during training potentially hindering adaptation to low-drop scenarios.
Vs. DeFog: On the medium-replay datasets (characterized by a high proportion of low-reward data), OnDeFog often outperformed DeFog. This is attributed to OnDeFog's ability to acquire high-reward trajectories through online exploration, whereas DeFog is limited by the low-quality data in the offline dataset.
Limitations: On the HalfCheetah medium-replay dataset and the standard medium dataset (excluding Hopper), OnDeFog performed slightly inferior to DeFog. The authors attribute this to insufficient exploration in large action spaces (HalfCheetah) and the inability of online exploration to discover trajectories with rewards higher than those already present in the medium dataset, potentially degrading replay buffer quality.

Significance and Claims
The paper claims that OnDeFog provides a robust solution for RL agents operating in real-world environments prone to communication delays and sensor failures. It demonstrates that integrating frame-dropping mechanisms into an online learning framework allows agents to maintain performance where conventional online methods (ODT) fail and where offline methods (DeFog) are constrained by dataset quality.

The authors modestly note that while OnDeFog effectively mitigates performance degradation in high-drop-rate environments and on low-reward datasets, it faces challenges in scenarios with extremely low-reward trajectories or large action spaces where exploration is difficult. Future work is identified as developing methods to automatically adjust the timing of switching between offline and online learning based on dataset quality and policy performance.