🤖 machine learning

Decoupled Guidance Diffusion for Adaptive Offline Safe Reinforcement Learning

This paper introduces Safe Decoupled Guidance Diffusion (SDGD), a novel offline safe reinforcement learning framework that combines cost-conditioned classifier-free guidance with Feasible Trajectory Relabeling to effectively decouple safety constraints from reward optimization, thereby achieving superior safety compliance and high returns in adaptive budget scenarios.

Original authors: Rufeng Chen, Zhaofan Zhang, Zhejiang Yang, Hechang Chen, Sihong Xie

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rufeng Chen, Zhaofan Zhang, Zhejiang Yang, Hechang Chen, Sihong Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to drive a car, but you only have a video recording of how other cars drove in the past. You can't let the robot drive around and crash to learn; it has to learn strictly from that old footage. This is Offline Reinforcement Learning.

Now, add a twist: sometimes you want the robot to be very cautious (like driving in a school zone), and other times you want it to be a bit more aggressive (like driving on an open highway). The "safety budget" changes depending on the situation. This is Adaptive Safe Reinforcement Learning.

The problem is that most existing methods are like a student trying to memorize a specific route for a specific speed limit. If the speed limit changes, they get confused or crash.

Here is how the paper's new method, SDGD, solves this, explained through simple analogies:

1. The Old Way: The "Step-by-Step" vs. The "Blurry Photo"

The Old Way (Autoregressive Models): Imagine trying to draw a long, winding road by drawing one tiny segment, then the next, then the next. If you make a tiny mistake in the first segment, the next segment has to compensate, and by the end, your road is completely off the map. This is how older AI planners worked; they made small errors that piled up, causing the robot to violate safety rules.
The New Way (Diffusion Models): Imagine taking a blurry, noisy photo of the entire road at once and slowly sharpening it until the whole path is clear. This is what Diffusion does. It generates the whole journey together, so small errors don't pile up.

2. The Core Problem: Mixing "Safety" and "Speed"

In the past, AI tried to balance safety and speed by treating them as two opposing forces pulling the robot in different directions.

The Analogy: Imagine a driver who is told, "Drive as fast as possible, but don't hit the wall." The AI would try to calculate the perfect angle to go fast and stay safe. But if the "wall" (safety limit) moves, the AI gets confused. It often tries to go fast and accidentally hits the wall because it thought the wall was somewhere else.

3. The SDGD Solution: The "Two-Coach" System

The authors realized that Safety and Speed shouldn't be fighting each other; they should have different jobs. They created a system with two distinct "coaches":

Coach 1: The Safety Enforcer (Classifier-Free Guidance)
- Job: This coach looks at the "Safety Budget" (e.g., "You can only spend 10 points on risk today").
- Action: It doesn't try to calculate the perfect angle. Instead, it simply says, "If the path you are drawing goes into the 'danger zone' (over the budget), erase it and redraw it." It acts like a filter that only allows safe paths to exist. It doesn't care how fast you go, only that you stay within the lines.
Coach 2: The Speed Coach (Reward-Gradient Guidance)
- Job: This coach looks at the safe paths and says, "Of all the safe paths we have, which one gets you to the finish line fastest?"
- Action: It nudges the robot toward the fastest safe route.

The Magic: By separating these two jobs, the robot can instantly switch from "School Zone Mode" (tight safety budget) to "Highway Mode" (loose budget) just by telling Coach 1 to change the rules. It doesn't need to relearn how to drive.

4. The Sneaky Trap: The "Feasible Trajectory Relabeling" (FTR)

There was one catch. Even with two coaches, the "Speed Coach" might get greedy.

The Problem: Sometimes, the fastest way to get a high score involves taking a huge risk at the very beginning of the trip. If the robot takes that risk, it might crash immediately, but the "Speed Coach" sees the high potential score and says, "Go for it!"
The Fix (FTR): The authors introduced a rule called Feasible Trajectory Relabeling.
- The Analogy: Imagine a student who gets an 'A' on a test but cheated on the first question. The teacher (FTR) says, "Even though you got an 'A', because you cheated on the first part, we are going to mark your whole test as 'F'."
- How it works: The system looks at the first few steps of the robot's plan. If those first steps are dangerous (even if the rest of the plan looks great), the system tells the "Speed Coach," "Don't give credit for this path." This stops the robot from taking dangerous shortcuts just to get a high score.

5. The Results: The Perfect Balance

The researchers tested this on 38 different driving and robotics tasks (like a car trying to push a button or a drone flying through a course).

The Score: Their method (SDGD) was safe enough to pass the rules in 36 out of 38 tasks.
The Performance: Among all the methods that were safe, SDGD was the fastest (highest reward) in 21 of those tasks.
The Flexibility: They could change the safety rules while the robot was running, and the robot adapted instantly without needing to be retrained.

Summary

Think of SDGD as a smart navigation system that doesn't just calculate the fastest route.

It first draws a map that only includes roads that are legal for your current speed limit (Safety Coach).
Then, it picks the fastest route from that legal map (Speed Coach).
It has a special rule that says, "If the first turn is illegal, the whole route is illegal," preventing the system from trying to cheat its way to a faster time.

This allows robots to be safe and efficient, even when the rules of the road change on the fly.

Technical Summary: Safe Decoupled Guidance Diffusion (SDGD)

Problem Definition
Offline Safe Reinforcement Learning (RL) aims to learn policies from pre-collected datasets that optimize task performance while satisfying safety constraints. A critical challenge arises during deployment where safety budgets (cost limits) are not fixed; they may vary across tasks, users, or operating conditions, and can even change dynamically within a single episode. Existing methods struggle with this adaptability. Autoregressive trajectory models suffer from accumulated rollout errors, leading to significant deviations in cumulative cost over long horizons. Conversely, diffusion-based planners, while capable of joint trajectory generation, often rely on classifier guidance or gradient switching to enforce safety. These approaches require cost classifiers to provide reliable signals on noisy intermediate trajectories, a condition that empirical evidence suggests is fragile; cost classifiers often exhibit low and unstable correlation with true cumulative costs, leading to unreliable constraint satisfaction.

Methodology
The paper proposes Safe Decoupled Guidance Diffusion (SDGD), a diffusion-based planner that reinterprets adaptive safe trajectory generation as sampling from a constrained trajectory distribution. The core insight is an asymmetry between safety and reward: safety acts as a hard restriction on the trajectory support (defining where to sample), while reward acts as a preference shaping the distribution within that support (defining which samples to prefer). SDGD decouples these roles into two distinct guidance mechanisms:

Cost-Limit Conditioning via Classifier-Free Guidance (CFG):
Instead of using a separate cost classifier to guide the reverse diffusion process, SDGD trains a diffusion model conditioned directly on the cost limit $l$ . During sampling, Classifier-Free Guidance is applied to bias the generation toward trajectories satisfying the specified limit. This approach avoids the instability of differentiating noisy cost predictions and allows the model to adapt to varying or dynamic cost limits without retraining.
Reward-Gradient Refinement:
Once the sampling is biased toward the safe region, a learned reward predictor provides gradient guidance to refine trajectories for higher return. However, the authors note that direct reward guidance can inadvertently increase cumulative cost, as high-return trajectories in offline data often correlate with high costs.
Feasible Trajectory Relabeling (FTR):
To mitigate the "reward-induced cost drift," SDGD introduces FTR. This technique reshapes the reward targets used to train the reward predictor. Specifically, trajectories that incur costs in the executable prefix (the initial steps of a trajectory that are immediately executed before replanning) are penalized. By training the reward predictor on these relabeled targets, the method discourages the reward gradient from steering samples toward regions where high returns are achieved at the expense of immediate safety violations.

Theoretical Analysis
The paper provides a first-order sampling-time analysis demonstrating that FTR suppresses reward-induced cost drift. Under a "prefix-restorative alignment condition," where the gradient of prefix infeasibility is positively aligned with the cumulative cost gradient, the FTR correction introduces a negative term to the cost drift equation. This ensures that the cost deviation induced by FTR-guided sampling is smaller than that of raw reward-guided sampling relative to the cost-conditioned baseline.

Key Contributions

Decoupled Guidance Architecture: A novel framework that separates safety enforcement (via CFG on cost limits) from reward optimization (via reward gradients), treating them as non-interchangeable objectives.
Feasible Trajectory Relabeling (FTR): A mechanism to reshape reward targets to prevent reward gradients from driving the policy into unsafe regions, specifically addressing the correlation between high reward and high cost in offline data.
Adaptive Sampling: A method that adapts to deployment-time cost limits (including dynamic changes within an episode) without requiring policy retraining or new data collection.

Experimental Results
Extensive evaluations were conducted on the DSRL benchmark, comprising 38 tasks from Safety-Gymnasium, Bullet-Safety-Gym, and MetaDrive.

Safety Compliance: SDGD satisfied the normalized cost constraint ( $\le 1$ ) on 36 out of 38 tasks (94.7%), outperforming all baselines (the next best satisfied 21 tasks).
Reward Performance: Among the methods that satisfied the safety constraints, SDGD achieved the highest reward on 21 tasks.
Adaptability: In experiments with varying cost limits (10, 20, 30) and time-varying limits within an episode, SDGD maintained safety while adjusting reward performance, whereas baselines either violated constraints at tight budgets or remained overly conservative at loose budgets.
Efficiency: The method achieved a decision efficiency of 2.1 Hz on an NVIDIA A6000 GPU, faster than prior diffusion-based methods.

Significance and Claims
The paper claims that SDGD represents a significant step forward in adaptive offline safe RL by addressing the limitations of sequential error accumulation and unreliable classifier guidance. By reinterpreting safety as a distributional restriction rather than a competing gradient objective, SDGD achieves robust safety compliance across diverse and dynamic cost budgets. The authors emphasize that their approach allows for reliable adaptation to changing safety constraints without retraining, a capability essential for safety-critical domains like robotics and autonomous driving. The work concludes that decoupling safety and reward guidance, combined with FTR, effectively resolves the trade-off between constraint satisfaction and performance optimization in offline settings.