🤖 machine learning

Confounding Robust Continuous Control via Automatic Reward Shaping

This paper proposes a method to automatically learn reward shaping functions from offline datasets using a causal Bellman equation to enable robust continuous control in environments containing unobserved confounding variables.

Original authors: Mateo Juliani, Mingxuan Li, Elias Bareinboim

Published 2026-02-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mateo Juliani, Mingxuan Li, Elias Bareinboim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn how to drive a car, but you are being taught by a driver who is wearing high-tech augmented reality glasses. This instructor can see everything: the exact distance to the car in front, the precise grip of the tires on a wet road, and even the subtle movements of pedestrians around a corner.

However, you are sitting in the driver's seat with a foggy windshield and no dashboard. You can only see the road directly in front of you.

When the instructor says, "Turn left now!", you don't know if they are turning because there is a pothole you can't see, or because they saw a pedestrian you can't see. Because you lack their "privileged information," you might think turning left is a bad habit, when in reality, it was the only safe move. This is the problem of "Unobserved Confounding" in AI.

The Problem: The "Hidden Secret" Bias

In Reinforcement Learning (RL), we often train AI agents using "offline data"—essentially a library of recordings from previous attempts.

The problem is that these recordings often come from "expert" systems that had access to much better sensors than the new AI will have once it's actually deployed. If the expert's decisions were based on "secrets" (hidden data) that the new AI can't see, the new AI will learn the wrong lessons. It will try to mimic the actions without understanding the reasons, leading to a confused and clumsy robot.

The Solution: The "Optimistic GPS" (Automatic Reward Shaping)

The researchers from Columbia University proposed a clever way to fix this. Instead of just telling the AI, "Do what the expert did," they give the AI a special kind of "GPS" called Causal Reward Shaping.

Here is how their method works, using a metaphor:

1. The "What If?" Engine (The Causal Bellman Equation)
Instead of just looking at what the expert did, the researchers' math asks: "What if the expert had made a different choice?"
Imagine you're watching a video of a chef cooking. Instead of just copying their movements, you use a mental simulator to ask: "If the chef had added salt instead of sugar, would the dish have been better or worse?" By accounting for the "road not taken," the AI can mathematically estimate the "hidden secrets" that influenced the expert.

2. The "Encouragement System" (Potential-Based Reward Shaping)
Once the AI has this "What If?" engine, it creates a map of "Good States" and "Bad States."
Think of this like a video game that gives you "bonus points" just for moving in the right direction, even before you reach the finish line. If the AI moves into a position that looks like it's heading toward a high-value state (based on the expert's hidden knowledge), it gets a little "nudge" of extra reward. This keeps the AI motivated and prevents it from getting lost in the fog.

Why does this matter?

The researchers tested this on complex robotic tasks (like a "Hopper" robot trying to jump or a robotic hand trying to open a door).

The Old Way: When the robot's "eyes" were partially blocked, it would stumble, fall, or simply fail to learn how to move.
The New Way: Even with "foggy" sensors, the AI used the "Optimistic GPS" to realize, "Even though I can't see the pothole, the expert's movement suggests that staying in this lane is a high-value state."

The Result: The AI was able to perform much closer to a "perfect" robot with full vision, even though it was effectively "driving blind."

In Short:

This paper provides a way to teach AI from "expert" data without the AI becoming a confused mimic. It allows the AI to look at biased, imperfect history and extract the true logic behind the actions, making it much more robust when it finally faces the real, messy world.

Technical Summary: Confounding Robust Continuous Control via Automatic Reward Shaping

1. Problem Statement

The paper addresses a critical challenge in Offline-to-Online Reinforcement Learning (RL): the presence of unobserved confounders. In many real-world scenarios, offline datasets are collected by "privileged" agents (e.g., humans or robots with superior sensors) that observe more information than the learner will have during online deployment.

This mismatch creates a Confounded Markov Decision Process (CMDP). When the learner attempts to use this offline data, the unobserved variables (confounders) influence both the actions taken and the resulting rewards/transitions. Standard RL algorithms implicitly assume a standard MDP (No Unobserved Confounders - NUC), leading to biased value estimations, poor policy performance, and failure to generalize to the online environment where those hidden variables are unavailable.

2. Methodology

The authors propose a framework to automatically learn a robust reward shaping function from confounded offline data to guide online learning.

Causal Bellman Equation: The core theoretical contribution is a new Bellman Optimality Equation designed for stationary infinite-horizon CMDPs. Unlike the standard Bellman equation, this version incorporates an "uncertainty term" that accounts for the "road not taken." It uses the probability of the behavioral policy ( $P(x|s)$ ) and the probability of unselected actions ( $P(\neg x|s)$ ) to provide a tight upper bound on the optimal interventional state values ( $V^*$ ).
Neural Approximation: Since the causal equation is analytically intractable in continuous spaces, the authors design a neural learning algorithm (Algorithm 1) to approximate it. They use:
- Gaussian Policy Modeling: To estimate the propensity scores of actions.
- State Difference Modeling: To estimate the distribution of next states ( $\Delta s$ ) for actions not taken in the dataset.
- Reparameterization Trick: To ensure the entire learning process is differentiable for high-dimensional continuous control.
Potential-Based Reward Shaping (PBRS): The learned causal upper bound is used as a potential function ( $\phi$ ). By adding the shaped reward $r' = r + \gamma\phi(s') - \phi(s)$ , the authors inject informative signals into the online agent (specifically Soft-Actor-Critic, SAC) without altering the optimal policy of the original task.

3. Key Contributions

Theoretical Derivation: Proved that the Causal Bellman Equation converges to a unique fixed point that serves as a valid upper bound for optimal interventional state values.
Algorithmic Innovation: Developed a scalable neural architecture to approximate causal bounds in high-dimensional, continuous state-action spaces.
Policy Invariance: Demonstrated that using these causal bounds within the PBRS framework preserves the optimality of the original MDP while mitigating confounding bias.
Empirical Validation: Provided a robust testing suite using partially observable MuJoCo and Adroit environments.

4. Results

The method was evaluated against several baselines, including vanilla SAC, CQL-based shaping, T-REX-based shaping, and Recurrent SAC.

Superior Robustness: The Causal PBRS method consistently outperformed causally unaware baselines (CQL and T-REX) across multiple benchmarks. It achieved a normalized mean score of 1.29 and an IQM of 1.09 relative to the baseline.
Performance Recovery: In environments where critical dimensions were removed (e.g., the thigh joint angle in Hopper), the Causal PBRS method significantly recovered performance, sometimes doubling the return compared to the baseline.
Sensitivity to Confounding: The authors used the Randomized Conditional Independence Test (RCIT) to show that their method is most effective when the removed dimensions have a strong causal link to the reward (e.g., forward velocity).
Data Quality Resilience: The method proved robust to varying levels of offline data expertise (simple, medium, and expert), showing that even suboptimal data can be used to form useful shaping potentials.

5. Significance

This work represents a significant step toward causally-aware reinforcement learning. By moving beyond the assumption of unconfounded data, it provides a principled way to bridge the "reality gap" between privileged offline datasets and restricted online deployment. The ability to automatically derive reward signals that are robust to hidden variables makes this approach highly applicable to real-world robotics, where sensor limitations and biased demonstrations are the norm rather than the exception.