⚡ electrical engineering

Density-Ratio Weighted Behavioral Cloning: Learning Control Policies from Corrupted Datasets

This paper introduces Density-Ratio Weighted Behavioral Cloning, a robust offline reinforcement learning method that leverages a small clean reference set to estimate and apply density-ratio weights, thereby enabling the learning of near-optimal control policies from datasets heavily contaminated by adversarial attacks or low-quality samples without prior knowledge of the corruption mechanism.

Original authors: Shriram Karpoora Sundara Pandian, Ali Baheri

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shriram Karpoora Sundara Pandian, Ali Baheri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to drive a car. You have a massive video library of driving footage to show the robot. This is "offline learning"—the robot learns from a static dataset rather than driving around and crashing in real life.

However, there's a problem: someone has tampered with the video library.

Some videos show the car driving off cliffs (bad data).
Some videos have the steering wheel spinning wildly for no reason (corrupted data).
Some videos show the car driving perfectly, but the "score" at the end says "0 points" instead of "100 points" (poisoned rewards).

If you just show all these videos to the robot equally, it will learn to drive off cliffs or spin its wheels. It will think the bad stuff is normal because it's in the dataset.

This paper introduces a new method called Weighted Behavioral Cloning to solve this. Here is how it works, using simple analogies:

1. The "Trusted Librarian" (The Reference Set)

The researchers assume you have a tiny, special folder of videos that you know are 100% perfect. Maybe these are recordings from a professional test drive where engineers verified every move. They call this the Reference Set. It's small compared to the massive, messy library, but it is pure gold.

2. The "Detective" (The Discriminator)

The method uses a smart AI "detective" (called a discriminator). Its only job is to look at a video from the messy library and ask: "Does this look like the videos in my trusted folder, or does it look weird?"

If the video looks like the trusted expert driving, the detective says, "Yes, this is good!"
If the video shows the car driving into a wall or the steering wheel spinning randomly, the detective says, "No, this is suspicious."

3. The "Weighted Lesson" (The Magic Trick)

In normal learning, the robot treats every video as equally important. In this new method, the robot listens to the detective.

Good videos: The detective gives them a heavy weight. The robot pays close attention to these.
Bad videos: The detective gives them a tiny weight (or almost zero). The robot basically ignores them, even though they are still in the dataset.

The robot doesn't need to know how the data was corrupted (whether it was a sensor glitch, a hacker, or a typo). It just needs to know: "Does this look like the expert I trust?"

4. The Results: "The Unbreakable Student"

The researchers tested this on computer simulations of robots (like a cheetah running or a walker balancing). They corrupted the data in four nasty ways:

Action Poisoning: Changing the robot's moves to be random.
State Poisoning: Blurring the robot's "eyes" (sensor data).
Transition Poisoning: Making the video jump around so the cause-and-effect is broken.
Reward Poisoning: Lying about how well the robot did.

The Outcome:
Even when 80% to 100% of the data was corrupted (meaning almost the entire library was trash), the robot using this new method still learned to drive almost perfectly.

Old methods (like standard Behavioral Cloning) crashed and burned, learning to drive off cliffs.
This new method kept the robot stable and efficient, ignoring the trash and focusing only on the few clean videos it could find.

Why This Matters

The paper proves mathematically that this method works even if the "trash" in the dataset is huge. It's like having a student who can sit in a room full of people shouting nonsense, but because they have one trusted teacher whispering the right answers, they can still pass the exam with flying colors.

In short: This paper teaches robots how to ignore bad data by comparing everything against a tiny, verified sample of "good" data, ensuring they learn the right skills even when the training data has been sabotaged.

Technical Summary: Density-Ratio Weighted Behavioral Cloning

Problem Statement
Offline reinforcement learning (RL) enables policy optimization from fixed datasets, offering a critical solution for safety-critical domains like autonomous vehicles and industrial robotics where online exploration is infeasible. However, real-world datasets are frequently compromised by data poisoning arising from sensor faults, cyber-physical attacks, annotation errors, or adversarial manipulations. Standard Behavioral Cloning (BC) and offline RL algorithms implicitly treat all trajectories as equally reliable, blending clean and corrupted samples during optimization. This leads to degraded policy performance, instability, or safety violations, particularly under moderate-to-severe poisoning where subtle anomalies can disrupt control tasks such as trajectory tracking. Existing robustness studies often focus on specific corruption channels or assume known mixture distributions, leaving a gap in handling arbitrary, unknown contamination mechanisms in offline control datasets.

Methodology
The paper proposes Density-Ratio Weighted Behavioral Cloning (Weighted BC), a robust imitation learning framework designed to learn from contaminated offline datasets without prior knowledge of the corruption mechanism. The core methodology relies on three stages:

Reference Set Utilization: The approach assumes access to a small, verified, and disjoint reference set ( $D_{ref}$ ) of clean expert trajectories, distinct from the potentially contaminated main dataset ( $D$ ).
Discriminator-Based Density Ratio Estimation: A binary discriminator $d_\phi$ is trained to distinguish between trajectories from the clean reference set (class 1) and the contaminated main dataset (class 0). The discriminator minimizes a binary cross-entropy loss. Under balanced sampling, the density ratio $r(\tau) = p_{clean}(\tau)/p(\tau)$ is recovered as $d_\phi(\tau) / (1 - d_\phi(\tau))$ .
Weighted Objective: These estimated ratios are clipped to a deterministic range $[\epsilon, C]$ to ensure numerical stability and used as weights in the Behavioral Cloning objective. The empirical loss becomes:
$L_{WBC}(\theta) = \frac{1}{N} \sum_{i=1}^{N} w_i \sum_{t=0}^{T-1} -\log \pi_\theta(a_t^{(i)} | s_t^{(i)})$
where $w_i = \text{clip}(r(\tau_i), \epsilon, C)$ . This weighting scheme prioritizes trajectories resembling the clean expert behavior while down-weighting or discarding anomalous segments.

The method is evaluated against four specific poisoning protocols applied to continuous control benchmarks:

Reward Poisoning: Inverting positive rewards to penalize good behavior.
State Poisoning: Adding Gaussian noise to states to simulate sensor failure.
Transition Poisoning: Shuffling next-state observations to disrupt causal linkages.
Action Poisoning: Perturbing actions with Gaussian noise.

Key Contributions
The paper makes three primary contributions:

Algorithmic Proposal: It introduces Weighted BC, which utilizes small verified reference sets to identify and down-weight corrupted trajectories via discriminator-based weighting. This enables robust policy learning without requiring knowledge of the specific corruption mechanism.
Theoretical Foundations: The authors establish theoretical guarantees, including uniform clean-risk approximation bounds and excess clean-risk guarantees for the learned policy. Crucially, these bounds are shown to be independent of the contamination rate ( $\alpha$ ) under appropriate clipping thresholds, separating finite-sample errors, discriminator errors, and clipping bias.
Evaluation Framework: A comprehensive evaluation framework is established, incorporating various poisoning scenarios (action, state, transition, and reward) across continuous control benchmarks (D4RL), addressing a gap in existing literature that often assumes specific corruption types.

Results
Experiments were conducted on D4RL benchmarks (HalfCheetah, Ant, Hopper, Walker2d) with contamination levels ranging from 20% to 100% ( $\alpha \in \{0.2, \dots, 1.0\}$ ). Weighted BC was compared against Traditional BC, Batch-Constrained Q-learning (BCQ), and Behavior Regularized Actor-Critic (BRAC).

Robustness: Weighted BC maintained near-optimal performance even at high contamination ratios ( $\alpha \ge 0.8$ ), whereas baselines suffered significant degradation.
Action Poisoning: In this scenario, baselines collapsed (e.g., Traditional BC degraded to ~2,500 return vs. Weighted BC maintaining >10,000 in HalfCheetah at $\alpha=1.0$ ). The discriminator successfully identified action-corrupted trajectories as deviating from the behavioral manifold.
State and Transition Poisoning: Weighted BC showed consistent stability, while BCQ's VAE-based constraints failed to model corrupted dynamics, and BRAC exhibited instability under state corruption.
Reward Poisoning: Since BC relies on state-action pairs rather than reward signals, Weighted BC maintained 80-90% of baseline performance, while BRAC failed in several environments.
Retention: Weighted BC retained over 80% of clean performance up to 60% contamination across all poisoning types, significantly outperforming the linear degradation observed in Traditional BC.

Significance and Claims
The paper claims that Weighted BC provides a robust solution for offline RL in the presence of arbitrary data poisoning, a critical requirement for deploying learning-based controllers in safety-critical systems. By leveraging a small, vetted reference set, the method effectively filters corrupted data without needing to model the specific nature of the attack. Theoretical analysis confirms that the approach converges to the clean expert policy with finite-sample bounds that do not depend on the severity of contamination. Empirically, the method demonstrates superior resilience compared to state-of-the-art offline RL algorithms, particularly in high-contamination regimes where standard methods fail. The work emphasizes that trajectory-level assessment is essential for capturing temporal consistency and control-relevant anomalies, distinguishing it from methods that only address isolated state-action artifacts.