💻 computer science

Position: Good Embodied Reward Models Need Bad Behavior Data

This position paper argues that the embodied AI community must prioritize collecting and utilizing "bad" robot data—such as failed, suboptimal, or hazardous behaviors—to train reward models that accurately align with human preferences and avoid over-rewarding unsafe or superficial task completions.

Original authors: Ran Tian, Yilin Wu, Andrea Bajcsy

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ran Tian, Yilin Wu, Andrea Bajcsy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to fold laundry or pour a glass of water. To teach it well, you need a "teacher" (a reward model) that can look at what the robot does and say, "Good job!" or "No, that was a mess."

This paper argues that our current robot teachers are failing because they have only ever seen perfect examples. They have never been shown what a disaster looks like.

Here is the breakdown of the paper's argument using simple analogies:

1. The Problem: The "Perfect Student" Bias

Currently, when we train these robot teachers, we only show them videos of robots doing tasks perfectly. It's like trying to teach a student how to drive a car by only showing them videos of professional drivers in perfect weather, never showing them a flat tire, a skid, or a near-miss.

Because the teachers have never seen "bad" behavior, they are overly optimistic.

The Result: If a robot knocks over a bowl while trying to pick up a cup, the teacher might still say, "Great job! You picked up the cup!" because it sees the cup moving. It misses the fact that the robot broke something else.
The Reality: The paper tested three top-of-the-line robot teachers. They all gave high scores to robots that were actually failing, being unsafe, or taking "cheats" to finish a task. As tasks got harder (like using a tool), the teachers got worse, essentially guessing randomly.

2. The Solution: We Need "Bad" Data

The authors say we need to stop throwing away the "failures." We need to collect and share videos of robots crashing, dropping things, or moving dangerously.

Think of it like a medical school. If you only study healthy patients, you won't know how to diagnose a disease. You need to study sick patients, too.

The Paper's Claim: Even a small amount of "bad" data (videos of robots failing) helps the teacher learn what not to reward.
The Experiment: The researchers tested this by showing a teacher a video of a robot failing (e.g., spilling nuts) alongside a text description of the error.
- Text only: Didn't help much. The teacher couldn't connect the words to the physical mess.
- Text + Video: Helped a little on simple tasks (like picking up an object).
- Text + Video + "Scorecard": This worked best. They gave the teacher a video of the failure plus a detailed scorecard showing exactly when and why the robot failed during the video. This allowed the teacher to learn to penalize the robot the moment it made a mistake, even if the rest of the task looked okay.

3. Why We Don't Have This Data Yet

You might ask, "Why don't we just record all the failures?"

The Barrier: In the digital world (like text chatbots), generating "bad" data is easy and cheap. You can just type nonsense.
The Robot World: In the real world, robots are physical. Making them fail often means breaking things, wasting time, or creating safety hazards. Also, most robotics labs are so focused on showing off success that they delete the "ugly" failure videos before anyone sees them.

4. The Call to Action

The authors are asking the robotics community to do four specific things:

Release the "Bad" Data: Stop hiding the failures. Labs and companies should share datasets of robots crashing, dropping things, or moving unsafely, just as they share success stories.
Fake the Failures (Synthetically): Since real crashes are expensive, we need better computer simulations that can generate realistic "what-if" failure scenarios (e.g., "What if the robot slips here?").
Decentralize Testing: Instead of one lab testing robots, we need many different places testing them in real-world conditions to catch more types of failures.
Better Tests for Teachers: We need new benchmarks to test the "teachers" themselves, ensuring they are actually learning from human feedback and not just guessing.

Summary

The paper argues that to build reliable robots, we must stop treating "failure" as a mistake to be hidden. Instead, we must treat failure data as a vital resource. Without seeing the "bad" behavior, our robot teachers will keep giving high scores to dangerous or sloppy robots, thinking they are doing a great job.

Technical Summary: Good Embodied Reward Models Need Bad Behavior Data

Problem Statement

The paper identifies a critical gap in the development of embodied reward models (RMs) for robotics. While reward models are central to the lifecycle of foundation models—enabling reinforcement learning (RL) post-training, test-time compute, and large-scale evaluation—current embodied RMs are systematically over-optimistic.

Unlike non-embodied domains (e.g., LLMs) where "bad" data (toxic language, logical fallacies) is abundant and cheap to generate, embodied data collection is constrained by wall-clock time and hardware safety. Consequently, existing robotics datasets and reward models are trained primarily on successful behaviors (expert demonstrations), with failures, hazardous interactions, and suboptimal executions filtered out or withheld. This data scarcity leads to reward models that fail to penalize behaviors humans would reject, such as:

Unsafe interactions (e.g., collisions).
Poor execution quality (e.g., spilling, dropping).
"Shortcut" strategies that superficially satisfy a task while violating implicit constraints.

The authors argue that without exposure to "bad" robot data, reward models cannot be properly calibrated, creating a widening gap between model capability and the ability to evaluate complex physical behaviors.

Methodology

Evaluation of State-of-the-Art (SOTA) Models

The authors evaluated three distinct families of open-sourced embodied reward models using the RoboArena benchmark, which provides real robot rollouts and human pairwise preference judgments across seven task categories of increasing complexity (from Pick/Place to Tool Use):

Preference-based (ReWind): Trained via synthetic negatives generated by perturbing expert trajectories.
Zero-shot VLM (GVL): Leverages Vision-Language Models to reason about frames and generate dense rewards without fine-tuning.
Fine-tuned VLM (Dopamine): Fine-tunes a pre-trained VLM on heuristically constructed task progress labels.

Evaluation Protocol:
Instead of relying on Value-Order Correlation (VOC), which only measures temporal progress, the authors used human pairwise preference rankings as the ground truth. They measured the preference ordering accuracy ( $A$ ), calculating the fraction of human-preferred pairwise comparisons correctly ranked by the reward model.

Intervention: Injecting "Bad" Data

To test the hypothesis that "bad" data improves calibration, the authors experimented with in-context learning using the GVL model. They progressively injected negative examples into the model's prompt:

Text-only: Concise textual summaries of failure modes distilled from human feedback.
Text + Video: Pairing failure descriptions with actual robot video snippets of the failure.
Text + Video + Dense Rewards: Adding dense, per-timestep reward labels derived via preference-guided self-distillation (selecting reward sequences that align with human A/B preferences).

Key Results

1. Systematic Over-Rewarding of Failures

Current SOTA models perform reasonably well on simple tasks (Pick/Place, accuracy ~0.72–0.77) but degrade significantly as task complexity increases.

Tool Use: Accuracy drops to 0.52–0.62, barely above random guessing.
Qualitative Failure: In visual examples, models assigned high rewards to trajectories where the robot collided with objects or spilled items, provided the robot made "progress" toward the goal. The models failed to distinguish between successful completion and unsafe or low-quality execution.

2. Impact of "Bad" Data Injection

Injecting negative data significantly improved the alignment of reward models with human preferences:

Text-only: Provided negligible improvement, suggesting VLMs struggle to ground abstract textual failure descriptions into specific physical behaviors.
Text + Video: Resulted in an ~8% accuracy improvement on simpler tasks. Visual grounding helped the model disambiguate "bad" behaviors (e.g., missing a grasp).
Text + Video + Dense Rewards: Yielded the largest gains, particularly on complex tasks like Tool Use and Pour Liquid, achieving an ~10% accuracy improvement. Dense reward traces provided a concrete calibration signal for nuanced constraints (e.g., safety violations occurring mid-trajectory) that text or single-frame video could not capture.

Key Contributions

Empirical Demonstration of Data Bias: The paper provides experimental evidence that current embodied reward models are systematically over-optimistic due to a lack of negative training data, failing to penalize unsafe or suboptimal behaviors that humans would reject.
Quantification of the Gap: Through the RoboArena benchmark, the authors quantify the degradation in reward model accuracy as task complexity increases, showing that models approach random guessing on nuanced tasks.
Validation of "Bad" Data Utility: The study demonstrates that even modest exposure to real robot failure data (specifically video and dense reward labels) significantly improves reward model calibration and reduces false positives.
Community Call to Action: The authors propose four specific actions to address the data gap:
- Release curated failure datasets: Industry and academia must stop filtering out negative data and release large-scale datasets of robot failures.
- Synthetic generation: Develop high-fidelity simulators and video generation models to create diverse, interactive failure trajectories.
- Decentralized evaluation: Scale physical evaluation systems (like RoboArena) with standardized protocols to gather diverse human feedback.
- New benchmarks: Create benchmarks for reward models that rely on dense, human-annotated labels rather than sparse or heuristic proxies.

Significance and Claims

The paper argues that the path to reliable embodied AI is blocked not just by model architecture, but by data curation practices. The authors claim that the community's reliance on "expert-only" datasets has created a blind spot for failure modes.

The significance of this work lies in its assertion that calibration requires contrast. Just as LLMs require diverse negative examples to learn safety and alignment, embodied reward models require "bad" robot data to learn the fine-grained physical consequences of failure. The paper posits that without a deliberate shift toward collecting and utilizing dangerous, failed, and noisy robot data, the deployment of foundation models in physical environments will remain prone to unsafe and suboptimal behaviors that reward models fail to detect.