💻 computer science

DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation

The paper introduces DenseReward, a dense robotic reward model trained on a synthetically generated dataset of diverse failure modes that predicts fine-grained, frame-level rewards from visual and language inputs to overcome the limitations of sparse feedback and improve robotic manipulation policies.

Original authors: Yu Fang, Wanxi Dong, Jiaqi Liu, Yue Yang, Mingxiao Huo, Yao Mu, Huaxiu Yao, Li Erran Li, Daniel Szafir, Mingyu Ding

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yu Fang, Wanxi Dong, Jiaqi Liu, Yue Yang, Mingxiao Huo, Yao Mu, Huaxiu Yao, Li Erran Li, Daniel Szafir, Mingyu Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to put a ball in a basket. In the old days, you'd have to watch the whole movie of the robot trying, failing, and trying again, and then just say, "Good job!" or "Bad job!" at the very end. That's like giving a student a test and only telling them their final grade after they've already forgotten every question they answered. The robot gets no clue why it failed or how it was doing halfway through.

Enter DenseReward, a new system that acts like a super-attentive coach who whispers a score after every single move the robot makes. Instead of a simple "pass/fail," it gives a number between 0 and 1, telling the robot exactly how close it is to success right now.

The Problem: The "Fake Failures" Trap

The big hurdle for teaching robots this way is that you need a massive library of examples showing everything that can go wrong. You need to see the robot crash into a table, drop the ball, miss the basket, or get stuck.

Usually, researchers tried to make these "failures" by taking a successful video and just cutting it off early or pretending it failed. But the authors of this paper argue that this is like trying to learn how to drive by watching a video of a car that just stopped moving; it doesn't teach you what a real crash feels like. These "fake" failures don't capture the messy, physical reality of a robot actually dropping an object or colliding with a wall.

The Solution: A Robot "Failure Factory"

To fix this, the team built an automated factory in a computer simulation. They didn't ask humans to break things manually (which is slow and boring). Instead, they programmed the robot to go through five specific stages: Reach, Grasp, Lift, Move, and Place.

Then, they introduced "targeted perturbations"—basically, they gave the robot a little nudge to make it mess up on purpose.

They made the robot aim its hand slightly off, causing a Miss.
They told it to ignore obstacles, causing a Collision.
They shook the robot while it was holding the object, causing a Fall.
They even made the robot crash and then figure out how to get back on track, creating a Recover scenario.

By doing this automatically, they generated a dataset of 27,000 episodes (that's 27k!) covering all these different ways a robot can fail, along with a precise score for every single frame of video.

The Star Player: DenseReward

Using this massive dataset, they trained a model called DenseReward. Think of this model as a robot's "sixth sense" for progress. When you give it a task (like "put ball in basket"), a picture of the current scene, and a few pictures of what happened just before, it instantly predicts a score.

If the robot is moving smoothly toward the basket, the score goes up.
If the robot bumps into the table, the score drops.
If the robot drops the ball but then picks it up again, the score dips and then climbs back up.

The paper shows that this model is much better at guessing these scores than other smart AI models (like general-purpose vision-language models) or older reward systems. In tests, the new model's predictions were off by only 0.081 on average, while other models were off by nearly 0.30. That's a huge difference in precision.

Does it Actually Work?

The authors didn't just stop at making a good guesser; they tested if this "coach" could actually help the robot learn better.

In the Simulation: They used the scores to guide a Model Predictive Control (MPC) system. Instead of guessing the next move, the robot looked at 28 possible moves, asked DenseReward which one looked best, and picked that one. The result? The robot got much closer to the target objects (reducing the distance to about 0.229 on average) compared to other methods.
In the Real World: They took a robot arm in a real lab and tried to teach it to stack cups and put a ball in a basket. Without the dense feedback, the robot only succeeded 40% of the time with the cups. But when they let DenseReward guide the learning process, the success rate jumped to 80%. For the ball-in-basket task, it went from 30% to 70%.

What the Paper Says (and Doesn't Say)

The authors are careful to say that while these results are impressive, they are based on specific simulations and a limited set of real-world tasks. They don't claim this solves every robot problem forever. They explicitly rule out the idea that "fake" failures (just cutting up successful videos) are good enough; they insist that physically realistic failure data is necessary.

They also suggest that this approach is a practical path forward, but they admit there's still work to do. They plan to tackle even harder tasks, like using tools or doing long, complex sequences of actions, and they hope to eventually include human feedback to make the rewards even better.

In short, DenseReward suggests that if you want a robot to learn fast, you need a coach that doesn't just wait until the end of the game to give a grade, but one that knows exactly how the robot is doing at every single step—and that coach needs to have seen a lot of real, messy failures to understand what "doing well" actually looks like.

Technical Summary: DenseReward

Problem Statement

Reinforcement learning (RL) offers a pathway for robotic policies to surpass the performance limits of imitation learning by engaging in trial-and-error interaction. However, the practical adoption of RL in robotic manipulation is bottlenecked by the lack of reliable vision-language reward models that provide dense and informative feedback. Current approaches face two primary challenges:

Data Scarcity for Failure Modes: Acquiring diverse failure data at scale is difficult. Human teleoperation is laborious, while existing data augmentation methods (e.g., truncating successful episodes or relabeling suboptimal rollouts) generate "pseudo-failures" that fail to capture the complex physical dynamics of real-world failures, such as collisions, missed grasps, object drops, and recovery behaviors.
Sparse Reward Signals: Most existing reward models predict sparse, binary, or trajectory-level labels (success/failure only at the end of an episode). This lack of per-timestep feedback creates severe credit assignment problems, making policy optimization sample-inefficient and difficult to scale for long-horizon tasks.

Methodology

The authors introduce DenseReward, a dense vision-language reward model designed to predict fine-grained, per-timestep reward scores based on visual observations and language instructions. The methodology consists of three core components:

1. Automated Data Generation Pipeline

To address the scarcity of diverse failure data without human labeling, the authors developed an automated pipeline that synthesizes physically realistic trajectories in simulation.

Phase Decomposition: Manipulation tasks are decomposed into five canonical phases: Reach, Grasp, Lift, Move, and Place.
Targeted Perturbations: By applying specific perturbations at different execution stages, the system induces six distinct trajectory types:
- Success: Unperturbed execution.
- Collision: Motion planned without collision avoidance, causing contact with the environment.
- Miss: Grasp target pose is offset, causing the gripper to close in mid-air.
- Fall: Random rotation perturbations during the "Move" phase cause the object to drop.
- Smooth: Gaussian joint noise is injected to create suboptimal, jittery motions.
- Recover: The robot encounters a collision but successfully replans and completes the task.
Dense Labeling: Instead of binary labels, the system assigns per-timestep reward scores ( $r_t \in [0, 1]$ ) that reflect task progress. The reward curve rises with progress, peaks at successful milestones, and decays or drops upon failure events (e.g., collisions or drops), capturing partial completion and recovery dynamics.

2. Dataset Construction

Using this pipeline, the authors curated a dataset of 27,000 episodes spanning diverse scenes and objects from DROID (real-world), RoboSuite, Isaac Sim, and LIBERO. The dataset includes both successful and failure trajectories with dense frame-level reward labels and failure mode annotations.

3. DenseReward Model Training

The model is built upon Qwen3-VL-4B-Instruct, a vision-language model. It is fine-tuned to take a language instruction, the current visual observation, and historical frames as input, outputting a scalar reward score that estimates task progress at the current timestep. The model is trained with dense frame-level supervision, retaining three decimal places to learn fine-grained differences in progress (e.g., moving closer to a target vs. recovering from a fall).

Key Contributions

Automated Failure Synthesis: A novel pipeline that generates physically realistic failure trajectories (collisions, misses, drops, recoveries) with dense reward labels, eliminating the need for manual annotation and overcoming the limitations of pseudo-failure data.
DenseReward Dataset and Model: The release of a 27k-episode dataset and a trained vision-language reward model capable of identifying execution states (success or specific failure modes) and estimating fine-grained task progress.
Downstream Efficacy: Demonstration that the model significantly outperforms general-purpose VLMs and existing robotic reward models in dense reward prediction and provides effective guidance for downstream Model Predictive Control (MPC) and Reinforcement Learning (RL).

Experimental Results

Reward Prediction Accuracy

In quantitative evaluations across DROID, Isaac Sim, RoboSuite, and LIBERO, DenseReward achieved a mean absolute error (MAE) of 0.081, significantly outperforming strong baselines like Qwen3-VL-8B (0.293) and RoboReward-8B (0.230). Qualitative analysis showed that DenseReward produces reward curves that align more closely with actual task progress compared to the noisy or binary predictions of baselines.

Model Predictive Control (MPC)

In an MPC setting where candidate actions are scored to select the best execution, DenseReward-guided control reduced the average minimum distance between the object and gripper to 0.229 across three manipulation tasks (can, cup, lemon). This outperformed RoboReward and VLAC baselines, indicating superior action selection capabilities.

Reinforcement Learning (RL) Fine-tuning

Simulation (LIBERO): When used to provide dense intermediate rewards for PPO fine-tuning of a $\pi_0$ policy, DenseReward improved success rates on LIBERO-Spatial and LIBERO-10 suites compared to sparse-reward baselines, while matching performance on LIBERO-Object.
Real-World (DROID Platform): Using DSRL to adapt a frozen $\pi_0$ policy on a Franka arm, DenseReward guidance increased success rates from 40% to 80% for "stack the cups" and from 30% to 70% for "put ball in basket," demonstrating effectiveness with limited real-world rollout budgets.

Ablation Studies

Failure Data: Removing failure trajectories from training increased MAE from 0.0809 to 0.1312, confirming that failure data is critical for learning fine-grained reward signals.
Historical Context: Using two historical frames provided the optimal trade-off (MAE 0.081), whereas using three frames slightly degraded performance, suggesting diminishing returns from excessive history.

Significance and Claims

The paper claims that DenseReward addresses the critical bottlenecks in scaling reinforcement learning for robotic manipulation by providing a practical, failure-aware, and dense reward modeling framework. By synthesizing diverse failure modes automatically, the authors enable the training of reward models that understand not just success, but the nuances of partial progress and recovery. The results suggest that failure-aware dense reward modeling is a viable path toward scalable RL, offering a significant improvement over sparse binary rewards and general-purpose VLMs for guiding policy optimization in both simulated and real-world environments. The authors release the dataset, models, and evaluation suite to support further research in this domain.