💻 computer science

Robot Critics that Sweat the Small Stuff

This paper introduces a method to fine-tune vision-language model critics using pairwise progress supervision from success and failure rollouts, enabling them to detect subtle visual differences and guide robot policies to significantly improve success rates in both real-world and simulated tasks.

Original authors: Sruthi Sudhakar, Junbang Liang, Sreehari Rammohan, Pavel Tokmakov, Richard Zemel, Carl Vondrick

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sruthi Sudhakar, Junbang Liang, Sreehari Rammohan, Pavel Tokmakov, Richard Zemel, Carl Vondrick

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to perform a delicate task, like stacking a cup on a block or picking up a Lego piece. You have a "Base Robot" (the policy) that has watched many videos of humans doing the task successfully. It tries to copy them. But here's the problem: the Base Robot is a bit clumsy. It might get almost the right position, but miss by a millimeter, causing the cup to tip over. It doesn't know why it failed because it only ever saw the "happy endings" in its training videos.

This paper introduces a solution: a Robot Critic. Think of this Critic as a hyper-observant, slightly obsessive coach standing right next to the robot, watching every move in real-time.

Here is how the system works, broken down into simple steps:

1. The "What-If" Simulator (The Crystal Ball)

Before the robot actually moves its arm, the system asks a "Crystal Ball" (an AI video generator) to imagine what would happen if the robot tried different moves.

The robot thinks of 5 or 10 different ways to reach for the object.
The Crystal Ball quickly generates short videos showing the future of each of those 5 or 10 attempts.
Now, instead of just guessing, the robot can see 5 different potential futures side-by-side.

2. The Critic's Job (The "Sweat the Small Stuff" Coach)

This is where the paper's main innovation comes in. The authors trained their Critic to be incredibly picky.

Old Critics: Previous AI coaches could tell the difference between "Robot holding a cup" and "Robot holding a cup perfectly." But they couldn't tell the difference between "Robot holding a cup perfectly" and "Robot holding a cup almost perfectly (but about to drop it)."
The New Critic: This Critic was trained on a special diet of data. It didn't just watch successful videos; it watched failures too. It saw videos where the robot dropped the cup, missed the Lego, or knocked things over.
The Training: The researchers showed the Critic pairs of images: "Look at this successful moment" vs. "Look at this failure moment that happened at the exact same time." The Critic learned to spot the tiny, 1-millimeter misalignments that lead to disaster.

3. The Selection Process (Picking the Winner)

Once the Crystal Ball shows the 5 potential futures, the Critic looks at them all. It compares them pairwise (like a tournament bracket).

"Future A looks like it will succeed."
"Future B looks like the gripper is slightly crooked; that's a failure."
The Critic votes for the future that looks the most successful and tells the robot, "Do this one."

Why This Matters

The paper tested this on real robots in a lab and in computer simulations.

The Result: By using this "obsessive" Critic to choose the best move before making it, the robots got significantly better at their jobs.
The Numbers: In the real world, the robots succeeded 11% more often than they did without the Critic. In simulations, they improved by 5.9%.
The Key Insight: The paper found that if you only train the Critic on success stories, it fails to spot the subtle mistakes. It must see the failures to learn what to avoid. It's like a driver who has only ever seen perfect driving lessons; they won't know how to correct a skid until they've seen what a skid looks like.

Summary Analogy

Imagine you are playing a video game.

The Base Robot is a player who has memorized the winning moves but gets stuck on hard levels because they make tiny mistakes.
The Crystal Ball is a "Save Scumming" feature that lets you preview 5 different paths forward.
The Critic is a super-expert guide who has played the game thousands of times, including all the "Game Over" screens. They look at your 5 previewed paths and say, "Don't go left, that looks like a trap. Go right; that path has the perfect alignment."

By adding this expert guide who knows exactly what failure looks like, the robot stops making tiny, fatal errors and starts completing tasks much more reliably.

Technical Summary: Robot Critics that Sweat the Small Stuff

Problem Statement

Large Vision-Language Models (VLMs) possess extensive priors about the physical world, making them promising candidates for acting as "critics" to steer robot policies during inference. However, a critical gap exists in fine-grained visual reasoning. While off-the-shelf VLMs and existing progress detection methods can distinguish between initial and final task states or identify high-level subtasks, they struggle to discriminate between subtle visual differences that separate success from failure.

For instance, a 1mm gripper misalignment may appear visually similar to a successful grasp but results in task failure. Current methods fail in this "fine-grained regime" where the time interval between frames is narrow. In experiments, baseline models like Gemini 3 achieved only 54.7% accuracy in distinguishing success from failure, while specialized progress detection methods (ROVER, ProgressLM) achieved 72.1% and 61.0% respectively, barely outperforming random chance (50%) in these subtle scenarios. This limitation prevents VLMs from effectively guiding stochastic robot policies away from failure modes in closed-loop execution.

Methodology

The authors propose a framework to fine-tune VLMs specifically for pairwise progress and failure discrimination, integrating them into a closed-loop robot policy via an action-conditioned video model.

1. Training VLM Critics

The core innovation is the construction of a pairwise progress supervision dataset derived from robot rollouts.

Data Construction: The system generates paired trajectories from the same initial conditions: one successful ( $S_s$ ) and one failure ( $S_f$ ).
Success-Success Pairs: Consecutive frames from a successful trajectory are paired to teach the model gradual forward progress (labeling the later frame as having more progress).
Success-Failure Pairs: Frames from a failure trajectory are paired with frames from a successful trajectory at the same time step. The successful frame is labeled as having more progress. This is crucial for teaching the critic to recognize policy-specific failure modes (e.g., specific ways the robot fails given its current policy) rather than just generic task failure.
Fine-Tuning: A VLM (specifically Qwen2.5-VL-7B) is supervised fine-tuned to predict a discrete decision $c(v_i, v_j) \in \{+1, -1\}$ indicating which frame shows greater progress toward the goal, minimizing cross-entropy loss.

2. In-the-Loop Policy Steering

The fine-tuned critic is integrated into the robot's execution loop to improve test-time performance:

Sampling: Given an observation $x_t$ , the base stochastic policy (e.g., Diffusion Policy or GR00T N1.5) samples $N$ candidate action sequences.
Diversity Pruning: A greedy farthest-point sampling (FPS) strategy selects a diverse subset of $K$ candidates.
Future Synthesis: An action-conditioned video model (HunyuanVideo 1.5, a Diffusion Transformer) synthesizes the terminal visual state $x_{t+H}$ for each candidate action sequence.
Critic Selection: The critic performs all pairwise comparisons among the $K$ generated terminal states. The candidate whose terminal state accumulates the most "votes" (indicating greater progress) is selected for execution.
Re-planning: The selected action is executed, and the process repeats until task completion.

Key Contributions

Fine-Grained Critic Training: A method to fine-tune VLMs using paired success and failure rollouts, enabling them to detect subtle visual differences (e.g., millimeter-scale misalignments) that standard VLMs miss.
Policy-Specific Failure Learning: Demonstrating that training on failure data specific to the deployed policy is essential. Critics trained only on successful demonstrations fail to recognize the specific failure modes of the policy, leading to a significant drop in accuracy when distinguishing success from failure.
Closed-Loop Integration: A complete pipeline combining stochastic policy sampling, generative video prediction, and VLM-based pairwise comparison to steer robot actions in real-time.
Generalization: The approach generalizes to out-of-distribution (OOD) objects and unseen tasks, suggesting a viable pretraining strategy for future task learning.

Experimental Results

The authors evaluated the method on both real-world hardware and simulation environments.

Accuracy Benchmarks

RoboCasa (Simulation): The fine-tuned critic achieved 93.6% average accuracy on in-distribution tasks and 91.4% on out-of-distribution tasks, significantly outperforming baselines like Gemini 3 (54.7%) and ROVER (72.1%).
LBM (Real-World): On the LBM dataset, the method achieved 90.0% average accuracy, compared to 62.9% for ROVER and 55.4% for Gemini 3.
Fine-Grained Regime: As the temporal interval between compared frames decreased (making visual differences subtler), baseline performance dropped from 73% to 56%. The proposed method maintained high performance with only a 2% drop, demonstrating robustness in fine-grained discrimination.

Policy Performance Improvement

Real-World Tasks: Using the critic-in-the-loop improved the success rate of the base Diffusion Policy by an average of +11%. Specific gains included an 18% increase in "Stacking" tasks and a 2.3x improvement in mean Intersection over Union (mIoU) for "Push-Bowl" tasks.
Simulation Tasks (RoboCasa365): The method improved the success rate of the GR00T N1.5 policy from 43.6% to 49.5%, an average gain of +5.9%. Significant improvements were observed in tasks requiring precise manipulation of buttons, doors, and levers.

Significance and Claims

The paper claims that while VLMs provide valuable semantic priors, their utility in robotics is limited by their inability to reason about fine-grained visual progress and policy-specific failures. By explicitly training critics on the nuances of success and failure trajectories, the authors demonstrate that VLMs can be transformed into effective "sweaters of the small stuff."

The significance lies in the ability to improve stochastic policies at test-time without retraining the policy itself. The method acts as a filter, selecting the most promising action candidates from a distribution of possibilities. The authors emphasize that recognizing failure is as important as recognizing progress; without failure data, critics cannot effectively steer policies away from the "needle in the haystack" of successful actions hidden among many failure modes. This approach bridges the gap between the high-level reasoning of foundation models and the low-level precision required for successful robot manipulation.