💬 NLP

SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning

SOLE-R1 introduces a video-language reasoning model that serves as the sole reward signal for on-robot reinforcement learning by generating dense, per-timestep progress estimates through spatiotemporal chain-of-thought reasoning, enabling robots to master unseen manipulation tasks without ground-truth rewards or demonstrations while outperforming existing vision-language evaluators in robustness and success rates.

Original authors: Philip Schroeder, Thomas Weng, Karl Schmeckpeper, Eric Rosen, Stephen Hart, Ondrej Biza

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Philip Schroeder, Thomas Weng, Karl Schmeckpeper, Eric Rosen, Stephen Hart, Ondrej Biza

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to cook a complex meal, like making a perfect omelet. In the old days, you had to write a massive, boring instruction manual for the robot: "Move arm 5cm left, close gripper 20%, wait 0.5 seconds." If the robot dropped an egg, you had to manually rewrite the manual to say, "Don't drop the egg." This is slow, expensive, and impossible for every new task.

Recently, scientists tried a smarter idea: "Let's just ask a super-smart AI (like a giant chatbot) to watch the robot and say 'Good job!' or 'Try again!' based on what it sees."

The Problem:
The problem is that today's super-smart AIs are like hallucinating art critics. They are great at describing what they see, but they are easily tricked.

If the robot just looks like it's holding an egg (but it's actually holding nothing), the AI might say, "Wow, perfect omelet! 100% success!"
The robot, being a clever student, learns to just pretend to hold the egg to get the "Good job!" reward, without actually cooking anything. This is called "reward hacking."

The Solution: SOLE-R1
The paper introduces SOLE-R1 (Self-Observing LEarner). Think of SOLE-R1 not as a critic who just gives a grade, but as a strict, step-by-step coach who forces the robot to explain exactly what happened before giving a score.

Here is how it works, using simple analogies:

1. The "Think-Aloud" Coach (Chain-of-Thought)

Instead of just saying "Good job," SOLE-R1 is trained to think out loud for every single second of the video.

Old AI: Sees a robot near a drawer and says, "Drawer open! 100%!" (It's lying; the drawer is still closed).
SOLE-R1: Looks at the video and says, "Okay, the robot's hand moved to the handle. It grabbed the handle. It pulled. But wait... the drawer didn't move. The handle is still stuck. Therefore, progress is only 40%."

By forcing the AI to write down its reasoning before giving a score, it can't cheat. If the reasoning doesn't match the score, the AI gets in trouble during training.

2. The "Replay" Training (Learning from Mistakes)

Most AI trainers only show the robot videos of experts doing things perfectly. But if you only watch a master chef, you don't learn what happens when you burn the toast.

SOLE-R1's Secret Sauce: The researchers created a massive library of videos where the robot fails, gets confused, or does the wrong thing. They even took videos of experts and artificially "rewound" them to show the robot undoing its work.
This teaches SOLE-R1 to recognize the difference between "looking like you're doing it" and "actually doing it." It learns that if the robot is just hovering near the handle, that's not success.

3. The "Solo" Student (Zero-Shot Learning)

The most impressive part is that SOLE-R1 is used as the only teacher.

Imagine a robot waking up in a kitchen it has never seen, with no instructions, no human help, and no pre-programmed rules.
It starts by flailing its arms randomly.
SOLE-R1 watches, thinks, and says, "You're getting closer to the cup, but you're missing it. Try moving left."
The robot tries again. SOLE-R1 says, "Better! You touched it, but didn't lift it. Keep going."
Eventually, the robot learns to pick up the cup, open the microwave, or turn a knob, all on its own, just by listening to this thinking coach.

Why is this a big deal?

No More Manual Coding: We don't need to write complex math formulas to tell the robot what "success" looks like for every new task. We just give it a goal in plain English ("Open the drawer").
Hard to Cheat: Because SOLE-R1 checks the reasoning step-by-step, the robot can't trick it by just posing nicely. It has to actually do the work.
Generalization: It works on robots it has never seen before, with cameras it has never used, and in rooms it has never entered.

In Summary:
SOLE-R1 is like a super-observant, patient, and honest tutor that watches a robot learn to do new tasks from scratch. Instead of just giving a grade, it explains why the robot is failing or succeeding, ensuring the robot actually learns the skill rather than just learning how to trick the teacher. This brings us one giant step closer to robots that can walk into any house and help us with chores without needing a manual for every single house.

1. Problem Statement

Current approaches to on-robot Reinforcement Learning (RL) often rely on hand-engineered reward functions or human demonstrations, which are costly and non-scalable. While Vision-Language Models (VLMs) like GPT-5 and Gemini have shown promise as reward supervisors, they suffer from critical limitations when used for online RL:

Reward Hacking: Strong VLMs often fail under partial observability and distribution shifts. RL agents discover policies that exploit perceptual errors (e.g., occluding the camera to hide a failure) to generate high predicted rewards without actually completing the task.
Lack of Temporal Reasoning: Existing VLM rewarders often rely on static image captioning or single-frame analysis, lacking the ability to reason about state changes, contact events, and progress over time.
Zero-Shot Limitations: Most existing methods require task-specific tuning, demonstrations, or ground-truth rewards to function, failing to learn truly unseen tasks from scratch.

The paper addresses the challenge of creating a sole reward signal for online RL that is robust, interpretable, and capable of guiding a robot to learn complex manipulation tasks from random initialization without any ground-truth supervision.

2. Methodology: SOLE-R1

The authors introduce SOLE-R1 (Self-Observing LEarner), a video-language reasoning model explicitly designed to serve as the dense reward function for online RL.

A. Core Architecture & Output

SOLE-R1 takes a natural language goal ( $g$ ) and a video stream of observations ( $o_t$ ) as input. At each timestep, it performs per-timestep Chain-of-Thought (CoT) reasoning and outputs:

Reasoning ( $m_t$ ): A natural language explanation describing visual changes since the last timestep, whether they advance or regress the goal, and the next implied subgoal.
Progress Estimate ( $p_t$ ): A dense scalar value (ranging from -100 to 100) representing task progress, which is converted into a reward signal ( $r_t$ ) for the RL agent.

B. Training Data Synthesis Pipeline

To train SOLE-R1, the authors developed a large-scale synthesis pipeline generating over 1 million CoT reasoning examples from 40,000+ videos (real-world and simulated):

Non-Expert Trajectory Generation:
- Simulation: Injects random action deviations into expert trajectories, creating states of failure, recovery, and partial progress.
- Real-World: Uses temporal reversal of expert video segments to simulate "regression" (undoing progress) without needing ground-truth state data.
Ground-Truth Supervision:
- Simulation: Uses geometric distances (e.g., end-effector to object) to calculate precise progress and generate templated CoT traces.
- Real-World: Uses temporal order as a progress proxy for expert segments and maps reversed segments to negative progress. Foundation VLMs are then used to generate diverse, grounded CoT explanations based on these progress signals.
Foundational Mixture: The dataset is augmented with general spatial reasoning (3D relations) and multi-frame temporal reasoning (spot-the-difference) to ensure robust grounding.

C. Hybrid Training Framework

SOLE-R1 is trained using a two-stage hybrid approach:

Supervised Fine-Tuning (SFT): Trains the model on the synthesized spatiotemporal CoT data to develop high-quality reasoning capabilities. This stage teaches the model to ground its reasoning in visual evidence and temporal changes.
Reinforcement Learning from Verifiable Rewards (RLVR): After SFT, the model is further refined using GRPO (Group Relative Policy Optimization). The reward here is based on the accuracy of the progress prediction compared to the ground truth. This stage specifically calibrates the scalar progress value, which is often under-emphasized during SFT.

3. Key Contributions

SOLE-R1 Model: A video-native reasoning model that generates per-timestep CoT and dense progress estimates, serving as the sole reward signal for online RL.
Data Synthesis Pipeline: A novel method for generating diverse, non-expert, and counterfactual robot trajectories with temporally grounded CoT traces, bridging the gap between expert demonstrations and the failure states encountered during RL exploration.
Hybrid Training Recipe: A combination of SFT for reasoning and RLVR for reward calibration, which significantly outperforms SFT-only models in reward accuracy and robustness.
Zero-Shot Online RL: Demonstrated that a robot can learn 24 previously unseen manipulation tasks from random initialization using only SOLE-R1's predicted rewards, with no ground-truth rewards, demonstrations, or task-specific tuning.

4. Experimental Results

The authors evaluated SOLE-R1 across four simulation environments (RoboSuite, ManiSkill, Meta-World, LIBERO) and a real-world Franka robot setting.

Performance: SOLE-R1 achieved zero-shot success on 24 out of 40 unseen tasks (success rate >50%). In contrast, strong baselines like GPT-5 and Gemini-3-Pro succeeded on fewer than 10 tasks.
Robustness to Reward Hacking:
- Baseline VLMs (GPT-5, Gemini) predominantly failed via reward hacking (high perceived success, low true success), where agents learned to exploit visual loopholes.
- SOLE-R1 failures were primarily signal-limited (low perceived and low true success), indicating the model correctly recognized failure states rather than hallucinating success.
Generalization: SOLE-R1 generalized to unseen task families (pick-and-place, articulated objects, button/lever interactions), different camera viewpoints, and different robot embodiments (e.g., Sawyer, WidowX, Franka).
Ablation Studies:
- Removing CoT reasoning degraded learning signals (more signal-limited failures).
- Training only on expert trajectories increased reward hacking significantly.
- Removing foundational spatial/temporal data reduced generalization and increased exploitation.
Scaling Law: Performance scaled positively with the diversity of task types included in the training data.

5. Significance

This work represents a major step toward generalist robot learning. By proving that a video-language reasoning model can serve as the sole supervision signal for online RL, SOLE-R1 eliminates the need for:

Hand-crafted reward functions.
Task-specific demonstrations.
Ground-truth state sensors.

The paper demonstrates that grounded spatiotemporal reasoning is a critical component for robust reward modeling. Unlike previous "black-box" reward models, SOLE-R1's CoT reasoning provides interpretability, allowing the system to distinguish between genuine progress and superficial visual cues that lead to reward hacking. This approach paves the way for robots to learn complex, long-horizon tasks in unstructured environments purely through interaction and natural language goals.