💬 NLP

$R^3$ : Training Robots to Reason in Natural Language via Reinforcement Learning

The paper introduces $R^3$ , a post-training framework that enhances off-the-shelf Vision-Language Models into robotic reasoners capable of generating free-form natural language guidance for low-level manipulation policies, thereby significantly improving exploration and generalization in long-horizon tasks through a combination of mid-training on expert traces and rubric-based reinforcement learning.

Original authors: Lehong Wu, Yuxiao Qu, Zheyuan Hu, Ivan Zhang, Limin Wei, Zackory Erickson, Aviral Kumar

Published 2026-08-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Lehong Wu, Yuxiao Qu, Zheyuan Hu, Ivan Zhang, Limin Wei, Zackory Erickson, Aviral Kumar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long been masters of repetition, capable of performing the same motion thousands of times with perfect precision. Yet, when faced with a new situation or a mistake, they often stumble, lacking the ability to pause and think through what went wrong. This limitation stems from a gap between seeing the world and understanding it. Modern artificial intelligence has made strides in both vision and language, allowing machines to recognize objects and follow spoken commands. However, connecting these abilities to physical action in a way that allows for flexible problem-solving remains a significant challenge. The core question driving recent research is whether giving a robot the ability to "think" in words before it moves can help it navigate complex, long-term tasks that require planning, tracking progress, and recovering from errors.

A team of researchers at Carnegie Mellon University has developed a new method to teach robots this kind of thinking. They created a system that trains a high-level artificial intelligence to generate natural language reasoning—essentially a running commentary on what the robot should do next—before it issues a command to a lower-level controller that actually moves the robot's arms. The researchers found that by training the system to produce these thoughts, the robot became significantly better at solving difficult manipulation tasks, such as arranging blocks into specific shapes or packing groceries into a bag. The key discovery is that the act of generating these thoughts in real time, rather than just using them as a training tool, acts as a form of mental computation that helps the robot steer its actions more effectively.

The researchers approached this problem by treating the robot's brain as a two-part system. One part is a "reasoner," a large language model that looks at the scene and the goal, and the other is a "doer," a pre-trained policy that executes the physical movements. In their experiments, the reasoner was tasked with looking at the current state of the world, the history of what has happened so far, and the final goal, and then writing down a short explanation of its plan before issuing a specific instruction to the doer. For example, if the goal was to arrange blocks in a line, the reasoner might first note that a green block is already in place, observe that a star is out of position, and then decide to move the star to complete the line. This process of writing down the reasoning happens every time the robot takes a step.

To teach the system how to do this, the researchers used a two-stage training process. First, they showed the system examples of experts solving tasks while verbalizing their thought processes. This step, which they call mid-training, helped the model learn the style of reasoning needed for these physical tasks, such as tracking partial progress or noticing when a plan has gone wrong. However, simply mimicking these examples was not enough. The researchers then used a second stage involving reinforcement learning, a method where the system learns by trial and error based on feedback. In this stage, the robot was given a task and asked to generate its own reasoning and instructions. It received a reward not for the final outcome alone, but for whether its generated instruction matched the intent of an expert's instruction. This allowed the system to refine its reasoning skills using a vast amount of data where only the final instructions were known, without needing to collect expensive examples of expert thoughts for every single move.

The results of these experiments were measured in two distinct environments. The first was a simulated table filled with colored blocks, where the robot had to arrange them into shapes like lines or V-shapes. The second was a more complex simulation of a bimanual robot packing groceries into bags. In both cases, the system trained with this reasoning method, which the researchers named R3, outperformed systems that were trained only to follow instructions without generating thoughts. On the block arrangement tasks, the reasoning-enabled robot showed a marked ability to generalize to new, unseen shapes, succeeding where other methods failed. In the grocery packing simulation, the robot that could reason about its actions achieved a success rate of nearly 47.9 percent on difficult, unseen tasks, compared to roughly 38.0 percent for a system that simply followed instructions without thinking.

Crucially, the researchers investigated whether the improvement came from the robot simply learning better visual patterns during training or from the act of thinking itself. They found that while the training did improve the robot's ability to understand the scene, the real benefit came from the reasoning process happening at the moment of decision. When they forced the robot to stop generating thoughts and just act, its performance dropped significantly. Conversely, when they allowed the robot to spend more time generating longer, more detailed thoughts, its success rate increased, particularly on harder tasks. This suggests that the language reasoning serves as a mechanism for the robot to spend extra mental effort on difficult problems, much like a human might pause to think through a complex puzzle before making a move.

The study also revealed specific behaviors that the reasoning system learned to adopt. The robot began to compare different possible actions before choosing one, re-examine the scene when it was uncertain about an object's position, and adjust its plan when it realized a previous step had failed. It learned to track the history of its interactions, remembering what it had already done to avoid repeating mistakes. These behaviors were not explicitly programmed; they emerged naturally as the system learned to use language to guide its actions. The researchers noted that this approach worked even when the robot was not given explicit examples of expert thoughts for the second stage of training, relying instead on the system's ability to infer the correct reasoning from the final instructions alone.

While the experiments were conducted in simulated environments, the findings offer a clear path forward for making robots more adaptable. The researchers suggest that this method could be applied to real-world robots to help them handle the unpredictability of physical tasks, such as dealing with noisy sensors or recovering from collisions. They also point out that future work could explore training the reasoning and action parts of the system together, rather than keeping them separate, to create even more coordinated behavior. For now, the work demonstrates that giving a robot the ability to talk to itself about what it is doing is not just a theoretical exercise, but a practical way to improve its ability to solve complex, long-term problems. The system does not need to be perfect to be useful; it simply needs to be able to think through the steps, and in doing so, it becomes far more capable than a machine that merely reacts.

Technical Summary: ℛ3: Training Robots to Reason in Natural Language via Reinforcement Learning

Problem Statement

While foundation models have demonstrated the ability to spend "test-time compute" on reasoning to solve complex problems (e.g., via Chain-of-Thought), it remains unclear if this mechanism translates effectively to robotic manipulation. Robotic tasks often require long-horizon planning, tracking partial progress, reasoning about object relations, and recovering from mistakes. Existing approaches to robotic reasoning typically rely on structured intermediate representations (e.g., object coordinates, depth maps, or fixed subtask plans) or use reasoning only as an auxiliary training signal. A critical gap exists: prior work suggests that while structured reasoning supervision improves training-time grounding, generating reasoning at test time often yields little additional benefit. The paper investigates whether Vision-Language Models (VLMs) can be trained to produce free-form natural language reasoning at test time to actively steer low-level manipulation policies, thereby acting as a mechanism for spending compute on hard robotic problems.

Methodology: The ℛ3 Framework

The authors propose ℛ3, a two-stage post-training recipe designed to turn off-the-shelf VLMs into robotic reasoners. The architecture employs a hierarchical setup where a high-level VLM generates a reasoning trace and a short-horizon instruction, which steers a fixed, pre-trained low-level policy.

Stage I: Mid-Training for Reasoning Style

To initialize the VLM with the necessary reasoning behaviors, the authors perform mid-training on a small subset of expert-generated reasoning traces.

Data Source: Expert trajectories (collected via an expert VLM in the Language Table environment or human teleoperation in grocery packing) containing interaction history, expert reasoning traces, and instructions.
Objective: Standard next-token prediction ( $\mathcal{L}_{SFT}$ ) on the concatenated sequence of reasoning and instruction.
Purpose: This stage teaches the model to track task progress, identify constraints, and handle partial failures, initializing a "reasoning style" that is often absent in base models suitable for real-time control (e.g., <10B parameters).

Stage II: Rubric-Based Single-Step RL

To refine the reasoner without requiring expensive multi-turn robot rollouts or human-written rewards, the authors apply single-step Reinforcement Learning (RL) on a broader dataset containing only expert instructions (no reasoning traces).

Setup: Given an interaction history and a goal, the model generates a free-form reasoning trace and an instruction.
Reward Mechanism:
- Language Table: A VLM-as-a-judge (Qwen3.5-35B) evaluates the generated instruction against the expert's instruction using a rubric. The reward is based on semantic alignment, feasibility, and intent, rather than string matching.
- Grocery Packing: A simple exact-match reward is used, as the instruction space is finite and unambiguous.
Optimization: The model is trained using Dr.GRPO (Group Relative Policy Optimization), optimizing the policy gradient to maximize the advantage of generating instructions that match the expert's intent.
Key Design Choice: The model must generate its own reasoning to justify the instruction, effectively learning to reason from action supervision alone.

Key Contributions

ℛ3 Recipe: A practical two-stage training pipeline that converts off-the-shelf VLMs into robotic reasoners using limited reasoning-labeled data for initialization and abundant instruction-only data for RL refinement.
Free-Form Reasoning as Test-Time Compute: The paper demonstrates that free-form natural language reasoning, generated at inference time, serves as an effective mechanism for steering low-level policies, outperforming methods that rely solely on structured intermediate representations or training-time supervision.
Causal Evidence of Reasoning: Through ablation studies (truncating reasoning tokens, comparing with non-reasoning baselines trained with reasoning supervision), the authors provide evidence that the performance gains are not merely due to better representation learning but are causally linked to the act of reasoning during inference.
Behavioral Analysis: The trained reasoners exhibit specific behaviors useful for long-horizon tasks, including comparing alternatives, self-correction, re-examining scenes to resolve ambiguity, and incremental planning.

Experimental Results

The framework was evaluated on two controlled testbeds: Language Table (block manipulation) and Simulated Bimanual Grocery Packing.

Language Table Results

Generalization: ℛ3 significantly outperforms instruction-only imitation learning (IL) baselines on Out-of-Distribution (OOD) tasks. While IL often degrades or shows marginal gains on unseen tasks, ℛ3 maintains or improves performance.
Ablation:
- RL Only: Improves performance over the base model, even without mid-training.
- Mid-Training + RL: Yields the best results, suggesting mid-training provides a strong behavioral prior that RL refines.
- Test-Time Compute: Truncating the reasoning budget at test time leads to significant performance drops, confirming that the reasoning process itself contributes to success.
- Structured vs. Free-Form: Adding structured CoT components (like ECoT bounding boxes) did not improve performance over free-form language reasoning, suggesting that for these tasks, tracking progress and planning is more critical than low-level visual grounding.

Grocery Packing Results

Transferability: The approach successfully transferred to a bimanual packing task.
Efficiency: In this domain, Stage I mid-training was skipped (as the base VLM already produced useful reasoning), and Stage II RL alone outperformed instruction-only IL.
Performance: ℛ3 (RL only) achieved higher success rates and normalized task progress compared to baselines on 12 held-out task configurations.

Significance and Claims

The paper claims that free-form language reasoning can function as a test-time compute mechanism for robotic manipulation. Unlike prior work that treats reasoning as a static training signal or relies on rigid structured templates, ℛ3 demonstrates that training a VLM to generate dynamic, free-form reasoning allows it to:

Decompose complex, long-horizon tasks.
Track partial progress and recover from mistakes.
Steer low-level policies more effectively than instruction-only models.

The authors emphasize that these gains are causal: the act of reasoning at inference time is necessary for the observed improvements in generalization and task success. The work suggests a path toward more robust robotic agents that can "think" before acting, leveraging the reasoning capabilities of foundation models without requiring end-to-end training of the entire control stack.

Limitations

The authors acknowledge that experiments are currently limited to simulated environments with fixed low-level policies. They note that Stage I still requires expert reasoning traces (though minimal), and Stage II relies on a VLM judge as a surrogate for true task success. Future work is needed to validate these methods on real robots and to explore multi-turn online RL.