🤖 machine learning

Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons

The paper introduces Robometer, a scalable reward modeling framework that combines frame-level progress supervision with trajectory-comparison preference learning, trained on the large-scale RBM-1M dataset to effectively learn generalizable reward functions from diverse expert and suboptimal trajectories.

Original authors: Anthony Liang, Yigit Korkmaz, Jiahui Zhang, Minyoung Hwang, Abrar Anwar, Sidhant Kaushik, Aditya Shah, Alex S. Huang, Luke Zettlemoyer, Dieter Fox, Yu Xiang, Anqi Li, Andreea Bobu, Abhishek Gupta, Ste

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Anthony Liang, Yigit Korkmaz, Jiahui Zhang, Minyoung Hwang, Abrar Anwar, Sidhant Kaushik, Aditya Shah, Alex S. Huang, Luke Zettlemoyer, Dieter Fox, Yu Xiang, Anqi Li, Andreea Bobu, Abhishek Gupta, Stephen Tu, Erdem Biyik, Jesse Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to cook dinner. In the past, to teach the robot, you had to act as a strict judge, watching every single second of its movements and assigning it a precise score (like "7.5 out of 10") for how well it was chopping onions. If the robot dropped the knife, you had to figure out exactly when the score dropped and by how much. This is incredibly hard to do, especially when the robot fails, and it means you can't use the thousands of "failed" attempts robots make in the real world because they are too messy to grade.

ROBOMETER is a new system that changes how we teach robots by using a smarter, more human-like approach to grading.

The Core Idea: Comparing Instead of Grading

Instead of trying to give a perfect score to every single moment, ROBOMETER learns by comparing two different attempts at the same task.

Think of it like a talent show judge. Instead of giving a contestant a score of 8.2 out of 10, the judge simply looks at two performances and says, "Performance A was clearly better than Performance B."

The Old Way: You have to watch a robot fail to put a cup on a table and try to calculate exactly how "bad" the failure was.
The ROBOMETER Way: You show the robot a video of a human doing it perfectly and a video of the robot dropping the cup. The system learns: "The human video is better." It doesn't need to know why or give a specific number; it just learns the order of "good" vs. "bad."

The "Giant Library" of Mistakes (RBM-1M)

To teach this system, the researchers built a massive library called RBM-1M.

The Old Problem: Most robot training libraries only have "perfect" videos. If a robot drops a cup, that video is thrown in the trash because it's hard to grade.
The New Solution: This library contains 1 million videos from 21 different types of robots (from single arms to human-like hands). Crucially, it is full of mistakes, failures, and clumsy attempts. Because ROBOMETER learns by comparing (e.g., "This failed attempt is worse than that successful one"), it can actually learn from the trash pile of failed videos. It treats failures as valuable lessons on what not to do.

How It Works (The Two-Step Dance)

ROBOMETER learns using two specific tricks at the same time:

The "Progress" Check: It watches a single video and tries to guess, "How far along is this task?" (0% to 100%). This anchors the robot's understanding of time and progress.
The "Preference" Check: It watches two videos side-by-side and decides, "Which one did the job better?" This teaches the robot to understand the quality of the action, even if it's a total failure.

By combining these, the robot learns a "feeling" for success that works even when it's looking at a robot it has never seen before, in a room it has never been in.

What It Actually Does (Proven Results)

The paper shows that this system works better than previous methods in four specific, real-world scenarios:

Automatic Online Learning: When a robot is learning a task in real-time (like putting a bowl on a table), ROBOMETER acts as a coach that instantly tells the robot, "You're doing it wrong," and guides it to fix the mistake. It helped a robot improve its success rate from 20% to 85% in a cluttered kitchen, whereas older methods got stuck at 55%.
Learning from Messy Data (Offline RL): If you have a mix of perfect videos and messy, failed videos, ROBOMETER can sort through them to teach a new robot. It improved success rates by 2.4 times compared to the best previous tools.
Finding the Good Stuff (Data Filtering): Imagine you have a huge pile of video clips and you need to find the 10 best ones to teach a robot how to "stir a pot." ROBOMETER is much better at finding the relevant, successful clips and ignoring the failures than other search tools. This led to a 4.5 times improvement in the robot's ability to learn the task.
Spotting Failures: If a robot is stuck, oscillating (wiggling back and forth without moving), or dropping an object, ROBOMETER can detect this failure immediately without needing to be told what a failure looks like. It correctly identified failures in 81% of cases, beating other systems.

The Bottom Line

ROBOMETER is a "universal reward model" that stops robots from needing a human to grade every single second of their work. By learning to compare "better" vs. "worse" using a massive library of both successes and failures, it helps robots learn faster, handle mistakes better, and adapt to new tasks and new robot bodies without needing to be retrained from scratch.

Technical Summary: ROBOMETER

Problem Statement

Current general-purpose robotic reward models face significant scalability and generalization challenges. Existing approaches rely exclusively on absolute progress labels derived from expert demonstrations (e.g., linearly interpolating progress from 0 to 1). While effective for successful trajectories, this paradigm fails to scale to large, heterogeneous datasets for two primary reasons:

Ambiguity in Failed Trajectories: Suboptimal or failed attempts, which are abundant in real-world data collection, lack well-defined absolute progress labels. Progress in these cases often fluctuates or regresses, making pointwise supervision costly and ill-defined.
Limited Generalization: Models trained solely on trajectory-local progress supervision struggle to generalize across different robot embodiments, scenes, and varying trajectory qualities. They often fail to distinguish between successful and suboptimal behaviors when absolute scores are ambiguous.

The paper posits that human cognition relies on comparative judgments rather than isolated absolute scales to internalize calibrated progress. Consequently, robotic reward models require a supervision signal that leverages relative ordering across diverse trajectories to learn robust, generalizable reward functions.

Methodology

1. The ROBOMETER Framework

ROBOMETER is a scalable reward modeling framework that combines intra-trajectory progress supervision with inter-trajectory preference supervision. It utilizes a dual-objective training recipe:

Frame-Level Progress Loss: Anchors reward magnitude to expert data by predicting dense, per-frame progress (discretized into 10 bins over [0, 1]) and binary success labels. This ensures the model learns the temporal structure of task completion.
Trajectory-Comparison Preference Loss: Imposes global ordering constraints by predicting which of two trajectories better satisfies a language instruction. This allows the model to learn from relative comparisons without requiring absolute scores for failed trajectories.

2. Architecture

ROBOMETER is built upon a pre-trained Vision-Language Model (VLM) backbone, specifically Qwen3-VL-4B-Instruct.

Tokenization: The model processes interleaved text and visual tokens. To enable dense reward estimation without disrupting pre-trained representations, it inserts learnable progress tokens (<|prog token|>) after each frame of the first trajectory and a single preference token (<|pref token|>) at the end of the prompt.
Causal Masking: The architecture uses causal masking to ensure progress tokens attend only to current and previous frames of the first trajectory (enabling online inference), while the preference token attends to both trajectories to make a relative judgment.
Heads: Lightweight MLP heads are attached to the hidden states of the progress tokens (for progress and success prediction) and the preference token (for binary preference classification).

3. The RBM-1M Dataset

To support this dual-objective training at scale, the authors curated RBM-1M, a dataset of over 1 million trajectories spanning 21 robot embodiments (including single-arm, bimanual, and mobile manipulators) and diverse scenes.

Composition: The dataset aggregates expert demonstrations (e.g., Open-X, AGIBotWorld), human videos (Epic-Kitchens), simulation data (LIBERO), and crucially, unlabeled failed and suboptimal trajectories from automated rollouts and failure detection datasets.
Augmentation Strategies: To construct preference pairs without additional human annotation, the authors employ three strategies:
1. Progress-Based Comparisons: Contrasting expert demonstrations against unlabeled failures or suboptimal trajectories.
2. Instruction Negatives: Pairing trajectories with distinct language instructions to ensure rewards are grounded in the command.
3. Video Rewind: Generating synthetic failures by reversing segments of successful trajectories to model "undoing" progress.

Key Contributions

Dual-Objective Training Recipe: The paper introduces a method to train reward models using both absolute progress labels (for successful data) and pairwise preference labels (for all data, including failures). This enables the effective utilization of previously unusable suboptimal data.
RBM-1M Dataset: The release of a large-scale, multi-embodiment dataset containing extensive failure data, specifically designed to train globally consistent preference relations.
Scalable Generalization: The framework demonstrates that preference supervision acts as a regularizer, inducing a more structured internal reward representation that improves the model's ability to distinguish successful from suboptimal trajectories, even when trained on limited expert data.

Experimental Results

Reward Evaluation

ROBOMETER was evaluated on RBM-EVAL-OOD, a set of 976 trajectories from 3 institutions and 6 embodiments (3 of which were unseen during training).

Reward Alignment: ROBOMETER achieved a Value Order Correlation (VOC) of 0.94 on OOD data, outperforming state-of-the-art baselines (e.g., RoboReward-8B at 0.88, VLAC-8B at 0.38).
Trajectory Ranking: In distinguishing successful from failed trajectories, ROBOMETER achieved a Kendall- $\tau_a$ of 0.64 on OOD data, significantly outperforming RoboReward-8B (0.47) and ReWiND (0.01).
Failure Detection: In zero-shot failure detection tasks, ROBOMETER achieved the highest average F1 score (0.81), effectively balancing true positive and true negative rates, whereas baselines like VLAC suffered from high false positives and token-uncertainty baselines suffered from high false negatives.

Downstream Policy Learning

The paper demonstrates that ROBOMETER's dense, well-calibrated rewards improve downstream robot learning across four settings:

Automatic Online RL: Using DSRL with ROBOMETER improved success rates from 20% to 85% on a single-stage task and 20% to 70% on a multi-stage task, outperforming RoboReward by 2.5 $\times$ .
Offline RL: In mixed-expertise settings (expert + noisy data), ROBOMETER enabled IQL policies to achieve a 2.4 $\times$ improvement in success rates over the best baseline.
Data Filtering & Retrieval: When used to retrieve task-relevant subtrajectories for imitation learning, ROBOMETER-filtered data yielded a 4.5 $\times$ higher success rate compared to baselines like STRAP and SigLIP.
Model-Based RL: Integrating ROBOMETER into DreamZero improved success rates from 20% to 70% in a cluttered environment.

Ablation Studies

Preference Supervision: Adding preference loss to progress-only training consistently improved ranking metrics (Kendall- $\tau$ ) and success-failure separation.
Failure Data: Incorporating unlabeled failure trajectories for preference training provided the largest gains in reward discrimination.
Backbone: Replacing the pre-trained VLM backbone with a smaller, non-pre-trained architecture (ReWiND variant) resulted in severe performance degradation, confirming the necessity of large-scale multimodal pre-training.

Significance and Claims

The paper claims that ROBOMETER represents a shift from absolute progress supervision to comparative preference supervision as a mechanism for scaling reward learning. By leveraging relative comparisons, the framework can effectively utilize the vast amounts of suboptimal and failed data that are ubiquitous in real-world robotics but previously discarded.

The authors emphasize that this approach enables:

Scalability: Learning from datasets containing millions of trajectories without the need for dense, absolute annotations on every frame.
Generalization: Robust performance across unseen robot embodiments, scenes, and camera viewpoints.
Practical Utility: Immediate applicability in online RL, offline RL, imitation learning data curation, and automated failure detection.

The work concludes that while current models struggle with the ambiguity of failure, a dual-objective approach anchored by expert progress but guided by global preferences allows reward models to internalize calibrated scales of task progress, thereby accelerating robot learning in complex, real-world environments.