🤖 machine learning

RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance

RynnValue is an open-source robotic value foundation model that scales to millions of instruction-conditioned clips by using temporal distance as a supervision target, outperforming preference-based methods in evaluation and significantly boosting real-world policy success without requiring explicit reward or progress annotations.

Original authors: Dongchi Huang, Hongyin Zhang, Bohan Hou, Siteng Huang, Zhian Su, Hang Guo, Tong Lu, Zhaofeng Xu, Jiahao Tang, Jianfei Yang, Donglin Wang, Peixi Peng, Mingxiu Chen, Deli Zhao, Xin Li

Published 2026-08-11

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Dongchi Huang, Hongyin Zhang, Bohan Hou, Siteng Huang, Zhian Su, Hang Guo, Tong Lu, Zhaofeng Xu, Jiahao Tang, Jianfei Yang, Donglin Wang, Peixi Peng, Mingxiu Chen, Deli Zhao, Xin Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to cook a meal, but instead of giving it a recipe, you just say, "Make it good!" and hope it figures out the rest. This is the world of robot learning, a branch of science where engineers try to teach machines to move and interact with the physical world. The biggest hurdle isn't usually the robot's brain (its ability to move); it's the reward. In the world of AI, a "reward" is like a scorecard or a high-five that tells the robot, "Good job!" or "Try again." The problem is that giving a robot a perfect scorecard for every single move is incredibly hard. If the scorecard is too vague, the robot gets confused. If it's too specific to one task, the robot can't learn anything new. Scientists have been trying to build a "general-purpose" scorecard that works for any robot doing any task, but they've been stuck on how to create it without needing a human to manually grade every single attempt.

This is where a new idea called RynnValue comes in. Think of it as a new way to grade a robot's performance. Instead of asking, "How close are you to the finish line?" (which is hard to measure if the finish line looks different for every task), RynnValue asks, "How much time is left until you're done?" It treats the robot's journey like a countdown timer. If the robot is far from the goal, the timer is high. If it's close, the timer is low. The magic of this paper is that it shows you don't need a human to press a button and say "Good job" to create this timer. You can just look at the clock. If a video of a robot cooking starts at 10 minutes and ends at 0, the computer can automatically figure out that at the 5-minute mark, the robot had 5 minutes left. This simple trick allows the robot to learn from thousands of hours of videos without anyone needing to manually grade them.

The Problem: The "Progress" Trap

For a long time, scientists tried to teach robots by measuring "progress." Imagine a robot trying to stack blocks. A progress model would try to guess if the robot is 10% done, 50% done, or 90% done. But here's the catch: "50% done" looks totally different for stacking blocks than it does for pouring water. One robot might be halfway through by height, while another is halfway through by time. This makes it really hard to train a robot that can do many different things, because the definition of "progress" changes every time the task changes. It's like trying to teach a student math by saying "You're halfway there!" without ever telling them what the final answer is supposed to be.

The authors of this paper argue that this "progress" approach is a dead end. They say we need to stop trying to guess how far along a robot is and start counting down the time until the job is finished. They call this temporal distance. It's the "cost-to-go," or simply, "how much time do I have left?"

The Solution: RynnValue and the Magic Clock

The team at Alibaba's DAMO Academy and Hupan Lab built a model called RynnValue. Instead of trying to guess a progress percentage, RynnValue looks at a video of a robot and a text instruction (like "put the cup on the table") and predicts exactly how many seconds are left until the task is done.

Here is the clever part: they didn't need humans to write down "5 seconds left" for every video. They just used the timestamps already on the video files. If a video starts at 0:00 and the robot finishes at 0:10, the computer knows that at 0:05, the robot had 5 seconds left. This means they could train the model on a massive library of over 7,000 hours of robot videos—roughly 3 million clips—without needing a single human to manually grade them. It's like teaching a student to read by letting them watch thousands of movies with subtitles, rather than having a teacher grade every sentence they read.

The Sneaky Tricks (and How They Stopped Them)

You might think, "If the robot just learns to look at the clock, isn't that exploiting a shortcut?" The scientists were worried about this too. They knew that if they just showed the robot videos in order, the robot might learn a shortcut: "Oh, the 3rd frame in the video is always halfway done!" or "If the video is 10 seconds long, the middle is always 5 seconds left." The robot would stop looking at the actual robot arms and just guess based on the video's position.

To stop this, they added some "chaos" to the training:

Random Sampling: They didn't show the robot the video in a straight line. They picked random moments from the video, so the robot couldn't guess the time based on the order of the frames.
Shuffling: They sometimes showed the robot the video backwards or in a jumbled order. If the robot saw a "finished" state before a "start" state, it had to realize that the "finished" state was actually further away in time, forcing it to actually look at the picture of the robot, not just the order of the pictures.
Isolation: They made sure the robot couldn't peek at the answers for the next frame while guessing the current one. It had to solve each moment on its own.

The Results: Does It Work?

The team tested RynnValue on a bunch of different robots and tasks it had never seen before. The results were surprisingly good.

Ranking Tasks: When asked to rank different robot attempts from "best" to "worst," RynnValue got a score of 0.675. This was better than the previous best models that relied on humans manually grading the videos (which scored 0.655).
Real-World Robots: They took the model and used it to teach real robots to do tricky tasks, like putting bread in a basket or moving a steak with a spatula.
- With RynnValue, the robots succeeded 72.5% of the time when learning online (while moving).
- Without it, using the old best methods, they only succeeded 52.5% of the time.
- In offline learning (learning from a database of past moves), RynnValue boosted success from 63.8% to 82.5%.

Why This Matters

The most exciting thing about this paper is that it suggests we don't need to be perfect at grading robots to teach them. We just need to know when they finished. By using the simple, automatic "clock" of time, RynnValue creates a universal language for robot rewards. It works for a robot with two arms, a robot with a wheel, or a robot with a hand, as long as we can tell when the job is done.

The authors are careful to note that this isn't a magic wand that solves every robot problem instantly. They found that for very tricky tasks requiring precise alignment (like putting a box in a drawer), the model still struggled a bit because the visual clues were confusing. But overall, they showed that temporal distance is a powerful, scalable way to teach robots. It turns the messy, hard problem of "what is a good move?" into the simple, solvable problem of "how much time is left?"

In short, RynnValue is like giving a robot a stopwatch instead of a report card. It turns the chaotic world of robot learning into a simple countdown, and it seems to be working better than the old ways.

Technical Summary: RynnValue – Scaling Robotic Value Foundation Models with Temporal Distance

Problem Statement

General-purpose robot policies are increasingly trained via reinforcement learning (RL), but scaling this loop is bottlenecked by reward supervision. Hand-designed rewards fail to generalize to open-ended tasks, while sparse success signals offer insufficient guidance for long-horizon behaviors. Existing general-purpose reward models rely on task-internal anchors such as human preferences, reference demonstrations, or normalized progress. These approaches tie supervision to specific trajectories or comparison sets, making them difficult to transfer across heterogeneous data sources, embodiments, and viewpoints. Furthermore, normalized progress (e.g., [0, 1] scales) is an intra-trajectory coordinate rather than a goal-conditioned cost-to-go, leading to inconsistencies across varying task durations and structures.

Methodology: RynnValue

The authors introduce RynnValue, an open-source value foundation model that reframes the reward modeling problem. Instead of scoring trajectory-level anchors, RynnValue predicts temporal distance—the directed, goal-conditioned cost-to-go from an observation to a language-specified goal.

1. Data Recipe and Labeling

RynnValue is trained on a massive, heterogeneous mixture of over 7,000 hours of robot data, comprising roughly 3 million instruction-conditioned trajectory clips. This dataset spans diverse embodiments (single-arm, dual-arm, mobile, dexterous hands), viewpoints, and task families (real-world, simulated, and egocentric).

Label Derivation: Unlike preference-based methods, RynnValue requires no preference pairs or progress annotations. Labels are derived directly from timestamps. By identifying a completion cutoff (via subtask segmentation and relabeling), the model learns the remaining time to completion for any observation.
Supervision Signal: The model predicts the absolute temporal distance to task completion and the signed relative temporal displacement between consecutive observations.

2. Model Architecture

Built upon RynnBrain, RynnValue is a language-conditioned visual model that processes interleaved multimodal sequences of instructions, observations, and query tokens.

Grouped Temporal Queries: To overcome information bottlenecks, each temporal prediction is represented by a group of $N=8$ repeated query tokens. These tokens interact via bidirectional attention before being concatenated to form a robust representation.
Dual Distributional Heads: The model employs two specialized heads:
- Absolute Head: Predicts the remaining time to completion ( $v_t$ ).
- Relative Head: Predicts the signed temporal displacement between adjacent observations ( $\Delta_t$ ).
- Both heads use symlog-spaced binning (256 bins) and two-hot targets to handle the wide dynamic range of temporal distances, casting regression as a stable classification problem.
Natural Language Branch: An auxiliary branch generates video descriptions, instruction-video matching judgments, and success predictions, providing semantic grounding without interfering with temporal value heads.

3. Shortcut Suppression Strategies

To ensure the model grounds predictions in visual evidence rather than exploiting data regularities, the authors introduce three key mechanisms:

Random Temporal Sampling: Observations are sampled at irregular timestamps to break arithmetic value patterns induced by uniform sampling.
Temporal-Order Shuffling: The chronological order of sampled observations is shuffled (with a "rewind" probability) during training. This prevents the model from inferring progress solely based on sequence position.
Value-Isolation Attention: A masking strategy where value-query groups associated with different observations cannot attend to one another. This prevents the model from extrapolating values from previous queries, forcing each prediction to rely on its specific visual context.
Instruction-Mismatch Augmentation: 10% of training samples use mismatched instructions to teach the model to detect inconsistencies and avoid assuming smooth progress toward an incorrect goal.

4. Inference and Reward Interface

At inference, RynnValue outputs a temporal distance estimate $v_t$ . To serve as a reward signal, this is converted into a potential-based reward via sign reversal: $\Phi_t = -v_t$ . This creates a dense reward interface where states closer to completion have higher potential (closer to zero), preserving the temporal scale without normalizing to a task-specific [0, 1] interval.

Key Results

Benchmark Performance (RBM-EVAL-OOD)

Evaluated on the RBM-EVAL-OOD trajectory ranking suite (976 trajectories across 6 OOD datasets):

RynnValue-8B achieved an average Kendall's $\tau_a$ of 0.675, surpassing the fully preference-supervised state-of-the-art (Robometer, 0.655) and more than doubling the progress-only counterpart (0.292).
Even the RynnValue-4B variant (0.670) outperformed the preference-supervised baseline, suggesting the gains stem from the temporal-distance formulation rather than sheer model scale.
RynnValue variants achieved the best results on all six datasets among methods trained without explicit trajectory-level preference supervision.

Instruction-Trajectory Alignment

In confusion matrix analyses, RynnValue demonstrated the clearest diagonal structure (normalized diagonal margin of 0.79), significantly outperforming baselines (0.67). This indicates the model effectively distinguishes matched instruction-trajectory pairs from mismatched ones, grounding rewards in the language-specified goal rather than generic visual progress.

Real-World Policy Learning

RynnValue was deployed as a zero-shot reward annotator for online and offline RL on four challenging real-world tasks (Bread Basket Placement, Steak Serving, Box-in-Drawer, Bimanual Box Transfer) with no target-domain fine-tuning.

Online RL: RynnValue improved success rates from 52.5% (Robometer) to 72.5%.
Offline RL: RynnValue improved success rates from 63.8% (Robometer) to 82.5%.
Notably, RynnValue enabled successful RL on tasks (Box-in-Drawer, Bimanual Box Transfer) where the SFT baseline achieved 0% success.

Scaling Analysis

Experiments disentangling data quantity from diversity showed that task diversity yields a consistently steeper error reduction than episode volume. Simply adding more data of the same tasks leads to early saturation, whereas increasing the variety of tasks drives continuous improvement in generalization.

Significance and Claims

The paper claims that temporal distance is a scalable, preference-free supervision target that unifies heterogeneous robotic data under a single interface. By deriving labels directly from timestamps, RynnValue bypasses the need for expensive preference annotations or dataset-specific progress normalization.

The authors assert that RynnValue establishes a practical path for training generalist robot policies:

Scalability: It leverages massive, heterogeneous datasets without the bottleneck of human preference labeling.
Robustness: Through temporal-order shuffling and value-isolation attention, it learns values grounded in visual evidence, making it sensitive to failures and regressions.
Utility: It serves as an effective dense reward interface that significantly improves both online and offline policy learning in real-world settings, outperforming strong baselines trained with preference supervision.

The work concludes that temporal distance offers a more scalable route to general-purpose robotic value learning compared to traditional preference-based or progress-normalized approaches.