🤖 machine learning

Rank-Then-Act: Reward-Free Control from Frame-Order Progress

The paper introduces Rank-Then-Act (RTA), a reward-free control framework that trains a Vision-Language Model to learn temporal progress from shuffled video frames and utilizes a Spearman rank correlation-based reward signal to enable stable policy learning and cross-task transfer without explicit environment rewards.

Original authors: Yuriy Maksyuta, George Bredis, Ruslan Rakhimov, Daniil Gavrilov

Published 2026-07-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuriy Maksyuta, George Bredis, Ruslan Rakhimov, Daniil Gavrilov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to play a video game, but you have no rulebook, no score counter, and no "Game Over" screen. You only have a video of a human expert playing the game perfectly. How do you teach the robot what "doing well" looks like without telling it what the points are?

This is the problem the paper Rank-Then-Act (RTA) solves. Here is how it works, explained through simple analogies.

The Core Idea: "Order, Not Numbers"

Most AI systems try to guess a specific number (like "You are 85% done") to know if they are doing well. The authors realized this is hard because "85%" means different things in different games.

Instead, RTA asks a simpler question: "Is this moment happening before or after that moment?"

Think of it like a stack of photos from a time-lapse of a flower blooming. If you shuffle the photos, a smart observer can look at them and say, "This one is definitely earlier than that one," even if they don't know exactly what time it is. RTA uses this logic to learn.

The Two-Step Process

The method works in two distinct stages, like training a coach and then training the player.

Stage 1: Training the "Time-Travel Coach" (The Scorer)

First, the researchers take a powerful AI model (called a Vision-Language Model, or VLM) and teach it to be a Time-Travel Coach.

The Trick: They take a video of an expert playing a game, cut it into pieces, and shuffle the order of the frames (like mixing up a deck of cards).
The Task: They ask the Coach: "Look at these mixed-up pictures. Which one happened first? Which one happened last?"
The Lesson: The Coach is rewarded only if it gets the order right. It doesn't matter if it thinks the first frame is "10 points" and the last is "20 points." It just needs to know that Frame A comes before Frame B.
The Result: The Coach learns to understand the story of the game purely from the visuals. It learns that "seeing the character jump" usually happens before "seeing the character land." Once this training is done, the Coach is frozen (it stops learning) and becomes a fixed tool.

Stage 2: The Player Learns to "Keep the Story Straight" (The Act)

Now, the robot (the player) starts playing the game. It has no score, no points, and no rewards from the game itself.

The Signal: Every few seconds, the robot takes a snapshot of its recent actions and shows them to the frozen Time-Travel Coach.
The Test: The Coach looks at the robot's recent actions and asks: "Does this sequence look like it's moving forward in time, or is it just a jumbled mess?"
The Reward: The robot gets a "reward" based on a math concept called Spearman Rank Correlation.
- If the robot's actions look like a logical, forward-moving story (just like the expert video), the Coach gives a high score.
- If the robot is going backward, getting stuck in loops, or doing random things, the Coach gives a low score.
The Magic: The robot learns to maximize this score. It realizes, "Oh! To get a high score, I need to make my actions look like a coherent story that moves forward." By trying to keep the "story" logical, it accidentally learns to solve the game.

Why Is This Special?

No "Cheating" the System: If you just tell an AI "Later is better," it might just wait around or spin in circles, thinking that time passing equals progress. By shuffling the frames during training, RTA forces the AI to look at what is happening (the visual story), not just when it is happening.
One Coach, Many Games: The paper shows that a Coach trained on one game (like Catrap) can often help a robot learn a different game (like Kirby) or even a robot arm moving in a simulation. The Coach understands the concept of progress, not just the specific pixels of one game.
It's Robust: Because the system only cares about the order of events, it doesn't matter if the robot's actions are slightly different from the expert's, as long as the general flow of the "story" is correct.

The Bottom Line

Rank-Then-Act is a way to teach robots by showing them a movie and asking, "Does this look like a story that makes sense?"

Instead of giving the robot a complex scorecard, the system simply rewards the robot for keeping the plot of its actions moving forward logically. It turns the chaotic task of "learning to play a game without rules" into the simpler task of "telling a coherent story."

Technical Summary: Rank-Then-Act (RTA)

Problem Statement

Learning control policies from visual observations without extrinsic environment rewards is a fundamental challenge for generalist agents. This is particularly critical in domains like retro games, simulators with weak APIs, and real-world robotics, where reward design is brittle, unavailable, or susceptible to specification gaming. Existing approaches face three primary obstacles:

Trivial Shortcuts: Chronological inputs often lead models to learn "later is better" shortcuts, resulting in vacuous, monotonically increasing progress scores that fail to capture task semantics.
Scale Ambiguity: Absolute progress scales are ambiguous and difficult to calibrate across different tasks and episodes.
Online Learning Constraints: Reinforcement learning (RL) requires reward signals that are inexpensive to compute, informative for exploration, and robust to distribution shifts.

Current methods, such as video-based reward models or Vision-Language Model (VLM) reward heads, often require engineered goal text, per-task tuning, or produce uncalibrated scalar values that are hard to reuse. Furthermore, many imitation-from-observation methods require action labels or environment rewards, which are unavailable in the "video-only" setting addressed by this work.

Methodology: Rank-Then-Act (RTA)

RTA is a two-stage framework that converts a model's understanding of frame order in expert videos into a bounded, scale-invariant learning signal for reward-free control.

Stage 1: Offline Listwise Progress Scorer

The first stage involves training a Vision-Language Model (VLM) as a "progress scorer" using Group Relative Policy Optimization (GRPO).

Data Preparation: Expert video demonstrations are segmented into frames. To prevent the "later is better" shortcut, the first frame is anchored as a fixed starting point ( $s_{anc}$ ), and the remaining frames are randomly shuffled.
Objective: The VLM is prompted to generate a reasoning trace and assign a predicted progress rank ( $p_i$ ) to each shuffled frame (where higher rank implies a later time in the original sequence).
Reward Signal: The training reward is the Spearman rank correlation ( $\rho$ ) between the predicted ranks ( $p$ ) and the ground-truth temporal indices ( $q$ ).
$R = \text{spr}(p, q)$
This objective forces the model to recover temporal ordering based on visual semantics rather than absolute time cues. The VLM is fine-tuned (e.g., using LoRA) and then frozen for the second stage.

Stage 2: Online Control via Correlation

In the second stage, the frozen scorer is used to generate rewards for an agent interacting with the environment.

Sliding Window Mechanism: At specific query steps (e.g., every $N$ steps), a window of recent states $W_t$ is constructed. The oldest state in the window serves as the anchor.
Reward Computation: The non-anchor frames in the window are shuffled $L$ times. The frozen VLM predicts progress ranks for these permutations. The reward $r_t$ is the average Spearman correlation between the predicted ranks and the true timestamps of the window.
Policy Optimization: The agent (using PPO or DrQv2) is trained to maximize this correlation signal. The reward is bounded in $[-1, 1]$ , scale-invariant, and depends solely on the agreement of ordering structure, not calibrated reward magnitude. No extrinsic environment rewards are used.

Key Contributions

VLM Progress Scorer: Introduction of a simple, effective VLM scorer trained offline on shuffled clips with a GRPO objective to output per-frame progress ranks, kept frozen during policy learning.
Correlation-Only Reward Signal: Proposal of a dense, shaped, reward-free feedback mechanism based on the Spearman correlation between predicted progress ranks and time indices over sliding windows. This decouples reward learning from absolute calibration.
Empirical Validation: Demonstration that this reward signal enables effective control across discrete (GameBoy: Catrap, Kirby) and continuous (PointMaze, MetaWorld) benchmarks without environment rewards, outperforming strong video-based baselines.
Cross-Task Transfer: Evidence that a single pretrained progress scorer can transfer across tasks and environments, suggesting strong generality of ordinal video supervision.

Results

Discrete Control (GameBoy): On Catrap levels and Kirby, RTA consistently matched or outperformed baselines like Rank2Reward, VLM-RM, and untrained GVL formulations. Notably, RTA was the only method to achieve a non-zero success rate on the Kirby task without environment rewards.
Continuous Control: In PointMaze-UMaze and MetaWorld environments, RTA outperformed Rank2Reward (R2R) in success rates. While R2R requires per-task model training, RTA utilized a single shared scorer.
Cross-Domain Generalization: A scorer trained on diverse sources (e.g., YouTube datasets, MetaWorld, COIN AssembleSofa) successfully transferred to GameBoy tasks, achieving performance comparable to or better than baselines trained on task-specific data.
Ablations: Experiments confirmed that performance gains are driven by the reward design rather than the specific policy backbone (MLP vs. VLM). Analysis of window sizes revealed that intermediate windows provide the most stable signal, balancing local consistency against global progress detection.

Significance and Claims

The paper claims that correlation-structured supervision over video-derived ordinal signals is sufficient for policy learning, offering a scalable alternative to explicit reward design. By relying on rank correlation rather than scalar regression, RTA avoids the pitfalls of reward hacking and specification gaming associated with brittle reward functions.

The authors position RTA as a step toward robust, generalist agents capable of understanding and acting in open, dynamic, and underspecified environments. The method reduces the need for manually designed rewards and demonstrates that a single, pretrained ordinal scorer can support domain-general inference across diverse tasks, from retro games to robotic manipulation. The work suggests that learning from expert video alone, via progress-time consistency, is a viable path for multimodal agency in reward-free settings.