💻 computer science

Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning

The paper introduces VOTP, a semi-supervised framework that leverages Video Foundation Models and optimal transport to generate high-fidelity pseudo-labels from minimal human feedback, enabling efficient and robust reward learning for offline preference-based reinforcement learning.

Original authors: Tung M. Luu, Hwanhee Kim, Younghwan Lee, Chang D. Yoo

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tung M. Luu, Hwanhee Kim, Younghwan Lee, Chang D. Yoo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to do a complex task, like opening a drawer or walking without stumbling. In the world of robotics, the robot needs a "reward system" to know what it's doing right. Usually, human engineers have to painstakingly write code to tell the robot, "Good job, you moved your arm up," or "Bad job, you dropped the object." This is like trying to write a recipe for a dish you've never cooked; it's hard, and you might get the instructions wrong.

To fix this, scientists use Preference-Based Reinforcement Learning (PbRL). Instead of writing code, they show the robot two videos of it doing the task and ask a human, "Which one looks better?" The robot learns from these choices.

The Problem:
Asking a human to watch videos and pick the "better" one is slow and expensive. To teach a robot well, you might need thousands of these comparisons. It's like trying to teach a child to ride a bike by stopping them after every single wobble to ask, "Was that good?" It takes forever.

The Solution: VOTP
The paper introduces a new method called VOTP (Video-based Optimal Transport Preference). Think of VOTP as a clever teaching assistant that helps you teach the robot with very few human questions.

Here is how it works, using a simple analogy:

1. The "Smart Eye" (Video Foundation Models)

First, VOTP uses a pre-trained "Smart Eye" (a Video Foundation Model). Imagine this eye has watched millions of hours of human videos—people dancing, cooking, running, and playing. Because it has seen so much, it understands what "good movement" looks like, even if it's never seen a robot before. It can turn a video of a robot moving into a mathematical "fingerprint" that captures the essence of the action.

2. The "Matchmaker" (Optimal Transport)

This is the magic part.

The Setup: You give the system a tiny handful of examples (say, 10 pairs of videos) where a human has already said, "Video A is better than Video B."
The Mountain of Data: You also have a huge mountain of unlabeled videos (thousands of pairs) where no human has said anything yet.
The Matchmaking: VOTP uses a mathematical tool called Optimal Transport. Imagine you have a pile of "Good" fingerprints and a pile of "Bad" fingerprints from your 10 human examples. Now, you have a huge pile of "Unknown" fingerprints from the unlabeled videos.
The Connection: Optimal Transport acts like a super-efficient matchmaker. It looks at an "Unknown" video and asks, "Which of the 10 human-approved videos does this look most like?" It doesn't just guess; it calculates the best possible way to connect the unknown videos to the known ones based on how similar their "fingerprints" are.

3. The "Inference" (Pseudo-Labels)

Once the matchmaker connects an unknown video to a known "Good" video, VOTP says, "If this unknown video looks like the one the human liked, then this unknown video must also be good!" It automatically assigns a label (a "pseudo-label") to the thousands of videos you didn't have time to watch.

4. The Result

Now, instead of teaching the robot with just 10 human examples, the robot gets to learn from 10 human examples plus thousands of automatically labeled examples. The robot learns much faster and better, even though you only asked the human for a tiny bit of help.

What the Paper Found

The authors tested this on robots that had to walk (locomotion) and robots that had to move objects (manipulation). They also tested it on a real robot arm in a lab.

Less Human Work: VOTP achieved top results with only a handful of human labels (sometimes as few as 10).
Better than the Rest: It beat other methods that tried to do the same thing, often needing hundreds or thousands of labels to get similar results.
Robustness: Even if the lighting changed or there were distracting things in the background (like a video playing on a TV behind the robot), VOTP still figured out what was good and what was bad.
Real World: When they tried it on a real robot arm lifting a banana or opening a drawer, VOTP helped the robot succeed much more often than methods that relied only on the few human labels.

In short: VOTP is a way to teach robots by showing them a few examples of what humans like, and then using a "smart eye" and a "mathematical matchmaker" to figure out what the robot should do for the rest of the videos on its own. It saves humans from doing the boring, repetitive work of labeling thousands of videos.

Technical Summary: Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning

Problem Statement

Reinforcement Learning (RL) agents require suitable reward functions to solve decision-making tasks. In real-world scenarios, hand-engineering dense and informative rewards is challenging, often requiring extensive instrumentation (e.g., motion capture, tactile sensors) and risking reward misspecification where agents exploit unintended shortcuts. Preference-based RL (PbRL) offers an alternative by learning reward functions from human feedback on pairs of behavior trajectories. However, the scalability of PbRL is hindered by the high cost of human labeling, often requiring thousands of queries to learn effective reward functions. While prior work has explored semi-supervised learning and meta-learning to mitigate this, the potential of leveraging human visual perception—specifically the structured representation space of Video Foundation Models (ViFMs)—to infer preferences for unlabeled data remains underexplored.

Methodology: Video-based Optimal Transport Preference (VOTP)

The authors propose VOTP, a semi-supervised framework designed to learn effective reward functions from a minimal number of human preference labels (e.g., 10 queries) by leveraging unlabeled data from offline datasets. The method operates in two primary stages:

1. Trajectory Representation via ViFMs

Instead of relying on raw state-action pairs, VOTP models trajectory segments as short video clips. These clips are embedded into a latent space using an off-the-shelf Video Foundation Model (ViFM), specifically S3D pretrained on HowTo100M. This choice is motivated by the ability of ViFMs to capture both spatial details and temporal dynamics, providing actor-agnostic, semantically rich embeddings that are robust to nuisance variations (lighting, viewpoint) and generalize to unseen robotic environments.

2. Pseudo-Preference Label Generation via Optimal Transport

The core innovation of VOTP is the use of Optimal Transport (OT) to align labeled and unlabeled segment pairs within the ViFM latent space to generate pseudo-labels.

Setup: Let $L$ be the set of labeled segments with known preference relations encoded in a skew-symmetric matrix $R$ , and $U$ be the set of unlabeled segments.
Optimal Transport Plan: VOTP computes an OT plan $\mu^*$ that minimizes the transport cost (Euclidean or cosine distance in the latent space) between the empirical measures of $L$ and $U$ . This plan $\mu_{ii'}$ represents the probability that an unlabeled segment $\bar{\sigma}_{i'}$ corresponds to a labeled segment $\sigma_i$ .
Inference: The preference score for an unlabeled pair $(\bar{\sigma}_{i'}, \bar{\sigma}_{j'})$ is inferred by aggregating preferences from all labeled pairs, weighted by their relative alignment strengths derived from the OT plan:
$S(\bar{\sigma}_{i'}, \bar{\sigma}_{j'}) = \sum_{i,j} R_{ij}(\mu_{ii'}\mu_{jj'} - \mu_{ij'}\mu_{ji'})$
This score is normalized and thresholded to assign a pseudo-label ( $\tilde{y} \in \{0, 1, 0.5\}$ ).
Reward Learning: The reward function is trained using both the original labeled data and the newly pseudo-labeled data. During the offline RL phase, the learned reward function replaces the original rewards in the dataset to optimize the policy.

Key Contributions

VOTP Framework: Introduction of a semi-supervised framework that utilizes Optimal Transport over the latent space of Video Foundation Models to automatically assign preference labels to unlabeled trajectory pairs.
Feedback Efficiency: Demonstration that VOTP can learn effective reward functions and policies from only a handful of human preference queries (e.g., 10), significantly reducing the annotation burden compared to state-of-the-art offline PbRL methods.
Robustness to Visual Variation: Validation that leveraging ViFMs allows the method to maintain performance in the presence of visual distractors (lighting changes, texture variations, dynamic backgrounds) where other methods might struggle.
Real-World Applicability: Successful application of the framework to real-world robotic manipulation tasks (Lift Banana, Drawer Open) using a 7-DoF Sawyer arm, achieving higher success rates than baselines with minimal human input.

Experimental Results

The authors evaluated VOTP across locomotion (D4RL), manipulation (MetaWorld), and real-world robotic tasks.

Benchmark Performance: In D4RL and MetaWorld, VOTP consistently outperformed state-of-the-art offline PbRL baselines (including P-IQL, SURF, LiRE, and FTB) under limited feedback budgets. Notably, VOTP achieved performance comparable to an "Oracle" baseline (which has access to ground-truth rewards for unlabeled data) on D4RL tasks.
Low-Data Regime: In experiments varying the number of labels, VOTP required significantly fewer labels to reach task-reward performance compared to baselines. In the door-open task, VOTP with only 10 labels outperformed policies trained with ground-truth rewards.
Ablation Studies:
- ViFMs vs. IFMs: Video Foundation Models (S3D, VideoCLIP, InternVideo) outperformed Image Foundation Models (R3M, CLIP), highlighting the importance of capturing temporal dynamics for preference learning.
- OT vs. Similarity: The OT-based pseudo-labeling strategy significantly outperformed baselines that relied on simple similarity metrics (SIM-individual, SIM-mean, SIM-weighted), demonstrating the efficacy of the transport plan in aggregating preference information.
- Reward Correlation: The reward function learned by VOTP showed a significantly higher Pearson correlation with ground-truth rewards compared to P-IQL, indicating better coverage of state and action spaces.
Real Robot Evaluation: On a physical Sawyer robot, VOTP achieved 80% success on "Lift Banana" and 70% on "Drawer Open," outperforming Behavior Cloning (20-40%) and P-IQL (50%). Crucially, VOTP correctly distinguished between successful and failed trajectories in reward outputs, whereas P-IQL assigned high rewards to failed behaviors.

Significance and Claims

The paper claims that VOTP represents a scalable and practical solution for offline Preference-based RL. By harnessing the expressive representation space of pre-trained Video Foundation Models and the mathematical rigor of Optimal Transport, the method effectively bridges the gap between limited human supervision and the need for extensive reward coverage. The authors assert that this approach substantially reduces the need for human supervision while maintaining robustness against visual distractors, making it a viable path for deploying RL agents in complex, real-world environments where reward engineering is infeasible.

The authors acknowledge limitations regarding potential biases inherited from the pre-trained ViFMs and the computational cost of solving the OT plan for very large datasets, suggesting future work in approximate transport methods. However, within the scope of their experiments, the method proves highly effective for feedback-efficient learning.