OpenGVL -- Benchmarking Visual Temporal Progress for Data Curation

Technical Summary: OpenGVL – Benchmarking Visual Temporal Progress for Data Curation

Problem Statement

The rapid advancement of robotics hardware and Vision-Language-Action (VLA) models has led to an exponential increase in available robotics data. However, a critical bottleneck remains: data scarcity and the lack of high-quality, well-curated datasets. While millions of episodes are being shared on platforms like Hugging Face, much of this data is "in the wild," meaning it often contains ambiguous instructions, sensor occlusions, failed executions, or out-of-distribution (OOD) examples. Manually filtering these massive datasets is economically and logistically unfeasible. There is an urgent need for automated tools that can assess the quality and temporal progress of robotic tasks to facilitate large-scale data curation.

Methodology

The authors build upon the Generative Value Learning (GVL) framework. GVL leverages the world knowledge embedded in Vision-Language Models (VLMs) to estimate a "value function"—a scalar score representing task progress—from visual observations.

1. Core Metric (VOC): To evaluate how well a model predicts progress, the authors use the Value-Order Correlation (VOC). This is a rank correlation (Spearman or Kendall) between the predicted values and the actual temporal order of frames in a trajectory. A VOC of 1 indicates perfect monotonic progress, while -1 indicates perfect inverse correlation.
2. OpenGVL Benchmark: The authors developed a comprehensive benchmark to evaluate various VLMs. To ensure scientific rigor and prevent data contamination, they created two "hidden" datasets involving complex, long-horizon tasks (electronic assembly) that were not part of the training sets of the evaluated models.
3. Experimental Setup:
   • Conditioning: Models are tested in zero-shot and two-shot (providing two example trajectories) scenarios.
   • Robustness: To prevent models from simply relying on the temporal order of the input sequence, the authors shuffle the input frames, forcing the model to rely on visual content rather than sequence position.
   • Model Diversity: The benchmark evaluates a wide spectrum of models, including open-source families (Gemma-3, Qwen2.5-VL), reasoning-capable models (GLM-4.1V-Thinking, MiMo-VL, Cosmos-Reason1), and proprietary models (GPT-4o, Gemini-2.5 Pro).

Key Contributions

• OpenGVL Benchmark: A new, standardized benchmark and codebase for estimating temporal task progress in robotics.
• Performance Gap Analysis: A systematic study revealing the disparity between open-source and closed-source VLMs in spatial and temporal reasoning.
• Automated Data Curation Framework: A practical demonstration of using VOC scores to identify three major data quality issues:
  1. Task Definition Problems: Ambiguous instructions (e.g., "dig grass") that make progress hard to define.
  2. Labeling Ambiguity: Unclear task boundaries or multiple valid ways to complete a task.
  3. OOD/Failed Examples: Identifying specific "bad" episodes (e.g., sensor failures or execution errors) within otherwise good datasets.
• Open-Source Infrastructure: Release of the benchmark, code, and an interactive Hugging Face Space for community contribution.

Key Results

• Scaling Laws: The results demonstrate that VLM scale significantly impacts performance; larger models (e.g., Qwen2.5-VL-32B, Gemma-3-27B) achieve much higher VOC scores than their smaller counterparts.
• The Proprietary Gap: Open-source models currently achieve only approximately 60–70% of the performance of closed-source models (like Gemini-2.5 Pro) on temporal progress tasks. This highlights a significant deficiency in the spatial and temporal reasoning capabilities of current open-source VLMs.
• Reasoning Models: Models with explicit "thinking" mechanisms (e.g., MiMo-VL-7B-RL) show strong performance, suggesting that chain-of-thought reasoning is a viable path toward closing the performance gap.
• Curation Efficacy: The authors successfully used OpenGVL to detect "noisy" episodes in real-world datasets, proving it can act as an automated "quality filter" for training VLA models.

Significance

OpenGVL shifts the focus from merely collecting robotics data to understanding and refining it. By providing a scalable way to annotate and filter datasets, OpenGVL addresses one of the most significant hurdles in the path toward general-purpose robotic intelligence. It provides the community with a tool to transform "raw, noisy data" into "high-quality training signals," which is essential for training the next generation of foundation models for embodied AI.