Technical Summary: RoboProcessBench

Problem Statement

Vision-language models (VLMs) are increasingly deployed as "judges" in robotic manipulation systems, serving as visual critics, online reward generators, and failure detectors. However, these roles require more than just recognizing final task success or generating generic scene descriptions; they demand fine-grained process understanding. A model must be able to judge whether a manipulation is physically and temporally progressing correctly—detecting contact, identifying motion states, estimating local progress within a primitive, and reasoning about temporal order.

Existing benchmarks and evaluations largely focus on final outcomes, open-world perception, or planning capabilities. They fail to systematically test whether VLMs possess the specific, process-aware understanding required to audit intermediate execution steps. Consequently, there is a gap in evaluating and training VLMs on the structured signals (contact, phase, motion, temporal consistency) necessary for robust robotic process monitoring.

Methodology

Benchmark Design: RoboProcessBench

The authors introduce RoboProcessBench, a benchmark designed to decompose process-aware understanding into two complementary dimensions: Static Monitoring and Dynamic Reasoning. These dimensions are instantiated as 12 diagnostic question families:

• Static Monitoring: Tasks requiring the identification of the current local process state from a single frame or short clip.
  • T1: Phase Recognition (identifying the coarse process phase).
  • T2: Contact Detection (establishing task-relevant contact).
  • T4: Bimanual Coordination State (how two arms coordinate).
  • T10: Current Primitive Recognition (identifying the active low-level primitive).
• Dynamic Reasoning: Tasks requiring temporal, predictive, or chain-level inference.
  • T3: Motion Direction Prediction.
  • T5: Primitive-Local Progress (estimating if a step is early, middle, or late).
  • T6: Motion State Recognition (active vs. stationary).
  • T7: Operation Outcome Prediction (will the attempt succeed?).
  • T8: Temporal Ordering (reconstructing the chronological order of shuffled frames).
  • T9: Temporal Priority Prediction (determining which of two frames happened earlier).
  • T11: Next Primitive Prediction.
  • T12: Primitive Chain Restoration (filling a masked slot in an action chain).

Data Construction: ProcessData

The benchmark is built upon ProcessData, a curated corpus of approximately 58,000 question-answer pairs spanning 260 manipulation tasks.

• Source Integration: The data aggregates four distinct robotic manipulation sources, each contributing specific types of supervision:
  • GM-100: Long-tail goal-conditioned tasks.
  • RH20T: Contact-rich multimodal traces.
  • REASSEMBLE: Explicit primitive-level action chains.
  • AIST-Bimanual: Bimanual motion and coordination.
• Local Process Units (LPUs): Long manipulation episodes are decomposed into temporally localized decision units (LPUs) anchored by contact events, primitive intervals, or stable interaction regimes. This allows the benchmark to evaluate local sub-progress rather than just global episode completion.
• Splitting: ProcessData is strictly split into ProcessData-SFT (for supervised fine-tuning) and ProcessData-Eval (for evaluation), ensuring episode and recording isolation to prevent data leakage.

Experimental Protocol

The authors evaluated a broad set of open-source (e.g., Qwen-VL, InternVL, RoboBrain) and closed-source (e.g., Gemini, GPT, Claude) VLMs on ProcessData-Eval. Additionally, they performed supervised fine-tuning (SFT) on Qwen2.5-VL-7B and InternVL-3-8B using ProcessData-SFT to test the learnability of process signals.

Key Results

1. Diagnostic Limitations of Zero-Shot Models

Evaluation on ProcessData-Eval reveals that current VLMs lack robust, uniform process-aware understanding:

• Fragmented Performance: No single zero-shot model dominates all task families. Performance varies significantly across different process cues (e.g., a model might excel at motion direction but fail at temporal ordering).
• Local State vs. Progress: Tasks grounded in visible local states (Contact Detection, Motion State) are easier for zero-shot models. In contrast, Primitive-Local Progress (T5) remains a significant bottleneck, with the strongest zero-shot baseline achieving only ~34.4% accuracy (near the 33.3% random baseline).
• Temporal Reasoning Deficit: Temporal ordering (T8) and priority prediction (T9) remain close to chance levels for most models, indicating that short visual contexts do not automatically yield reliable temporal reasoning in robotic manipulation.

2. Learnability via ProcessData-SFT

Supervised fine-tuning on ProcessData-SFT demonstrates that process-aware signals are learnable:

• Significant Gains: Both post-trained models (ProcessData-SFT-Qwen and ProcessData-SFT-Intern) showed consistent improvements over their zero-shot baselines across static monitoring and dynamic reasoning.
• Specific Improvements:
  • Static Monitoring: Improved by ~43–44 points, reaching ~77% accuracy.
  • Dynamic Reasoning: Improved by ~21–24 points, reaching ~69% accuracy.
  • Progress Estimation: T5 accuracy improved by roughly 11 points (from ~34% to ~45%), suggesting that structured supervision can teach models to recognize visual evidence of advancement.
• Persistent Bottlenecks: Despite gains, temporal ordering (T8) remained near chance levels even after post-training, and outcome prediction (T7) showed inconsistent improvements. This suggests that while local state and primitive transitions are learnable, robust temporal reconstruction and outcome prediction require further advancements (e.g., stronger video modeling or memory).

Significance and Claims

The paper positions RoboProcessBench not as a replacement for closed-loop policy benchmarks, but as a diagnostic tool and learnable substrate for developing VLM-based process evaluators.

• Diagnostic Value: The benchmark decomposes process understanding into 12 separable dimensions, revealing that current models have fragmented capabilities. It highlights specific weaknesses in primitive-local progress estimation and temporal reasoning that aggregate scores would obscure.
• Supervision Source: The results demonstrate that process-aware understanding is not an inherent, unlearnable trait of VLMs. Structured supervision derived from physically grounded traces (ProcessData-SFT) can significantly enhance a model's ability to monitor local states, motion, and primitive transitions.
• Foundation for Future Systems: By establishing that process signals are learnable, RoboProcessBench provides a foundation for developing future "judge-like" robotic modules, such as Process Reward Models (PRMs) and progress-aware critics, which require fine-grained understanding of execution unfolding rather than just final outcomes.

The authors conclude that while current models are far from uniformly process-aware, the benchmark successfully identifies the specific gaps and provides a pathway for training VLMs to better monitor and evaluate robotic manipulation processes.