Technical Summary: ProcVLM

Problem Statement

Long-horizon robotic manipulation requires dense feedback signals that accurately reflect task advancement through procedural stages, rather than relying solely on binary success/failure outcomes at the trajectory level. Existing reward models often depend on sparse terminal labels or time-based interpolation. The authors argue that these approaches fail to capture critical states such as unfinished steps, stagnation, and failure recovery, as time elapsed does not necessarily correlate with true task progress in complex manipulation tasks. Furthermore, most large-scale robot datasets lack explicit annotations of procedural structure (e.g., subtask boundaries and remaining actions), forcing reward models to rely on indirect supervision that conflates temporal proxies with semantic progress.

Methodology

1. Procedural Supervision Synthesis (ProcCorpus-60M)

To address the lack of dense procedural supervision, the authors developed a scalable pipeline to synthesize frame-level annotations from raw robot trajectories.

• Hierarchical Annotation: Using large Vision-Language Models (VLMs) as annotators, the pipeline first infers high-level task plans and temporal spans of subtasks from the entire video. It then expands these segments into frame-wise assignments, generating reasoning cues (state reasoning, completion status, remaining actions) on visually deduplicated keyframes.
• Dataset Construction: This pipeline was applied to 30 embodied datasets (including DROID, BridgeData V2, Fractal, and simulation benchmarks), resulting in ProcCorpus-60M, containing over 400K trajectories and 60M annotated frames.
• Progress Labeling: Continuous progress targets are derived by combining subtask-level structure with intra-subtask visual motion. A progress budget is assigned to each subtask based on its relative duration, and this budget is distributed over time according to local visual changes ($\|\dot{\phi}(\tau)\|$), avoiding simple linear time interpolation.

2. ProcVQA and Training Paradigm

The annotations are converted into ProcVQA, a 20B-token training corpus centered on task progress estimation. The training follows a "reasoning-before-estimation" paradigm, where the model must first infer the current execution stage and remaining atomic actions before predicting a continuous progress score.

• Task Families:
  • Subtask-structured progress prediction: The model outputs remaining atomic subtasks followed by a continuous completion percentage wrapped in <progress> tags.
  • Action segmentation: Predicting temporal boundaries of atomic actions.
  • Future planning: Predicting the sequence of remaining actions required to complete the task.
• Architecture: ProcVLM is built on a Qwen3-VL-2B-Instruct backbone. It utilizes a standard autoregressive language modeling head for text generation and a dedicated progress value head (a 3-layer MLP) for continuous regression.
• Training Objectives: A joint loss function combines the language modeling loss ($L_{LM}$) with a regression loss ($L_{value}$) for the progress score. A gating mechanism activates the value head only when progress supervision is present.
• Two-Stage Training:
  1. Pretraining: Large-scale training on the full ProcVQA dataset (~20B tokens) to learn general procedural representations.
  2. Refinement: Fine-tuning on a curated, high-quality subset (~2.8B tokens) to sharpen subtask alignment and progress estimation.

Key Contributions

1. ProcVLM: A progress-aware embodied VLM that predicts dense, procedure-grounded rewards. It links continuous progress estimation to subtask semantics, remaining actions, and within-stage advancement via a reasoning-before-estimation paradigm.
2. Scalable Supervision Pipeline: A method for converting raw manipulation trajectories into frame-level, subtask-structured annotations, yielding ProcCorpus-60M and the ProcVQA benchmark.
3. Systematic Evaluation: Comprehensive evaluation across procedure-aware understanding, reward modeling, cross-task adaptation, and downstream policy optimization.

Experimental Results

Embodied Procedural Understanding (ProcVQA)

Evaluated on in-distribution (ID) and out-of-distribution (OOD) splits, ProcVLM outperformed mainstream VLMs (including GPT-5.4, Gemini 3.1 Pro, and Qwen3.5) across action segmentation, future planning, and task progress estimation. It achieved the highest BF1@5, Success, VOC (Value-Order Correlation), and EPR@50 (Effective Progress Resolution) scores.

Generalizable Progress Reward Evaluation

• Zero-Shot Comparison: On the ProcVQA-OOD subset, ProcVLM achieved the best VOC, indicating superior trajectory-internal progress ordering compared to baselines like Robometer and RoboDopamine.
• One-Shot Adaptation: On the RoboFAC benchmark (real-robot failure analysis), adding a single successful demonstration improved ProcVLM's VOC by 85.7% and MCC by 18.8%. In contrast, baselines like Robometer showed minimal improvement, suggesting sparse preference supervision is less effective for single-demonstration adaptation.
• Ablation Studies: Removing procedure-aware pretraining caused a severe degradation (e.g., -88.8% in MAE for fault localization), confirming that large-scale pretraining is essential for transferable progress perception. Removing reasoning-formatted supervision also led to consistent drops, highlighting the importance of explicit subtask structure.

Downstream Policy Optimization

In reward-guided fine-tuning (RFT) experiments using the Evo-RL framework:

• Simulation (LIBERO-10): RFT with ProcVLM rewards yielded moderate gains (+0.4 to +1.2 percentage points) over supervised fine-tuning (SFT).
• Real Robot (Stack Bowls): RFT demonstrated significant improvements, increasing success rates by +25.0% at 5k steps and +12.5% at 10k steps compared to SFT. The authors attribute this to ProcVLM's ability to downweight noisy segments (e.g., repeated grasp retries) in teleoperated data, stabilizing policy optimization.

Significance and Claims

The paper claims that procedure-grounded progress estimation is a critical enabler for transferable reward modeling in robotics. By grounding progress in procedural structure rather than temporal proxies, ProcVLM provides discriminative feedback that supports stable policy improvement, particularly in scenarios involving noisy real-robot data and long-horizon tasks. The work suggests that large-scale procedure-aware pretraining is a promising route for developing reward models that can generalize across tasks and failure modes, facilitating reward-guided policy optimization beyond offline demonstrations.

The authors acknowledge limitations, noting that estimates depend on the quality of subtask decomposition and that the current work focuses on a specific set of reward-guided optimization settings rather than exhaustively covering all policy-gradient or preference-based integrations.