Eval-Actions: Fine-Grained Execution Quality Evaluation for Robotic Manipulation

Technical Summary: Eval-Actions

Problem Statement
Current evaluation protocols for Vision–Action (VA) and Vision–Language–Action (VLA) robotic manipulation policies rely predominantly on binary success rates (i.e., whether a task was completed). This outcome-centric metric obscures critical process-level differences among executions that achieve the same goal. Two policies may both successfully place an object, yet one may do so with smooth, efficient motion while the other exhibits jerky trajectories, multiple retries, or visible collisions. The paper argues that binary labels fail to capture execution quality, deployment reliability, and diagnostic value, particularly in real-robot scenarios where motion stability and contact behavior are paramount.

Methodology
The authors propose Eval-Actions, a diagnostic evaluation methodology and real-robot benchmark designed to assess execution quality through fine-grained Action Quality Assessment (AQA). The framework introduces a triadic supervision protocol:

1. Expert Grading (EG): Criteria-based expert scoring (1–10 scale) derived from unified rubrics covering task completion, motion smoothness, visible collision-related events, and execution efficiency.
2. Rank-Guided (RG) Labels: A metric-grounded approach that calibrates measurable physical indicators (e.g., joint velocity variance, acceleration, collision penalties) against expert rankings. A Genetic Algorithm optimizes the weights of these indicators to align rule-based scores with human consensus, creating a reproducible, physical-grounded quality label.
3. Chain-of-Thought (CoT) Annotations: Textual diagnostic explanations that describe observable evidence (e.g., "motion is jerky," "object dropped and re-grasped") linking specific criteria to the final quality assessment.

To operationalize this, the authors introduce AutoEval, a reference multimodal evaluator based on Vision-Language Models (VLMs). AutoEval processes RGB-D video sequences and compact kinematic summaries (joint states, velocity, and acceleration statistics). It features two variants:

• AutoEval-S: Performs structured prediction of quality scores (EG/RG) and success labels. It employs a Spatio-Temporal Aggregation strategy, tiling neighboring frames to preserve short-term motion cues within fixed VLM input budgets.
• AutoEval-P: Generates CoT-style diagnostic explanations. It utilizes Group Relative Policy Optimization (GRPO) with a hybrid reward function to ensure consistency between the generated text, the predicted score, and the success label.

Key Contributions

• Formulation: The paper reframes robotic manipulation evaluation as a fine-grained AQA problem, moving beyond binary success to assess process quality.
• Benchmark (Eval-Actions): A real-robot dataset containing 13,000+ episodes across 150+ tasks (approx. 52 hours of recordings). It uniquely includes a high proportion of failure cases and low-quality successes, providing RGB-D videos, robot-state trajectories, and the EG/RG/CoT annotations. A densely annotated subset (EAS) is provided for training and evaluation.
• Evaluator (AutoEval): A multimodal framework that predicts quality scores and diagnostic explanations from visual and kinematic data, validated through in-distribution testing, generalization splits, and physical-metric baselines.

Results
On the Eval-Actions test split, the proposed methods demonstrate strong alignment with human evaluation:

• AutoEval-S achieves Spearman Rank Correlation Coefficients (SRCC) of 0.81 (under EG) and 0.84 (under RG).
• AutoEval-P achieves an SRCC of 0.70 for CoT-based scoring.
• Success detection accuracies reach 90.6% (EG) and 91.0% (RG).
• Comparative analysis shows that while direct physical scoring (without expert calibration) improves over binary success (0.78 vs. 0.53 SRCC against EG), the Rank-Guided (RG) approach (0.90 SRCC) significantly better approximates human consensus, validating the necessity of human-guided calibration for physical metrics.
• Expert inter-annotator reliability is high (ICC(2,K) = 0.99), establishing the EG scores as a robust reference.

Significance
The paper claims that Eval-Actions provides standardized, interpretable diagnostic signals that complement traditional success-rate evaluation. By explicitly distinguishing criteria-based expert judgments from measurable kinematic indicators and providing explanatory annotations, the framework enables systematic analysis of execution quality. This allows researchers to diagnose policy failures, compare execution behaviors (e.g., smoothness vs. retries), and evaluate policies on dimensions beyond mere task completion, addressing a critical gap in the current robot learning evaluation landscape.