Can Vision Language Models Judge Action Quality? An Empirical Evaluation

1. Problem Statement

Action Quality Assessment (AQA) is a computer vision task focused on automatically evaluating the execution quality of human actions, with applications in physical therapy, sports coaching, and competitive judging. Traditionally, AQA has been framed as a regression task (predicting a scalar score) or a classification task (identifying specific errors).

While Vision Language Models (VLMs) offer theoretical advantages for AQA—such as zero-shot generalization, conversational interaction, and explainable reasoning via Chain-of-Thought (CoT)—their actual efficacy in this domain remains uncharacterized. The core research question is: Can off-the-shelf, state-of-the-art VLMs reliably perform AQA across diverse human activities and tasks without task-specific fine-tuning?

2. Methodology

The authors conducted a comprehensive empirical evaluation involving five datasets, three VLM families, multiple preprocessing strategies, and seven prompting techniques.

2.1. Datasets and Tasks

The evaluation covered a broad spectrum of activities and modalities:

• LLM-FMS: Keyframe-based dataset for Functional Movement Screen (7 exercises, 1,812 frames). Task: Visual Question Answering (VQA).
• EgoExo-Fitness: Synchronized egocentric/exocentric videos (12 exercise types). Task: Adherence to technical guidelines (Binary classification).
• Fitness-AQA: Gym exercises (barbell rows, overhead press, squat). Task: Error detection (Binary classification).
• FineFS: Figure skating competition videos. Task: Grade of Execution (GOE) score regression (-5 to 5).
• MTL-AQA: Diving competition videos. Task: Execution score regression (0 to 10).

2.2. Models Evaluated

The study tested state-of-the-art models from three families, covering both "Instruct" and "Thinking" (reasoning) paradigms:

• Gemini 3.1 Pro (Closed-source).
• Qwen3-VL (Open-source, 235B parameters, Instruct and Thinking variants).
• InternVL3.5 (Open-source, 241B parameters, Standard and Thinking variants).

2.3. Experimental Variables

To isolate factors affecting performance, the authors manipulated:

1. Visual Preprocessing:
   • Cropped: Subject isolated via bounding box.
   • Skeleton Overlays: Skeleton joints drawn on original frames.
   • Skeleton-only: Frames replaced by skeleton renders on white background.
2. Prompt Engineering Strategies:
   • Base: Minimal task description.
   • Visual Grounding: Explicit instruction to analyze visual content first.
   • Two-Step: Generate a description before answering.
   • Guidelines: Providing positive (correct form) or negative (common errors) technical criteria.
   • Structured Reasoning: Enforcing a specific XML-tagged reasoning flow (Look $\to$ Decompose $\to$ Analyse $\to$ Assess $\to$ Output).
   • In-Context Learning (ICL): Providing few-shot examples in the context window.
3. Bias Mitigation:
   • Contrastive Tasks: Reformulating tasks to ask "Which of two videos is better?" to eliminate reliance on absolute priors or linguistic framing.

2.4. Evaluation Metrics

• Classification: Balanced Accuracy (to handle class imbalance).
• Regression: Spearman Rank Correlation ($\rho$) and Relative $\ell_2$ distance ($R\text{-}\ell_2$).

3. Key Results

3.1. Baseline Performance

• Near-Random Performance: Across all datasets and models, VLMs performed only marginally better than random guessing.
  • Classification: Balanced accuracy hovered around 0.50–0.60 (random is 0.50 for binary, ~0.33 for 3-class).
  • Regression: Spearman correlations were weak (mostly $< 0.30$), and $R\text{-}\ell_2$ errors were high.
• Model Differences: Gemini 3.1 Pro showed slight advantages on image-based tasks (LLM-FMS), while Qwen3-VL and InternVL3.5 showed inconsistent results across video datasets.

3.2. Impact of Preprocessing and Prompting

• Skeleton Data: Incorporating skeleton information (overlays or skeleton-only) did not consistently improve performance. In regression tasks, skeleton-only renders often degraded performance to near-zero correlation, suggesting VLMs struggle to interpret movement dynamics from skeletal abstractions alone.
• Prompt Engineering:
  • Strategies like Visual Grounding, Two-Step, and Structured Reasoning yielded no consistent gains.
  • Guidelines: Provided marginal, inconsistent improvements.
  • In-Context Learning (ICL): The only strategy showing meaningful gains, particularly for regression tasks (FineFS) and image datasets (LLM-FMS). However, ICL failed to generalize to video datasets due to context window limitations and token overhead.

3.3. Systematic Biases

The analysis uncovered two critical failure modes:

1. Correctness Bias: Models exhibit a strong tendency to predict that actions are performed correctly regardless of visual evidence. In classification tasks, the rate of "correct" predictions far exceeded the ground truth. In regression, models defaulted to conservative, mean-centric scores.
2. Linguistic Sensitivity: Models are highly susceptible to the phrasing of prompts.
   • When asked with "Positive Guidelines" (describing good form), models predicted higher correctness.
   • When asked with "Negative Guidelines" (describing errors), predictions shifted significantly, even though the semantic content was identical.
   • This indicates models rely on language priors rather than visual grounding.

3.4. Contrastive Task Results

When tasks were reformulated as contrastive comparisons (e.g., "Which video is better?"), performance remained poor (near chance), except for Gemini 3.1 Pro on LLM-FMS.

• Significance: This proves that the poor performance is not solely due to the identified biases (correctness bias or linguistic framing). Even when these biases are neutralized, VLMs lack the fundamental capability to perform fine-grained movement quality assessment.
• Score Gap Sensitivity: Accuracy improved only when the quality gap between two compared samples was large, indicating VLMs can detect obvious differences but fail at fine-grained discrimination.

4. Key Contributions

1. Rigorous Baseline: Establishes the first comprehensive benchmark for off-the-shelf VLMs in AQA, covering diverse domains (fitness, diving, skating) and tasks.
2. Failure Mode Analysis: Identifies and quantifies two systematic biases (Correctness Bias and Linguistic Sensitivity) that plague VLMs in this domain.
3. Negative Evidence on Preprocessing: Demonstrates that standard AQA preprocessing techniques (like skeleton extraction) do not transfer well to VLMs, challenging the assumption that VLMs can easily leverage traditional computer vision features.
4. Contrastive Evaluation: Introduces a contrastive evaluation framework to decouple model capability from linguistic priors, revealing that current VLMs lack the intrinsic reasoning required for fine-grained AQA.

5. Significance and Conclusion

The paper concludes that current state-of-the-art VLMs are not yet reliable for real-world Action Quality Assessment.

• Fundamental Limitation: The inability to perform AQA persists even after mitigating biases and applying advanced prompting. This suggests a fundamental gap in the models' ability to interpret fine-grained kinematic details and temporal dynamics from visual inputs.
• Implications: Researchers should not rely on zero-shot VLMs for AQA applications (e.g., medical rehab or sports coaching) without significant domain-specific fine-tuning or architectural changes.
• Future Work: The study calls for new training approaches that specifically address fine-grained movement quality, rather than relying on general-purpose pre-training. It provides a reference benchmark to track future progress in this specific niche.

In summary, while VLMs offer promising interfaces for AQA, their current "off-the-shelf" capabilities are insufficient, characterized by a reliance on prior knowledge and linguistic cues rather than genuine visual understanding of human movement quality.