💬 NLP

NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models

This paper introduces NextMotionQA, a comprehensive benchmark featuring multi-task evaluations across varying complexity levels to rigorously assess human motion understanding in vision-language models, revealing critical capability gaps and defining the limits of using VLMs as judges for fine-grained motion analysis.

Original authors: Yong Cao, Chuqiao Li, Xianghui Xie, Gerard Pons-Moll, Andreas Geiger

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yong Cao, Chuqiao Li, Xianghui Xie, Gerard Pons-Moll, Andreas Geiger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to understand human movement, like a dance instructor or a film director. To do this, you need a way to test if the robot actually "gets" what the person is doing, or if it's just guessing.

This paper introduces a new, much tougher test called NextMotionQA. Think of it as upgrading from a simple "True or False" quiz to a complex, multi-level video game for Artificial Intelligence (AI).

Here is a breakdown of what they did, using simple analogies:

1. The Problem: The Old Tests Were "Broken"

The authors looked at existing tests for AI motion understanding and found them wanting.

The Analogy: Imagine a driving test where the instructor asks, "Did the car move?" and the answer is always "Yes." It's too easy, and it doesn't tell you if the driver can actually park, merge, or handle a storm.
The Reality: Old tests had vague questions, didn't have different difficulty levels, and often had answers that even human experts couldn't agree on. If the "gold standard" answer is blurry, you can't tell if the AI is smart or just lucky.

2. The Solution: NextMotionQA (The "3x3x3" Challenge)

The team built a new benchmark with 1,307 expert-verified examples. They structured it like a 3D grid (3x3x3) to test AI from every angle:

3 Task Types (The "What"):
1. Multiple Choice (Recognition): "Which body parts are moving?" (Like picking the right ingredients from a list).
2. Captioning (Description): "Describe the motion in your own words." (Like writing a recipe for the dance).
3. Error Correction (Critique): "Here is a wrong description; find the mistake and fix it." (Like a proofreader finding typos in a manuscript).
3 Semantic Axes (The "Focus"):
- Body Parts: Which limbs are involved?
- Direction: Which way are they moving?
- Action: What is the specific move?
3 Difficulty Levels (The "Hardness"):
- Easy: Simple, single moves.
- Medium: Two moves in a row.
- Hard: Complex moves with speed changes or specific body parts.

The Result: They tested 12 different AI models (both open-source and big commercial ones). The results were surprising: No single AI was good at everything. Some were great at picking multiple-choice answers but terrible at writing descriptions. Others struggled with "direction" (left vs. right) across the board.

3. The "Judge" Experiment: Can AI Judge Other AIs?

Recently, people started using AI models to grade other AI models (like using a robot to grade a student's essay). The authors asked: If the AI is bad at understanding motion, is it also bad at judging motion?

The Analogy: Imagine using a robot to judge a gymnastics competition.
- The Finding: The AI judge was great at spotting obvious, big mistakes (like "The gymnast fell"). This is the "Coarse" level.
- The Failure: When asked to judge tiny, specific details (like "The gymnast's elbow was bent 5 degrees too much"), the AI judge completely fell apart. It couldn't agree with human experts.
The Takeaway: AI is a reliable judge for the "big picture," but it is currently useless for fine-tuning the tiny details of human movement.

4. Why This Matters

The paper doesn't claim this will immediately fix robots or cure diseases. Instead, it provides a diagnostic tool.

It tells us exactly where current AI fails (e.g., "They can't tell left from right," or "They can't write a good description").
It proves that just making AI models bigger doesn't always make them better at understanding motion.
It warns us that using AI to judge AI is risky if you need high-precision, detailed feedback.

In short: The authors built a harder, smarter test to show us exactly where today's AI is blind to human movement, and they showed us that while AI can be a good "general" judge, it's not yet a "specialist" judge for the fine details.

Technical Summary: NextMotionQA

Problem Statement

Reliable evaluation of human motion understanding is critical for advancing embodied AI, robotics, and animation. However, existing benchmarks suffer from three interrelated weaknesses:

Coarse Semantic Granularity: They often lack the specificity to diagnose where models fail (e.g., confusing body parts or directions).
Undifferentiated Difficulty: Benchmarks typically present a flat difficulty landscape, failing to stratify tasks by complexity.
Limited Annotation Quality: Pilot studies on existing datasets (e.g., HumanMotionQA) reveal low inter-annotator agreement among domain experts (34.3% unanimous agreement) and mean human accuracy (52.9%) that is comparable to or worse than specialized models. This suggests that reported model accuracies often reflect label noise rather than genuine capability.

Furthermore, while Vision-Language Models (VLMs) are increasingly used as "judges" for text-to-motion (T2M) generation, it remains unclear if their evaluation reliability degrades under fine-grained, part-level scrutiny similar to their performance in understanding tasks.

Methodology: NextMotionQA Benchmark

To address these gaps, the authors introduce NextMotionQA, a comprehensive benchmark comprising 1,307 expert-verified instances. The benchmark is structured as a 3×3×3 matrix:

Three Task Formats:
1. Multiple-Choice QA (T1): Multi-select questions probing recognition of body parts, directions, or actions.
2. Video Captioning (T2): Free-form description requiring the model to surface axis-relevant attributes in open vocabulary.
3. Fine-Grained Error Correction (T3): The model must identify errors in a corrupted caption and provide a semantically correct rewrite.
Three Semantic Axes:
1. Body-Part (A1): Which parts execute the motion.
2. Direction (A2): Translation direction (e.g., forward, lateral).
3. Action (A3): The semantic nature of the motion.
Three Difficulty Tiers:
- Easy: Single action, no modifiers.
- Medium: Two adjacent actions with fast transitions.
- Hard: Three+ actions or presence of modifiers (direction, speed, manner, fine-grained body parts).

Construction Pipeline

The dataset is constructed via a semi-automated, two-pass pipeline using a frontier VLM (Qwen3.6-Plus) followed by human expert verification:

Pass 1 (Metadata-Conditioned Drafting): The VLM drafts candidate items based on BABEL action labels and HumanML3D captions without viewing the video, ensuring the draft is grounded in symbolic labels.
Pass 2 (Video-Conditioned Refinement): The VLM reviews the draft against the rendered video to correct inconsistencies between metadata and visual reality.
Expert Verification: Three motion-domain experts independently judge items. Items are accepted only if all three agree; otherwise, they are revised or rejected. This process strictly enforces a rejection rubric to eliminate four failure modes identified in a pilot study: body-part granularity collisions, ambiguous spatial frames, temporal scope mismatches, and composite-action dominance.

VLM-as-a-Judge Protocol

In a complementary evaluation, the authors repurpose the tested VLMs as judges for T2M generation. They evaluate alignment between VLM ratings and human expert ratings across three complexity levels:

V1: Single-action prompts (recognition-level).
V2: Composite, chained actions.
V3: Fine-grained, temporally segmented motions requiring part-level resolution.

Key Results

1. VLM Performance on NextMotionQA

The authors evaluated 12 representative VLMs (open-source and closed-source).

Overall Ranking: Frontier closed-source models lead (Gemini-3.1-Flash: 58.44, Qwen3.6-Plus: 54.85). The strongest open-source model (Qwen3.5-27B) scores 49.75.
Task Disparity: Performance degrades significantly as task complexity increases.
- T1 (Recognition): Highest scores (e.g., Qwen3.6-Plus: 64.07 Jaccard).
- T2 (Captioning): Universal bottleneck; top models score ~43.53. Open-vocabulary production is strictly harder than closed-set selection.
- T3 (Correction): Performance drops further; models struggle to generate faithful corrections even when they can identify errors.
Semantic Weakness: Direction (A2) is a universal weak axis across all model families, with a V-shaped performance curve where direction scores are significantly lower than body-part or action scores.
Scaling Effects: Scaling generally helps (e.g., Qwen3.5 from 0.8B to 27B), but is not monotonic. Notably, InternVL3.5-14B underperforms its smaller 4B and 8B variants on T1, indicating parameter count alone does not guarantee motion competence.

2. VLM-as-a-Judge Reliability

The study validates the "VLM-as-a-judge" paradigm in specific regimes while highlighting its limits:

Coarse Criteria (V1): Strong alignment with human experts (Cohen's $\kappa = 0.70$ ). VLMs are reliable for system-level ranking of single-action generations.
Fine-Grained Criteria (V3): Alignment collapses on part-level judgment (Cohen's $\kappa = 0.10$ ). At this level, the VLM's system-level correlation with human preference flips sign ( $r = -0.146$ ), indicating it becomes anti-correlated with human preference for fine-grained editing.
Diagnosis: The failure in V3 mirrors the understanding gaps found in the main benchmark: VLMs struggle with joint resolution of body-part identity and temporal localization.

Significance and Contributions

The paper claims three primary contributions:

Diagnostic Benchmark: NextMotionQA provides the first unified benchmark with explicit difficulty stratification and multi-axis reasoning, moving beyond aggregate scores to localize model failures (e.g., direction vs. body-part).
Capability Gap Characterization: The evaluation reveals that no system dominates across all formats; specifically, open-vocabulary description and fine-grained correction remain significant bottlenecks, and translation direction is a universal weak point for current VLMs.
Judging Paradigm Limits: The work establishes that while VLMs are effective judges for coarse T2M evaluation, they are currently unsuitable for fine-grained, part-level assessment. This clarifies the operating envelope for VLM-as-a-judge protocols and identifies camera-frame grounding, temporal localization, and part-aware pretraining as necessary prerequisites for future progress.

The authors conclude that reliable human motion understanding requires moving beyond single-task evaluations and that current VLMs, while strong in coarse regimes, lack the specific grounding required for fine-grained motion critique.