Technical Summary: CoMPAS3D

Problem Statement

The development of socially interactive humanoid robots requires systems capable of engaging with humans through physical movement, adapting in real-time to a partner's intent, ability, and cues. While research in interactive motion generation has advanced, current evaluation frameworks are insufficient for assessing the semantic quality of these interactions. Existing metrics primarily rely on kinematic measures (e.g., Fréchet Inception Distance, beat alignment) that evaluate physical realism and distribution similarity but fail to determine if generated motion is legible within a shared movement vocabulary or appropriate to the proficiency level of the interaction partner.

This gap exists because:

1. Current evaluation frameworks lack metrics for semantic alignment and partner adaptation.
2. Existing improvised interaction datasets lack fine-grained move annotations and variation in proficiency levels.
3. Most available datasets focus on scripted, short-duration interactions or professional-only performers, lacking the diversity required to test skill-appropriate generation.

Methodology

The authors introduce CoMPAS3D (Complex Multi-Level Person-Interaction Annotated Salsa Dataset) and a corresponding three-level evaluation framework.

The CoMPAS3D Dataset

• Content: 3 hours of improvised salsa partner dance involving 18 dancers (9 pairs) across three proficiency levels: beginner, intermediate, and professional.
• Data Capture: Recorded using a Vicon motion capture system (20 cameras, 120 fps) with 53 markers per dancer, resulting in 55-joint SMPL-X trajectories.
• Annotations: Over 2,800 expert-annotated segments (8-beat cycles) covering 30 move categories, execution errors (e.g., off-beat, mixed signals), and stylistic elements. Annotations were performed by an expert with 15 years of experience and verified by a second expert (Cohen's Kappa = 0.752).
• Design Philosophy: Analogous to the Switchboard speech corpus, CoMPAS3D prioritizes depth within a single movement vocabulary (salsa) rather than shallow coverage across multiple genres, enabling the study of naturalistic, improvised dyadic interaction.

Benchmark Tasks

Drawing an analogy between partner dance and spoken dialogue, the paper defines three tasks:

1. Move Classification: Identifying the specific salsa move performed (analogous to transcription).
2. Proficiency Estimation: Determining the skill level of the dancing pair (analogous to fluency assessment).
3. Follower Generation: Predicting the follower's motion given the leader's cues, music, and a target proficiency level (analogous to dialogue response generation).

Evaluation Framework

The framework evaluates generated follower motion using:

• Kinematic Metrics: Standard metrics including FID (kinematic/graphical), diversity, cross-distance FID, Beat Echo Degree (BED), and Beat Alignment Score (BAS).
• Objective Semantic Metrics:
  • Move Legibility: Measured by the F1-score of a fine-tuned Vision-Language Model (VLM) classifying moves in generated sequences.
  • Proficiency Appropriateness: Measured by the F1-score of a fine-tuned VLM estimating the proficiency level of generated sequences.
• Subjective Metrics: Human evaluations across six real-world competition dimensions: timing, musicality, technique, difficulty, partner coordination, and originality.

Key Contributions

1. CoMPAS3D Dataset: The first publicly available improvised partner dance dataset featuring expert move transcriptions and spanning three proficiency levels.
2. Benchmark Tasks & Models: Benchmarks for three pretrained and fine-tuned Vision-Language Models (Qwen2.5-VL, LLaVA-NeXT-Video, InternVL3) on move classification and proficiency estimation.
3. New Evaluation Metrics: Objective metrics for reactive dance generation that assess move legibility and proficiency appropriateness, revealing failures that kinematic metrics miss.
4. Subjective Evaluation: A human study utilizing community-based dance competition criteria, introducing three new metrics compared to prior dance generation work.

Experimental Results

The framework was applied to two state-of-the-art generation methods: Duolando (dance-specific) and InterGen (general human-human interaction).

• Model Performance on Benchmarks: Fine-tuned VLMs significantly outperformed zero-shot baselines. In dyadic settings, Qwen2.5-VL achieved 69.87% accuracy and 49.30% F1 for move classification, and 84.63% accuracy for proficiency estimation.
• Kinematic Evaluation: Both Duolando and InterGen showed comparable kinematic scores to each other, with FID and diversity metrics suggesting reasonable physical realism.
• Objective Semantic Evaluation: Both generative methods failed significantly compared to ground truth on the new objective metrics:
  • Move Legibility: Ground truth achieved a Move F1 of 53.55, while InterGen and Duolando scored 7.69 and 6.70, respectively.
  • Proficiency Appropriateness: Ground truth achieved a Proficiency F1 of 51.64, while InterGen and Duolando scored 37.12 and 22.47, respectively.
• Subjective Evaluation: Human evaluators rated ground truth significantly higher (p < 0.001) than both generative methods across all six competition dimensions. InterGen and Duolando received comparable scores, consistent with their similar kinematic performance.

Significance and Claims

The paper claims that CoMPAS3D and its evaluation framework address a critical gap in socially interactive embodied AI. By moving beyond kinematic realism, the authors demonstrate that current generative methods produce follower motions that are not legible within the shared movement vocabulary and are inappropriate for the partner's skill level, despite achieving comparable kinematic scores.

The authors position CoMPAS3D as a foundational resource similar to early naturalistic speech datasets, enabling the study of responsive, context-aware physical interaction. They assert that their objective metrics (legibility and proficiency) serve as meaningful measures for semantic quality, analogous to how Automatic Speech Recognition (ASR) is used to evaluate speech synthesis intelligibility. The work highlights that while physical realism is a saturated measure, the ability to generate socially appropriate and semantically clear motion remains a significant open challenge.