Prompt-to-Gesture: Measuring the Capabilities of Image-to-Video Deictic Gesture Generation

1. Problem Statement

Gesture recognition research faces a critical bottleneck: data scarcity. Unlike Natural Language Processing (NLP), which benefits from massive, standardized datasets, gesture corpora are limited by:

• High Costs: Collecting data requires expensive human recording sessions, extensive annotation, and ethical approvals.
• Limited Variability: Existing datasets are often constrained to controlled laboratory settings, lacking the "in-the-wild" variability (lighting, background noise, motion speed) necessary for robust generalization.
• Imbalance: There is a significant disparity in research output and infrastructure between NLP and nonverbal communication (gesture) research.

Current synthetic approaches often rely on pose-level representations (e.g., skeletons) that lack photorealism, making them unsuitable for downstream perception tasks requiring visual fidelity. While large-scale image-to-video foundation models (e.g., SORA, Veo) exist, their specific capability to generate authentic, semantically consistent deictic (pointing) gestures for human-robot interaction (HRI) remains unexplored.

2. Methodology

The authors propose a zero-shot data generation pipeline that leverages state-of-the-art image-to-video models to synthesize realistic pointing gestures.

A. Data Source & Reference

• Real Dataset: A custom dataset of 68 videos featuring 8 human participants pointing at YCB objects on a table in front of a humanoid robot (NICOL). Faces were anonymized using synthetic replacements.
• Reference Frames: The pipeline uses start and end keyframes sampled from this real dataset to guide the generation process, ensuring visual consistency of the subject and environment.

B. Generation Model

• Model: Vidu (specifically the Q3 version), a transformer-based image-to-video model using a U-ViT backbone with diffusion-based denoising.
• Rationale: Vidu was chosen over models like SORA or Veo because it offers superior reference-guided generation, maintaining the consistency of the human subject's appearance and the environment across frames, which is crucial for gesture tasks.

C. Prompt Engineering Pipeline

The system uses structured text prompts to guide the generation in a zero-shot manner. Each prompt consists of four blocks:

1. Participant Description: Clothing and appearance to ensure identity consistency.
2. Pose Description: Specifics of the pointing action (e.g., "intently pointing at the object").
3. Environment Description: Background details to introduce noise (e.g., people walking, office setting).
4. Camera Settings: Shot size, angle, and motion (e.g., static, fast-forward, slow-motion).

• Output: For each prompt, four 8-second video samples are generated. The final synthetic dataset comprises 1,632 videos covering various conditions (static, noisy, fast/slow motion, dynamic shifts, color shifts).

3. Key Contributions

1. Synthetic Gesture Pipeline: A novel, accessible pipeline for generating photorealistic deictic gestures using image-to-video models, bridging the gap between behavioral psychology/robotics and Generative AI.
2. Comprehensive Evaluation Framework: A multi-dimensional evaluation strategy assessing:
   • Visual Fidelity: Using Fréchet Inception Distance (FID) and Fréchet Video Distance (FVD).
   • Semantic Alignment: Using CLIP similarity and a new metric called Gesture Alignment Score (GAS).
   • Kinematic Analysis: Analyzing motion derivatives (velocity, acceleration, jerk) and joint angle distributions (KL divergence, Earth Mover's Distance).
   • Downstream Utility: Testing the synthetic data's ability to train gesture recognition models.
3. Open Resource: The release of both the real and synthetic datasets, along with the generation prompts, to the research community.

4. Key Results

A. Quality and Fidelity

• Hand Detection: Synthetic gestures achieved hand confidence scores (via MediaPipe) comparable to or slightly higher than real data (0.77 vs. 0.75), indicating no significant detection artifacts.
• Visual & Temporal Coherence:
  • Static/Slow Motion: Showed low FID (<30) and FVD, indicating high similarity to real data.
  • Fast Motion/Noise: Showed higher FVD, highlighting the current difficulty in maintaining temporal coherence during rapid transitions, though frame-level quality (FID) remained acceptable.
• Semantic Alignment: High CLIP similarity scores (0.94–0.96) confirmed that the synthetic videos semantically match the real data (e.g., correct objects, environment).
• Gesture Alignment Score (GAS): A new metric combining Visual Alignment Score (VAS) and Prompt Alignment Score (PAS). Results showed that while fast motion introduced variability, the overall GAS remained stable, proving the model's robustness.

B. Kinematic and Anatomical Validity

• Motion Derivatives: Synthetic data exhibited realistic velocity, acceleration, and jerk profiles. "Noisy" scenes showed higher motion derivatives, while "slow" scenes showed subtle motion, mirroring real-world variability.
• Joint Angles: The distribution of finger joint angles (MCP, PIP, DIP) in synthetic data showed moderate KL divergence from real data. Crucially, the index finger's MCP joint (critical for pointing direction) had very low divergence, proving the semantic structure of the gesture was preserved.

C. Downstream Performance (Machine Learning)

The authors trained three models (CNNLSTM, MM-ITF, VideoMAE) under three paradigms:

1. Baseline: Trained on Real Data only.
2. Pre-training: Trained on Synthetic Data, tested on Real Data.
3. Fine-tuning: Pre-trained on Synthetic, then Fine-tuned on Real.

Findings:

• Hybrid Training (Fine-tuning) yielded the best performance across all models (e.g., VideoMAE achieved 95.0% accuracy and 94.4% F1 with hybrid training vs. 85.0% baseline).
• Pre-training on synthetic data alone generalized well to real-world "out-of-distribution" (OOD) scenarios, outperforming the baseline in some metrics.
• This confirms that synthetic data is not just visually plausible but semantically useful for training robust gesture recognition systems.

5. Significance and Limitations

Significance

• Scalability: Offers a cost-effective method to generate large-scale, diverse gesture datasets without the logistical burden of human recording.
• Generalization: Demonstrates that AI-generated data can improve the generalization of deep learning models, particularly for underrepresented gestures or scenarios.
• Cross-Disciplinary Impact: Provides a tool for researchers in robotics, psychology, and HCI who may lack deep ML expertise to generate reproducible gesture data.

Limitations

• Temporal Artifacts: Fast motion and complex transitions still suffer from jitter or unnatural limb distortions.
• Model Bias: The generative model exhibited internal biases toward "cinematic" or "anime-like" motions and occasional filler movements not requested by the prompt.
• Control Granularity: The current zero-shot approach lacks fine-grained control parameters (e.g., guidance scale, seed) available in diffusion models, limiting reproducibility.
• Few-Shot Learning: The pipeline currently relies on reference frames but lacks video-based in-context learning for specific cultural or regional gestures.

Conclusion

The paper establishes that image-to-video foundation models are a viable, powerful tool for synthesizing deictic gestures. The generated data is visually realistic, kinematically plausible, and, most importantly, effective for training downstream machine learning models, often outperforming models trained solely on limited real-world data. This work paves the way for a new era of data augmentation in gesture research, moving beyond static poses to dynamic, photorealistic video generation.