Technical Summary: InterPet4D and InterPetMoGen

1. Problem Statement

The estimation and generation of human-pet interactions remain significantly underexplored in computer vision and robotics. Existing research has largely focused on human-human or human-object interactions, primarily due to the absence of large-scale, high-quality datasets capturing natural cross-species dynamics. Modeling these interactions presents unique challenges:

• Data Scarcity: Achieving structured, repeatable coordination between humans and animals is difficult without controlled training, hindering large-scale data collection.
• Occlusion: Close-range interactions often result in severe cross-occlusion, degrading the reconstruction of fine-grained details.
• Morphological Asymmetry: The structural differences between humans and animals complicate the modeling of joint distributions compared to same-species interactions.
• Lack of Multimodal Signals: Existing animal datasets rarely capture synchronized audio, human motion, and pet motion simultaneously.

2. Methodology

2.1 The InterPet4D Dataset

The authors introduce InterPet4D, the first large-scale multimodal 4D dataset dedicated to natural human-dog interactions.

• Data Collection: The dataset comprises 6.83 million frames collected from 23 human participants (including 11 professional trainers) and 13 dogs across 11 breeds.
• Capture Setup: A synchronized multi-view system was employed, featuring:
  • Third-Person View: 12 wire-synchronized GoPro Hero13 cameras (1920×1080, 60 fps) arranged in a square, with 8 cameras positioned low to capture pet and hand details.
  • Egocentric View: Ray-Ban Meta glasses worn by participants, providing first-person RGB video and audio.
  • Synchronization: Achieved via software alignment using hand-clapping signals, with a maximum error of one frame.
• Interaction Taxonomy: The data covers four structured categories:
  1. Petting: Physical contact and postural responses.
  2. Commanding: Verbal and gestural commands (e.g., "sit," "shake").
  3. Calling: Distance-based voice and hand beckoning.
  4. Free-form: Unscripted interactions (e.g., fetch, tug-of-war).
• Annotations: The dataset includes synchronized multiview/egocentric videos, segmentations, 2D/3D keypoints, meshes (SMPL-X/MANO for humans, SMAL for dogs), and aligned audio tracks with text captions generated via Large Language Models (LLMs).

2.2 Data Processing Pipeline

• Human Reconstruction: Utilizes a pipeline combining Mask R-CNN, HRNet-WholeBody, and DeepSORT for localization and tracking. 2D keypoints are triangulated and refined using energy functions (symmetry, temporal smoothness, bone-length constraints) before fitting SMPL-X and MANO models.
• Dog Reconstruction: Detects dogs via RTMDet and estimates 2D keypoints using HRNet. Human masks filter occluded regions. 3D joints are triangulated and smoothed (RTS smoother) before fitting the SMAL model. Shape parameters are stabilized by manual fitting on a canonical frame.
• Audio/Text Processing: Raw audio is transcribed and translated to English using ASR. Temporal boundaries are annotated, and interaction descriptions are generated using a Qwen3-based LLM.

2.3 InterPetMoGen (IPMG) Framework

The authors propose InterPetMoGen, a MotionGPT-based framework for gesture-to-pet motion generation.

• Objective: To learn the conditional distribution $p(P | H, A)$, where $P$ is the pet motion sequence, $H$ is the human motion (body + hands), and $A$ is the audio sequence.
• Architecture:
  • Multimodal Tokenization:
    • PetVAE: A Variational Autoencoder with Vector Quantization (VQ-VAE) that learns a discrete latent space for dog motion. It includes a Bone Length Regressor to enforce skeletal consistency.
    • Human Tokenizer: Separately encodes body and hand gestures into discrete tokens using modality-specific encoders.
    • Translation & Audio Tokenizers: Encode global root translations and audio features (via MERT) into discrete tokens.
  • Autoregressive Transformer: A prefix-LM GPT model that predicts the next pet motion token conditioned on human and audio tokens.
  • Modality-Aware Attention (MAA): A novel attention mechanism that controls information flow. Human tokens utilize bidirectional attention (global context), while translation and pet tokens use causal (autoregressive) attention, establishing a coarse-to-fine generation hierarchy.
• Training: The model is trained in two stages: first, the VQ-VAEs are trained to tokenize modalities; second, the transformer is trained to generate pet motion tokens conditioned on the other modalities.

3. Key Contributions

1. InterPet4D Dataset: The first large-scale multimodal 4D dataset of human-pet interactions, featuring 6.8M synchronized frames, 12-view third-person video, egocentric video, audio, and 3D reconstructions for both humans and dogs.
2. Systematic Protocol: A standardized interaction taxonomy covering four categories of human-dog interaction, enabling structured analysis of cross-species behavior.
3. InterPetMoGen (IPMG): A novel framework incorporating a PetVAE tokenizer and Modality-Aware Attention (MAA) to generate diverse, realistic dog motion responses conditioned on human gestures and audio.

4. Experimental Results

The authors evaluated IPMG against baselines including Seq2Seq-Transformer, Diffusion-Transformer (DiT), and ablated variants of their own model.

• Quantitative Performance:

  • FID (Fréchet Inception Distance): IPMG achieved a kinetic FID ($FID_k$) of 11.21, substantially outperforming the Seq2Seq baseline (21.22) and the DiT baseline (64.37).
  • Alignment (R-Precision): IPMG showed superior alignment with conditioning signals, achieving $R_{hand}^{Prec.}$ of 0.63 and $R_{body}^{Prec.}$ of 0.59, compared to 0.41/0.38 for Seq2Seq.
  • Diversity: The model generated more varied motion patterns (Div$_k$ = 5.93) compared to baselines.
  • Ablation: Removing the PetVAE module increased $FID_k$ to 14.15. Removing the bone constraint in PetVAE increased $FID$ to 12.53 and velocity error to 1.79, confirming the necessity of skeletal constraints. Using both body and hand inputs yielded the best results compared to single-modality inputs.

• Qualitative & User Study:

  • Visualizations demonstrated that IPMG produced temporally smooth and responsive motions (e.g., turning toward commands), whereas baselines often failed to react to gestures or produced unstable motions.
  • A user study with 12 participants rated IPMG significantly higher than all baselines in Naturality (6.58 vs. 4.04 for Seq2Seq), Responsiveness (6.55 vs. 3.67), and Overall Quality (6.63 vs. 3.67). Statistical analysis confirmed these differences were significant.

5. Significance and Claims

The paper claims that InterPet4D and InterPetMoGen represent a foundational step toward modeling realistic cross-species interactions. By providing the first synchronized multimodal dataset and a specialized generation framework, the work addresses the critical lack of data and methods for human-pet dynamics. The authors assert that their approach, particularly the use of discrete motion tokens and modality-aware attention, effectively captures the structured relationships between human agents and animal responses, outperforming generic motion generation architectures in both objective metrics and perceptual quality.

Limitations Acknowledged:

• The dataset is currently limited to dogs; extending to other species (e.g., cats) requires species-specific models.
• The framework does not model physical contact forces, which are relevant for petting interactions.
• Current generation is limited to fixed-length clips (10 seconds); variable-length generation is a future direction.