Pulp Motion: Framing-aware multimodal camera and human motion generation

1. Problem Statement

The paper addresses the challenge of jointly generating human motion and camera trajectories based on a text prompt.

• The Core Issue: Existing approaches typically treat human motion generation and camera trajectory generation as separate tasks. However, in cinematography, these are intrinsically linked; the camera must frame the actor, and the actor often adjusts their performance based on the camera's presence. Treating them separately leads to multimodal incoherence, resulting in poor framing (e.g., the subject moving out of the frame) or unnatural camera movements that fail to capture the intended performance.
• The Gap: Current multimodal generation methods often rely solely on paired data (which may have limited mode coverage) or require complex architectural changes and external pre-trained models (like ImageBind) to enforce correlations. The authors argue for a more general, model-agnostic approach that explicitly bridges the modalities without heavy architectural overhead.

2. Methodology

The authors propose a unified framework called Pulp Motion, which introduces an auxiliary modality (on-screen framing) to enforce coherence between human motion ($x$) and camera trajectories ($y$).

A. The Auxiliary Modality: On-Screen Framing

Instead of generating $x$ and $y$ directly, the model leverages an intermediate representation $z$: the 2D projection of human joints onto the camera view (Normalized Device Coordinates). This "framing" latent naturally captures the geometric relationship between the actor and the camera.

B. Multimodal Latent Space & Autoencoder

1. Joint Encoder: A joint encoder $E_\phi$ maps raw human motion ($x_{raw}$) and camera trajectory ($y_{raw}$) into a shared latent space, producing latent vectors $x$ and $y$.
2. Linear Transform: A lightweight, learnable linear transform $W$ maps the concatenated latents $[x, y]^\top$ into the auxiliary framing latent $z$:
   $$z = W [x, y]^\top$$
   Crucially, $z$ is not directly encoded from data; it is learned exclusively through this linear projection and supervised via a reconstruction loss.
3. Decoders: Three independent decoders reconstruct the raw modalities ($\hat{x}_{raw}, \hat{y}_{raw}, \hat{z}_{raw}$) from their respective latents.

C. Auxiliary Sampling (Inference Strategy)

The core innovation is a sampling technique that steers the generation process toward coherent framing without requiring $z$ during training.

• Mathematical Basis: The authors decompose the joint latent space $u = [x, y]^\top$ into two orthogonal components:
  • $u_{//}$: The component parallel to the auxiliary modality $z$ (captured by the projection $P_{//}$).
  • $u_{\perp}$: The component orthogonal to $z$ (acting as the "unconditional" term).
• Guidance Mechanism: Using the properties of Gaussian distributions and the Moore-Penrose pseudo-inverse, they derive a sampling update rule. The noise prediction $\epsilon_\theta$ is adjusted by an auxiliary guidance term:
  $$\epsilon_\theta(x_t, y_t, c, t) = \epsilon_\theta(x_t, y_t, \emptyset, t) + w_z P_{//} \epsilon_\theta(x_t, y_t, \emptyset, t) + w_c(\epsilon_\theta(x_t, y_t, c, t) - \epsilon_\theta(x_t, y_t, \emptyset, t))$$
  Where $w_z$ is the auxiliary guidance weight. This term pushes the generated $x$ and $y$ toward a region where their projection $z$ is consistent, ensuring the camera frames the human correctly.

3. Key Contributions

1. Unified Framework: A model-agnostic framework for joint human-camera generation that enforces multimodal coherence via an auxiliary "on-screen framing" modality.
2. Auxiliary Sampling: A novel inference strategy that uses a linear transform in the latent space to steer sampling toward coherent joint distributions, avoiding the need for complex architectural changes or external pre-trained models.
3. PulpMotion Dataset: A new, large-scale dataset containing 193K samples (314 hours) of paired human motions and camera trajectories with rich textual captions. It improves upon previous datasets (like E.T.) by offering higher-quality motion (via refinement) and comprehensive text descriptions for both modalities.
4. Extensive Evaluation: Validation across two distinct architectures (DiT and MAR) demonstrating the method's generality.

4. Results

The method was evaluated on the PulpMotion dataset using both DiT (Diffusion Transformer) and MAR (Masked Autoregressive) architectures.

• Framing Quality: The proposed method significantly reduces the Out-of-Frame rate (instances where the subject leaves the view) and improves the Fréchet Distance of framing ($FD_{framing}$).
  • Example (DiT): Out-of-frame rate dropped from 25.98% (joint generation without Aux) to 16.76% with auxiliary sampling.
  • Example (MAR): Out-of-frame rate dropped from 40.75% to 33.65%.
• Text Alignment: The method improves text-to-motion and text-to-camera alignment (measured by TMR-Score and CLaTr-Score) while maintaining high generation quality.
• Ablation Studies:
  • Guidance Weight ($w_z$): Moderate values of $w_z$ yield the best trade-off between framing quality and sample diversity. High weights improve framing but degrade motion fidelity.
  • Architecture Agnosticism: The auxiliary sampling technique consistently outperforms baselines (independent generation, dual-modality generation, and human-conditioned camera generation) across both DiT and MAR backbones.
• Qualitative Results: Visualizations show that the method generates cinematic sequences where the camera smoothly tracks the actor, maintaining the subject in the frame and adhering to the prompt (e.g., "camera trucks right" while "person walks").

5. Significance

• Cinematic Realism: This work bridges the gap between computer vision and cinematography by explicitly modeling the "actor-camera" relationship, a core principle often ignored in generative AI.
• Efficiency: Unlike previous multimodal approaches that require heavy architectural modifications or massive external pre-trained models, Pulp Motion achieves state-of-the-art results with a simple, model-agnostic auxiliary sampling strategy.
• Dataset Contribution: The release of PulpMotion provides the community with a high-quality, large-scale resource for training and evaluating joint human-camera generation, addressing the scarcity of such paired data.
• Future Direction: The paper sets a new state-of-the-art for text-conditioned joint generation, suggesting that auxiliary modalities can be a powerful tool for enforcing coherence in complex multimodal tasks beyond just human-camera interactions.