Technical Summary: HOMIE – Human-object Centric Video Personalization via Multimodal Intelligent Enhancement

1. Problem Statement

Human-object centric video personalization (HOCVP) is a critical task in subject-driven video generation, aiming to synthesize videos featuring diverse interactions between humans and objects based on reference images. Despite recent advancements, existing methods face two primary limitations:

1. Inter-Subject Challenges: Current approaches struggle to balance high subject fidelity with accurate interaction patterns, particularly when dealing with multiple subjects or abstract concepts (e.g., logos). They often produce copy-paste artifacts, exhibit restricted expressiveness, and lack the inferential reasoning required to correctly place abstract references within a scene without explicit prompt instructions.
2. Intra-Subject Challenges: While intra-subject references (such as multi-view images or OCR maps) are intended to enhance fidelity and consistency, existing works lack mechanisms to understand the latent correspondence between these multiple references. Consequently, models often fail to improve multi-view consistency or text fidelity, sometimes treating distinct views of the same object as separate entities.

The core difficulty lies in integrating Multimodal Large Language Models (MLLMs) effectively. Prior strategies either align MLLM features with text embedding spaces (diminishing controllability and bottlenecking feature space) or replace text encoders entirely with MLLMs (incurring high re-alignment costs and distracting the model from its primary control tasks).

2. Methodology

The authors propose HOMIE, a unified framework designed to handle both inter- and intra-subject HOCVP scenarios by leveraging MLLM knowledge without compromising the controllability of the underlying video diffusion model. The architecture is built upon a DiT-based text-to-video backbone (specifically Wan2.1-14B and Wan2.2-14B) and introduces three key innovations:

2.1 Multimodal Input Paradigm

HOMIE adopts a unified input stream comprising:

• Video Tokens ($F_v$): Latent representations of the target video.
• Reference Image Tokens ($F_r$): Encoded reference images.
• Multimodal Features ($F_M$): High-semantic features extracted by an MLLM (Qwen3-VL-2B-Thinking) from the combination of system prompts, user prompts, and reference images.

Unlike methods that concatenate MLLM features directly into text embeddings, HOMIE concatenates $F_M$ directly with the latent video and reference tokens ($F = [F_v \oplus F_r \oplus F_M]$) to facilitate intrinsic cross-modal interaction.

2.2 Global Multimodal Guidance (GMG)

To address the distributional gap between MLLM features and 3D VAE tokens, and to ensure stable temporal control, HOMIE introduces Global Multimodal Guidance.

• Mechanism: During the self-attention query-key computation, the MLLM query ($Q_M$) and key ($K_M$) matrices are pooled along the temporal dimension to create global representations ($\tilde{Q}_M, \tilde{K}_M$).
• Modulation: These global representations are passed through lightweight projection networks ($\gamma, \beta$) to derive scaling and shifting factors. These factors are applied via an affine transformation to the video token queries ($Q_v$) and keys ($K_v$).
• Effect: This injects high-level semantic guidance into the video generation process, enhancing semantic reasoning and interaction modeling without altering the native self-attention patterns of the reference images.

2.3 Modality-Reference Embedding (MRE)

To distinguish between different input modalities and manage the complex relationships between multiple references, HOMIE proposes Modality-Reference Embedding.

• Modality Embeddings (ME): Learnable vectors are added to tokens based on their source (video, reference image, or MLLM features) to help the model disentangle heterogeneous inputs.
• Reference Embeddings (RE): Learnable vectors are assigned to reference tokens based on their identity. Crucially, multiple images representing the same subject (intra-subject) share the same reference embedding, while distinct subjects (inter-subject) receive unique embeddings.
• Effect: This allows the model to bind intra-subject references (improving consistency for multi-view or OCR inputs) while maintaining distinct identities for inter-subject interactions.

2.4 Training Strategy

The model is trained using a flow matching paradigm on a curated dataset derived from OpenS2V-5M and PhantomData, augmented with a self-curated high-quality subset of 20K HOC video clips. The training employs a three-stage tuning strategy over approximately 11K GPU-hours.

3. Key Contributions

The paper claims three primary contributions:

1. Unified Framework: The proposal of HOMIE, a framework that unifies inter- and intra-subject HOCVP tasks, enhanced by MLLM knowledge.
2. Global Multimodal Guidance (GMG): A novel mechanism that injects multimodal features into video tokens via self-attention modulation, improving semantic reasoning and interaction modeling without costly re-alignment.
3. Modality-Reference Embedding (MRE): A learnable module that distinguishes token modalities and captures intra-subject associations, significantly improving identity fidelity in complex reference scenarios.

4. Experimental Results

Extensive evaluations were conducted on a self-curated dataset of 200 human-object combinations, covering diverse inter- and intra-subject settings.

• Quantitative Performance: HOMIE achieves state-of-the-art (SOTA) performance among open-source methods on video quality (Aesthetic Score, Motion Smoothness) and text following (GMEScore). Notably, it demonstrates a 21.8% relative improvement in OCR accuracy compared to the previous best (SkyReels-V3) and achieves the highest Object Similarity (Obj-Sim) and Face Similarity scores.
• Intra-Subject Consistency: In multi-view scenarios, HOMIE outperforms baselines in DINO-based metrics (DINOrec, DINOacc), demonstrating superior ability to synthesize diverse views of the same object while maintaining fidelity.
• Qualitative Analysis: Visual comparisons show HOMIE successfully handles complex 4-subject interactions and abstract logo placements (e.g., attaching a "COSTA" logo to a cup) where other models fail to reason the relationship or produce degraded subjects. In intra-subject tasks, HOMIE correctly interprets OCR maps and multi-view inputs, whereas baselines often treat them as distinct, unrelated subjects.
• User Study: In a study with 40 participants, HOMIE was preferred over baselines (Kling, Phantom, SkyReels-V3, UniVideo) across Subject Consistency, Text Following, and Overall Quality metrics.

5. Significance and Claims

The paper positions HOMIE as a significant step forward in controllable video generation by addressing the specific limitations of current MLLM integration strategies. The authors claim that by preserving the text encoder and introducing targeted guidance (GMG) and embedding mechanisms (MRE), HOMIE successfully extracts reference-level relationships without compromising the controllability of text encoders or incurring the high costs of full model re-alignment.

The framework is presented as a robust solution for real-world deployment scenarios requiring high-fidelity human-object interactions, particularly in cases involving abstract concepts (logos) and the need for consistent multi-view or text-enhanced object representations. The work validates that a unified approach to inter- and intra-subject personalization is feasible and superior to treating these components in isolation.