DISPLAY: Directable Human-Object Interaction Video Generation via Sparse Motion Guidance and Multi-Task Auxiliary

The paper introduces DISPLAY, a framework for generating controllable and physically consistent human-object interaction videos by utilizing sparse motion guidance (wrist coordinates and object bounding boxes), an object-stressed attention mechanism, and a multi-task auxiliary training strategy to overcome limitations in flexibility, generalization, and data scarcity.

The Gist

Imagine you are a movie director. You have a script (a text prompt) and you want to film a scene where an actor picks up a specific prop, like a coffee mug, and takes a sip.

In the past, asking a computer to do this was like giving a vague instruction to a chaotic intern: "Make a video of a guy drinking coffee." The computer might give you a guy drinking a giant soda, or a guy floating in space, or a guy whose hand passes right through the mug like a ghost. It was hard to control exactly what happened.

Other methods tried to fix this by giving the computer a "template" video (a reference film) and asking it to copy the movements. But this was like trying to paint a new picture by only being allowed to trace over an old one. You couldn't easily swap the coffee mug for a toaster, or change the actor's movements without breaking the whole video.

Enter DISPLAY.

The paper introduces DISPLAY, a new AI system that acts like a super-smart, obedient director's assistant. It solves the problem of making videos where humans interact with objects in a way that looks real, feels physical, and follows your exact instructions.

Here is how it works, broken down into simple concepts:

1. The "Sparse Motion Guidance" (The Minimalist Sketch)

Most previous systems tried to control the video by drawing a complex skeleton for the whole body and a 3D map for the object. It was like trying to direct a play by writing a 50-page script for every single muscle twitch.

DISPLAY takes a different approach. It uses Sparse Motion Guidance.

• The Analogy: Imagine you are directing a puppet show. Instead of telling the puppeteer exactly how to move every finger, you just give them two simple instructions:
  1. Where the wrist goes: You draw a simple line on the screen showing where the actor's hand should move (start here, end there).
  2. Where the object is: You draw a simple box around where the object (like the mug) should be.
• Why it's cool: This is "shape-agnostic." It doesn't matter if the object is a round mug, a flat iPad, or a weirdly shaped banana. The AI just knows, "The hand goes to the box." It fills in the rest of the details (how the fingers curl, how the light hits the object) on its own. This gives you total freedom to swap objects without retraining the AI.

2. The "Object-Stressed Attention" (The Object's Bodyguard)

When you ask an AI to generate a video, it often gets distracted. It focuses so much on the human's face or the background that the object they are holding looks weird, melts, or disappears.

DISPLAY introduces a mechanism called Object-Stressed Attention.

• The Analogy: Imagine the AI is a student taking a test. Usually, the student ignores the difficult questions about the "object" and just guesses. Object-Stressed Attention is like a strict teacher who taps the student on the shoulder and says, "Hey, look at the object! Make sure it stays solid, looks like the reference photo, and doesn't pass through the hand."
• The Result: The AI pays extra attention to the object, ensuring it looks real and interacts physically correctly with the human hand.

3. The "Multi-Task Auxiliary Training" (The Intern's Internship)

The biggest problem with teaching AI to do Human-Object Interaction (HOI) is that there aren't enough high-quality videos of people holding specific things. It's like trying to teach a chef to make a specific rare dish, but you only have 10 examples of that dish in the whole world.

DISPLAY solves this with Multi-Task Auxiliary Training.

• The Analogy: Instead of only studying the 10 rare dishes, the AI goes to a massive culinary school where it learns to cook everything. It learns to chop vegetables, boil water, and plate food (general human motion).
• The Strategy: The system trains on a mix of:
  1. The rare, perfect videos of people holding objects (the specific task).
  2. Thousands of videos of people just moving around, without specific objects (the general task).
• The Result: By learning the general rules of movement from the huge dataset, the AI becomes much better at the specific task of holding objects, even when it hasn't seen that exact object before.

What Can You Actually Do With It?

The paper shows three main superpowers:

1. Object Replacement: You have a video of a guy holding an iPad. You want him to hold a toaster instead. You upload a picture of a toaster, and DISPLAY swaps the iPad for the toaster, making the hands grip it naturally.
2. Object Insertion: You have a video of a guy sitting at an empty table. You want him to pick up a mug that wasn't there before. You draw a box where the mug should be and a line for his hand to reach it. The AI invents the mug and the interaction from scratch.
3. Environmental Interaction: You have a video of a guy walking past a chair. You want him to stop and sit down. You draw a path for his hand to touch the chair, and the AI animates the whole sitting motion.

The Bottom Line

DISPLAY is like giving a director a magic wand. Instead of needing a full script, a 3D model, and a template video, you just need a simple sketch of where the hand goes and a picture of the object. The AI fills in the physics, the lighting, and the realistic details, allowing you to create high-quality, controllable videos of people interacting with the world around them.

Technical Summary

Here is a detailed technical summary of the paper "DISPLAY: Directable Human-Object Interaction Video Generation via Sparse Motion Guidance and Multi-Task Auxiliary".

1. Problem Statement

Human-centric video generation has advanced significantly, yet existing methods struggle to produce controllable and physically consistent Human-Object Interaction (HOI) videos. Current approaches face three main limitations:

• Over-reliance on Text: Large Video Generation Models (LVGMs) often require meticulously crafted text prompts, leading to non-deterministic outputs and a lack of precise spatial control (e.g., grasping an object at a specific location).
• Representation Imbalance: Methods using explicit guidance (e.g., 2D poses, 3D meshes) often provide dense control for human hands but lack explicit structural representations for objects. This asymmetry causes models to overfit to hand signals, resulting in geometric interpenetration, object deformation, or failure with novel objects.
• Data Scarcity: High-quality, annotated HOI video data is rare. Models trained solely on limited HOI data often suffer from poor generalization and reduced structural fidelity.
• Rigid Control: Many existing methods rely on template or driving videos, restricting the ability to generate arbitrary content or edit existing footage freely.

2. Methodology: The DISPLAY Framework

The authors propose DISPLAY, a framework designed to generate high-fidelity HOI videos using Sparse Motion Guidance and Multi-Task Auxiliary Training.

A. Sparse Motion Guidance

Instead of dense control signals (like full hand meshes or object depth maps), DISPLAY uses a lightweight, user-friendly input consisting of:

1. Wrist Joint Coordinates: Only the 2D coordinates of the left and right wrists are used to guide hand trajectories. This aligns with the "end-effector" concept, focusing on the interaction point without over-constraining the hand structure.
2. Shape-Agnostic Object Bounding Box: A bounding box representing the object's location and size, which is independent of the object's specific shape.

• Advantage: This alleviates the imbalance between human and object representations, prevents overfitting to specific object shapes, and allows for intuitive user control (e.g., simple clicking on keyframes).

B. Architecture & Object-Stressed Attention

The framework is built upon a pre-trained Flow Matching DiT (Diffusion Transformer) model (specifically Wan2.1-14B).

• ControlNet-style Design: A frozen pre-trained T2V model is paired with a trainable Condition Branch. This branch clones a subset of transformer layers from the backbone to inject multi-modal conditions (text, visual reference, object reference, motion, background) via residual injection.
• Object-Stressed Attention (OSA): To address the challenge of generating realistic interactions with sparse guidance, the authors introduce a modified attention mechanism. It applies a hyperparameter $\alpha$ to weight the attention scores between object tokens and other tokens (hands, scene).
  • Formula: The attention matrix is scaled such that object-object interactions are emphasized ($\alpha^2$) and object-other interactions are boosted ($\alpha$), ensuring the generated object remains physically consistent with the scene and hand pose.

C. Multi-Task Auxiliary Training

To overcome the scarcity of high-quality HOI data, the authors propose a Multi-Task Auxiliary Training strategy with a curated data pipeline:

• Data Curation: A three-stage filtering process (Score-based, Human-centric, and VLM-based) extracts high-quality rigid-object interaction clips from web videos.
• Mixed Training Paradigm: The model is trained on both:
  1. HOI-annotated data: Videos with explicit object and wrist annotations.
  2. Weakly annotated data: General human-centric videos without specific object interactions.
• Masking Strategies:
  • Human-Body Masking: Masks the body (or entire frame) to force the model to learn plausible motion reconstruction.
  • Multi-Task Training Mask: Randomly drops parts of the motion or masked sequences during training (Bernoulli sampling). This enables the model to learn from incomplete inputs, supporting tasks like image-to-video generation and video in-betweening during inference.

3. Key Contributions

1. Sparse Motion Guidance Framework: A novel approach enabling arbitrary, high-fidelity HOI generation using only wrist coordinates and object bounding boxes, offering superior flexibility compared to dense control signals.
2. Object-Stressed Attention: A mechanism that enhances object robustness and physical consistency under sparse conditions by explicitly weighting object-related attention tokens.
3. Multi-Task Auxiliary Training: A comprehensive training strategy and data curation pipeline that leverages both high-quality HOI data and general human motion data to overcome data scarcity and improve generalization.
4. Versatile Generation Capabilities: The framework supports Object Replacement (swapping objects), Object Insertion (adding new objects), and Environmental Interaction (defining interactions with existing scene objects) via a user-friendly motion-authoring interface.

4. Experimental Results

The method was evaluated on a self-filmed test set and in-the-wild videos, comparing against state-of-the-art models like VACE, HunyuanCustom, HuMo, and Re-HOLD.

• Quantitative Performance:
  • Appearance Quality: Achieved the best FID (67.5) and Aesthetics (0.547) scores, indicating superior visual fidelity.
  • Temporal Consistency: Achieved the best FVD (560.29), demonstrating smooth and coherent motion.
  • Object Fidelity: Significantly outperformed baselines in O-CLIP (0.890) and O-DINO (0.832), proving the effectiveness of the Object-Stressed Attention in preserving object identity.
  • Contact Agreement (CA): Achieved the highest score (0.891), indicating superior physical consistency between hands and objects.
• Qualitative Results:
  • DISPLAY successfully generates realistic interactions with novel objects (e.g., inserting a mug or iPad) without the deformation or texture loss seen in other methods.
  • It supports long-video manipulation (up to 1 minute) without noticeable error accumulation.
  • Ablation studies confirmed that removing OSA, Multi-Task Training, or Visual References leads to significant drops in object quality and motion realism.

5. Significance

The DISPLAY framework represents a significant leap forward in controllable video generation. By decoupling the need for complex, dense control signals (like 3D meshes) and relying on sparse, intuitive user inputs, it democratizes the creation of complex HOI scenarios. The integration of Object-Stressed Attention solves a critical bottleneck in physical consistency, while the Multi-Task Auxiliary Training strategy offers a scalable solution to the data scarcity problem. This work has broad implications for applications in e-commerce (virtual product demonstrations), entertainment (digital human interactions), and education, enabling the generation of high-quality, user-directed interactive content with unprecedented ease and realism.