HECTOR: Hybrid Editable Compositional Object References for Video Generation

HECTOR is a novel video generation pipeline that enables fine-grained compositional control by supporting hybrid reference conditioning from static images and dynamic videos, while allowing users to explicitly specify the trajectories, locations, scales, and speeds of individual objects to synthesize coherent, high-fidelity videos.

The Gist

Imagine you are a movie director. In the past, if you wanted to make a video using AI, you had to give it a vague instruction like, "Make a video of a dog running in a park." The AI would then magically conjure up a scene from scratch. But here's the problem: you couldn't tell the AI exactly which dog to use, exactly where it should run, or exactly how fast it should go. If you wanted to swap the dog for a cat halfway through, or zoom the camera in while the dog jumps, the AI would just get confused and make a mess.

HECTOR is like giving that director a super-powerful, magical editing suite where they can control every single character and object in the scene with surgical precision.

Here is a simple breakdown of how it works, using some everyday analogies:

1. The Problem: The "All-or-Nothing" Approach

Current AI video generators are like a painter who sees the whole canvas at once. If you ask them to paint a specific person, they might get the face right, but they might accidentally move the person's arm or change their clothes because they are painting the whole picture as one big blob. They lack "compositional" control—they can't easily say, "Keep the background exactly the same, but move this specific person here."

2. The Solution: HECTOR (The "Lego Master")

HECTOR treats a video not as one big blob, but as a collection of Lego bricks. It allows you to pick up individual bricks (objects, people, backgrounds) and move them around independently without breaking the whole structure.

It does this through three main "magic tricks":

A. The Video Decompositor (The "Smart Scanner")

Before the AI can edit a video, it needs to understand what's in it.

• Old Way: You draw a rough box around a person, and the AI guesses where they are.
• HECTOR Way: It uses a "Smart Scanner" (called the Video Decompositor). Imagine a super-precise robot that watches a video and places tiny, invisible dots on a person's shoulders, elbows, and knees. It tracks every single dot as the person moves.
• Why it matters: Instead of a shaky, rough box, HECTOR knows the exact path, size, and speed of the object. It's the difference between guessing a car's speed by looking at it and having a GPS tracker on the car.

B. The Spatio-Temporal Alignment Module (The "Conductor")

Now that HECTOR knows exactly where everything is, it needs to mix different ingredients together.

• The Mix: You might want to use a static photo of a dog (for its look) and a video of a cat jumping (for its movement).
• The Magic: HECTOR has a "Conductor" (STAM) that takes these different sources and lines them up perfectly in time and space. It's like a DJ mixing two different songs so they play in perfect sync. It takes the "look" from the photo and the "dance moves" from the video and blends them so the dog looks like the photo but moves exactly like the cat in the video.

C. The "Ghost-Free" Glue (Gaussian Masks)

When you paste a new object into a video, the edges often look jagged or "ghostly" (like a bad Photoshop job).

• HECTOR's Trick: Instead of using a hard, sharp line to cut out an object, HECTOR uses a "soft, fuzzy edge" (Gaussian masking). Imagine using a soft brush to blend the new object into the background rather than a pair of scissors. This makes the new object look like it was always part of the scene, with perfect lighting and shadows.

3. What Can You Actually Do With It?

Because HECTOR treats every object as a separate, controllable piece, you can do things that were previously impossible:

• The "Swap" Trick: You can take a video of a person walking and instantly replace them with a different person (or a robot) while keeping the exact same walking style and background.
• The "Zoom" Trick: You can tell the AI, "Zoom the camera in on the dog, but keep the background still," or "Make the dog run faster without changing the background."
• The "Add-On" Trick: You can drop a flying eagle into a video of a girl dancing, and the eagle will fly over her perfectly, respecting her position and the lighting, without messing up the girl's dance.

The Bottom Line

Think of HECTOR as the difference between clay modeling and Lego building.

• Old AI: You squish a lump of clay (the video) and hope it looks right. If you want to change the nose, you might ruin the ears.
• HECTOR: You build with Lego bricks. You can take the "dog" brick out, swap it for a "cat" brick, move the "background" brick, and zoom the "camera" brick, and the whole scene stays perfect.

It turns video generation from a "magic trick" where you hope for the best, into a precise "construction project" where you are the master architect.

Technical Summary

Here is a detailed technical summary of the paper "HECTOR: Hybrid Editable Compositional Object References for Video Generation."

1. Problem Statement

Current video generation models (Text-to-Video and Image-to-Video) typically synthesize scenes holistically. While they produce high-fidelity dynamic content, they lack mechanisms for explicit compositional manipulation. Users cannot easily dictate the specific appearance, location, scale, or motion trajectory of individual objects within a scene.

Existing attempts at fine-grained control face significant limitations:

• Instance-level customization (e.g., DreamVideo, MotionBooth): Often rely on computationally expensive test-time optimization and struggle to scale to complex scenes with multiple interacting objects.
• Control signal integration (e.g., Tora2, VACE): While more efficient, these models often fail to maintain precise boundaries or identity consistency when multiple entities are present. They also lack support for independent background conditioning and dynamic video references (i.e., they cannot easily use a video to dictate specific gestures while preserving identity).

The Core Challenge: How to generate videos that allow for fully compositional control over distinct visual elements (background, multiple objects) using hybrid references (static images and dynamic videos) while maintaining high fidelity and precise spatiotemporal alignment.

---

2. Methodology

The authors propose HECTOR, a generative pipeline comprising two primary systems: the Video Decompositor (for data processing and inference) and the HECTOR Generative Model (based on a Diffusion Transformer).

A. Video Decompositor

This module serves as both a curation engine for training data and a processing suite for inference. It moves beyond rigid bounding boxes to extract precise compositional structures:

1. Segmentation & Anchoring: Uses SAM2 to segment objects. Instead of a single bounding box, it samples multiple anchor points (centroids of patches) based on object size and aspect ratio.
2. Trajectory Extraction: Uses Cotracker3 to propagate these anchor points across time.
3. Point-to-Scale Formulation: Calculates the object's scale ($s_t$) dynamically by measuring the expansion/contraction of the tracked point cluster relative to a reference frame. This provides a smoother, more stable motion prior than traditional bounding box heuristics.
4. Visibility Indicators: Aggregates tracker confidence scores to determine when an object enters, exits, or is occluded.

B. HECTOR Generative Model

Built upon a pre-trained Diffusion Transformer (DiT) backbone (specifically Wan2.1), the model introduces a novel Spatio-Temporal Alignment Module (STAM).

• Hybrid Reference Conditioning: STAM accepts heterogeneous inputs:
  • Static Images: For identity preservation.
  • Dynamic Videos: For gesture and motion priors.
• Latent Space Alignment:
  • References are encoded into VAE latents.
  • Inverse Warping: Features are "placed" into the latent canvas using a sampling grid derived from the tracked anchor points ($p_t$) and scales ($s_t$).
  • Gaussian Softening: A Gaussian visibility mask is applied to blend features smoothly, preventing hard edges and ghosting.
• Multi-Channel Masking: The conditioning tensor includes a 4-channel mask ($M$) that explicitly tells the DiT which features come from static images, dynamic videos, or their union. This allows the model to distinguish between appearance constraints and motion priors.
• Foreground-Background Gating: During inference, if trajectories intersect, a gating mechanism prioritizes specific modalities (e.g., forcing a static object to remain in the foreground) to prevent feature bleeding.

C. Training Objective

The model is trained using a flow-matching objective with velocity prediction. It minimizes the error between the predicted flow velocity and the ground truth, learning to straighten the generative trajectory from noise to data while adhering to the provided structural priors ($z_{cond}$) and masks ($M$).

---

3. Key Contributions

1. First Fully Compositional Framework: HECTOR is the first framework to enable precise, independent control over every element in a video (background, multiple objects) using hybrid references.
2. Spatio-Temporal Alignment Module (STAM): A novel architecture that processes static and dynamic references simultaneously in the latent space, fusing identity (from images) and motion (from videos) into a unified conditioning tensor.
3. Video Decompositor: A mechanism that automatically extracts compositional structures (point-based tracks and dynamic scales) from video data, replacing rigid bounding boxes with more robust, jitter-free motion priors.
4. Advanced Editing Capabilities: The framework supports complex operations such as object insertion, replacement, background locking, and dynamic object entry/exit without disrupting global temporal flow.

---

4. Experimental Results

The authors evaluated HECTOR on an internal dataset of 2.4 million clips and the DAVIS benchmark, comparing against baselines like MotionBooth, VACE, and VACE-bbox.

• Quantitative Performance:
  • Subject Fidelity: HECTOR significantly outperforms baselines in identity preservation metrics (R-DINO, DINO-I), particularly in multi-object scenarios.
  • Motion Control: It achieves nearly double the accuracy of competitors in Mean Intersection over Union (mIoU) and Centroid Distance (CD), demonstrating superior adherence to user-defined trajectories.
  • Consistency: Maintains high temporal consistency (T-Cons) and semantic alignment (CLIP-T).
• Qualitative Performance:
  • Multi-Object Scenes: Unlike baselines which suffer from identity drift or spatial confusion in complex scenes, HECTOR maintains distinct boundaries and identities for multiple moving objects.
  • Video References: Successfully integrates dynamic video references to control specific gestures (e.g., a person dancing) while preserving the subject's identity.
  • Editing: Demonstrates high-fidelity object replacement and "Background-Locked Motion Editing," where the foreground is manipulated while the background remains frozen.

Ablation Studies:

• Replacing point-based tracking with bounding boxes degraded motion control and fidelity.
• Training on hybrid (image + video) data was crucial for superior motion adherence compared to image-only training.
• Using Gaussian-softened masks was essential for blending features; binary masks led to identity loss.

---

5. Significance

HECTOR represents a paradigm shift from holistic generation to compositional generation in video synthesis. By decoupling identity from motion and allowing users to explicitly control the trajectory, scale, and speed of individual elements via hybrid references, it bridges the gap between generative AI and professional video editing workflows.

This work enables:

• Precise Creative Control: Users can construct complex scenes by assembling specific assets with defined behaviors.
• Efficient Editing: High-fidelity object replacement and insertion without re-rendering the entire scene.
• Scalability: The point-based decomposition approach scales better to complex, multi-entity interactions than previous test-time optimization methods.

Ultimately, HECTOR provides a robust foundation for the next generation of controllable video generation tools, making them viable for professional content creation, animation, and creative expression.