💻 computer science

LooseControlVideo: Directorial Video Control using Spatial Blocking

LooseControlVideo is a novel framework that enables intuitive and precise 3D spatial control in text-to-video generation by using sparse, oriented 3D boxes as a "blocking" proxy, which significantly improves trajectory accuracy, motion consistency, and occlusion handling compared to existing 2D-based methods.

Original authors: Shariq Farooq Bhat, Niloy J. Mitra, Kalyan Sunkavalli

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shariq Farooq Bhat, Niloy J. Mitra, Kalyan Sunkavalli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a movie director trying to film a complex scene: an eagle swooping down to catch a rabbit, or a horse jumping over a fence. In the real world, you wouldn't ask your actors to calculate the exact angle of every wingbeat or the precise physics of every muscle contraction. Instead, you would use "blocking." You'd tell the actors, "Stand here, walk there, jump at this moment," using simple, rough movements to set the stage. The actors (and the camera crew) then fill in the realistic details: the flapping wings, the wind in the fur, the shadows.

LooseControlVideo (LCV) is a new AI tool that brings this "director's blocking" concept to computer-generated videos. It allows users to choreograph complex, multi-object videos using simple 3D boxes, while the AI handles the difficult work of making everything look realistic.

Here is a breakdown of how it works, using everyday analogies:

1. The Problem: The "Too Much, Too Hard" Dilemma

Current AI video generators are great at making things look real, but they are terrible at following specific instructions about where things move.

Text is too vague: If you type "an eagle catches a rabbit," the AI might make the eagle fly in the wrong direction or miss the rabbit entirely.
Detailed maps are too hard: If you try to draw a precise 3D map for every single frame (like a depth map), it's like asking a director to draw every single feather on the eagle's wing before the movie starts. It's too much work and impossible for complex scenes.

2. The Solution: The "3D Proxy" (The Rough Draft)

LCV solves this by letting you use sparse, oriented 3D boxes.

The Analogy: Think of these boxes as "stand-ins" or "dummy actors" in a rehearsal. You don't need to dress them up or give them faces. You just need to tell them: "This box (the eagle) starts here, rotates like this, and moves to that spot."
The Magic: You provide the high-level choreography (the path, the timing, the general shape), and the AI fills in the low-level details (the feathers, the muscle flexing, the realistic shadows).

3. The Secret Sauce: DNOCS (The "Color-Coded Map")

The biggest challenge is that a simple 3D box doesn't tell the AI how deep the object is or how it's rotated relative to the camera. To fix this, the researchers invented a special way to color these boxes called DNOCS.

How it works: Imagine painting the 3D boxes with a special color code.
- Hue (Color): Tells the AI which way the object is facing (like a compass).
- Brightness: Tells the AI how far away the object is (brighter = closer, darker = farther).
The Result: The AI sees a colorful, glowing map that instantly understands the 3D layout, depth, and orientation without needing to guess. It's like giving the AI a "cheat sheet" that translates your rough 3D boxes into a language the video generator understands perfectly.

4. What Can It Do?

The paper demonstrates that this method is powerful for two main tasks:

Creating New Videos: You can draw a rough path for a car weaving through traffic or a dog jumping, and the AI generates a photorealistic video where the car drifts realistically and the dog's fur moves naturally.
Editing Existing Videos: You can take an existing video (like a horse jumping) and use these 3D boxes to change the horse's path. If you move the box to make the horse jump higher, the AI adjusts the horse's body, the shadows, and the background to make the new jump look real.

5. The Results: Better Than the Competition

The researchers tested this against other methods that use 2D drawings or flow maps.

The Analogy: Imagine trying to navigate a city. Other methods give you a flat 2D paper map (which is confusing when you need to know which building is in front of the other). LCV gives you a 3D model with clear depth markers.
The Outcome: LCV was significantly better at:
- Trajectory: Objects stayed on the path you drew (1.2 to 3 times more accurate).
- Occlusion: Objects correctly blocked each other (e.g., a tree correctly hiding a car behind it) 1.5 to 2 times better than before.
- Motion: The movement felt more rigid and consistent, like real physics, rather than "wobbly."

Summary

LooseControlVideo is like giving a director a set of simple 3D blocks to arrange a scene. The director decides the story and the movement, and the AI acts as the special effects team, filling in the realistic details, shadows, and physics. It bridges the gap between "I have a creative idea" and "I have a professional-looking video," without requiring the user to be a 3D modeling expert.

Technical Summary: LooseControlVideo

Problem Statement

Precise 3D spatial orchestration in text-to-video generation remains a significant challenge, particularly for multi-object scenes where semantic layout and temporal dynamics are entangled. Existing control modalities present a difficult trade-off: natural language is too imprecise for exact spatial trajectories, while dense video signals (e.g., per-frame depth or edge maps) are nearly impossible for users to manually author for dynamic scenes. Current depth-conditioned models achieve structural fidelity but require labor-intensive, frame-accurate guidance. Furthermore, existing structure guidance conflates two distinct control axes: (i) the layout, motion, and interactions of the camera and objects, and (ii) the fine-grained object poses and deformations arising from those interactions. This makes it impractical for users to provide the detailed geometric data (e.g., capturing every wing-beat of an eagle) required to drive complex physical events.

Methodology

The authors propose LooseControlVideo (LCV), a framework that enables intuitive and expressive control using sparse, oriented 3D boxes as a "blocking" proxy. This approach decouples high-level choreography (layout, motion path, timing) from secondary dynamics and realistic deformations, which are inferred by the generative model.

1. DNOCS Representation

To bridge the gap between coarse 3D proxies and high-fidelity synthesis, the authors introduce Depth-modulated Normalized Object Coordinate Space (DNOCS). This is a novel encoding scheme that jointly encodes local orientation and global depth into a single 3-channel image signal.

Orientation: Inspired by NOCS maps, each 3D box is colored using its local normalized coordinate frame, allowing the model to infer object orientation and relative pose.
Depth Modulation: The color intensity is modulated by the object's depth relative to the camera, providing a global distance cue that standard NOCS lacks.
Rendering: The 3D boxes are rendered from the target camera's perspective into a 2D control signal ( $v_{ctrl}$ ). This pre-computed rendering exposes depth ordering, occlusion, and perspective cues explicitly, aligning the control representation with the 2D image domain of the base video model.

2. Architecture and Training

Backbone: The method fine-tunes a Wan 2.2 backbone (specifically the VACE architecture, a ControlNet-style adaptation of Diffusion Transformers).
Conditioning: The rendered DNOCS control signal ( $v_{ctrl}$ ) is injected into the backbone DiT model via the standard VACE conditioning pathway. The base model remains frozen; only the VACE modules are LoRA fine-tuned (rank 64).
Data Pipeline: To avoid manual 3D annotation, the authors developed an automated pipeline to extract temporally tracked 3D oriented boxes from large-scale in-the-wild video datasets (nuScenes, HO-3D, BEHAVE). This pipeline uses GroundingDINO, SAM, monocular depth prediction, and 3D Kalman filtering to create a training corpus of ~10k aligned RGB-DNOCS video pairs.
Versatility: The system supports both video generation (using only DNOCS renders) and video editing (compositing DNOCS renders with input video frames, allowing users to specify regions to keep, remove, or fill).

Key Contributions

3D-Aware Spatial Blocking Paradigm: A method using oriented 3D boxes to decouple choreography from local deformation, enabling the authoring of complex, multi-object events without dense 3D rigging.
Specialized Control Mechanism (DNOCS): A novel encoding for Diffusion Transformers that provides intuitive "loose" control by rendering 3D primitives into a 2D signal that explicitly conveys orientation, depth, and occlusion.
Automated Training Pipeline: A system for lifting 2D video datasets into 3D-conditioned training sets, eliminating the need for manual 3D annotation.
Evaluation Metrics: Novel metrics for evaluating spatial blocking adherence, including Trajectory Error, Occlusion Accuracy, and Rigid Motion Consistency.

Results

Extensive evaluations on nuScenes (navigation), HO-3D (hand-object interactions), and BEHAVE (full-body human interactions) demonstrate that LCV significantly outperforms existing 2D-box and flow-based baselines.

Trajectory Error: Achieves a 1.2–3x improvement over state-of-the-art layout-conditioned models.
Rigid Motion Consistency: Shows a 2x improvement.
Occlusion Accuracy: Increases by 1.5–2x.
Qualitative Performance: The model successfully generates realistic occlusions, temporal synchronization, and secondary dynamics (e.g., tire skids, morphological deformations) based on sparse 3D inputs. It maintains visual identity and scene consistency during editing tasks, such as modifying a horse's jump trajectory or a vehicle's path.

Significance and Claims

The paper claims that oriented 3D primitives provide a robust geometric prior for complex, multi-agent video authoring. By adopting the "blocking" metaphor from professional cinematography, LCV bridges the gap between casual authoring and cinematic precision. The authors argue that their approach effectively offloads the burden of geometric modeling and complex object deformations to the diffusion process, allowing users to focus on high-level intent.

The work positions itself as a step toward an intuitive, "director-in-the-loop" paradigm for video generation, where complex physical interactions can be authored with ease without sacrificing photorealistic fidelity or structural rigor. The authors acknowledge limitations, specifically that they do not explicitly bind visual identities to specific 3D boxes (a challenge in residual architectures) and that users must still author the temporal rhythm of interactions, though they suggest future work could allow the model to infer timing from sparse spatial trajectories.