💻 computer science

LAMP: Language-Assisted Motion Planning for Controllable Video Generation

LAMP is a novel framework that leverages large language models to translate natural language descriptions into structured, executable 3D motion programs for both dynamic objects and cameras, thereby enabling precise and cinematic motion control in video generation.

Original authors: Muhammed Burak Kizil, Enes Sanli, Niloy J. Mitra, Erkut Erdem, Aykut Erdem, Duygu Ceylan

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Muhammed Burak Kizil, Enes Sanli, Niloy J. Mitra, Erkut Erdem, Aykut Erdem, Duygu Ceylan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to make a movie, but you don't have a camera crew, a director, or even a scriptwriter. You just have a computer that can generate videos, but it's a bit stubborn: if you tell it "make a cool video," it might just show a static picture or a chaotic mess. It doesn't understand how the camera should move or how the actors should run.

LAMP is like hiring a super-smart, virtual film director who speaks your language.

Here is how it works, broken down into simple analogies:

1. The Problem: The "Magic Wand" vs. The "Blueprint"

Current video AI is like a magic wand. You wave it (type a text prompt), and it tries to guess what you want. But if you want a specific camera move—like "the camera should slowly circle the ship while the ship sails right"—the AI often gets confused. It might spin the wrong way, or the ship might teleport.

Trying to tell the AI exactly where to move the camera using numbers (coordinates) is like trying to teach someone to dance by giving them a spreadsheet of X and Y coordinates. It's too hard for a human to visualize and too rigid for the AI to interpret naturally.

2. The Solution: LAMP (The Virtual Director)

LAMP solves this by acting as a translator between your English words and the AI's math.

You speak: "A large ship sails right, and the camera orbits it slowly."
LAMP thinks: Instead of guessing random numbers, LAMP acts like a human director. It translates your sentence into a structured script (called a DSL or Domain-Specific Language).
The Script: It writes something like: Ship: Move Right + Camera: Orbit_90_degrees.

Think of this script as a choreography sheet. Just like a dance instructor writes down "Step left, spin, jump," LAMP writes down the exact moves for the ship and the camera using simple, pre-defined commands.

3. The "Motion Primitives" (The Lego Blocks)

How does LAMP know what "orbit" or "follow" means? It uses a set of Lego blocks called Motion Primitives.

Imagine you are building a movie scene with toys. You have specific blocks for:

Free-form: Just moving anywhere.
Orbit Track: Spinning around a target (like a planet around a sun).
Tail Track: Following behind something (like a dog following a runner).
Rotation Track: Spinning in place to look around.

LAMP takes your messy sentence and snaps these Lego blocks together into a perfect, logical sequence. Because these blocks are pre-made and tested, the camera movement is always smooth and makes sense physically.

4. The Assembly Line

Once LAMP has written the script (the Lego plan), it does two things:

It builds the 3D path: It turns the script into a literal 3D line showing exactly where the camera and the ship will go over time.
It hands it to the AI: It gives this 3D path to the video generator. The video generator doesn't have to guess anymore; it just has to "paint" the video along the path LAMP drew.

5. Why This is a Big Deal

It's Editable: If you don't like the result, you don't have to start over. You can just tell LAMP, "Move the camera lower," and it rewrites the script and updates the 3D path instantly. It's like editing a screenplay rather than reshooting the whole movie.
It's Cinematic: Because LAMP uses language inspired by real filmmaking (like "pan," "tilt," "dolly"), the videos it makes look like professional movies, not just random animations.
It's Consistent: The ship and the camera stay in sync. They don't get lost or drift apart because they are following the same strict script.

The Bottom Line

LAMP is a Language-Assisted Motion Planner. It takes your natural language ideas and turns them into a precise, step-by-step movie script for the camera and the objects. It bridges the gap between "I have a cool idea" and "Here is a perfect, controllable video," making it much easier for anyone to direct complex scenes without needing to be a math genius or a professional cinematographer.

1. Problem Statement

While recent text-to-video (T2V) models have achieved high visual fidelity, controlling complex motion (specifically the coordination of object dynamics and camera trajectories) remains a significant bottleneck.

Limitations of Current Interfaces: Existing methods rely on text prompts, 2D sketches, or extracting annotations from existing videos. These are often insufficient for describing choreographed, cinematic scenes where object movement and camera paths are inherently entangled (e.g., a camera tracking a moving subject).
The Gap: Directly regressing raw 3D coordinates from text is difficult due to the lack of large-scale paired data and the complexity of spatial reasoning. Users struggle to mentally visualize and specify precise 3D trajectories for both objects and cameras simultaneously.
Goal: To create a framework that translates natural language descriptions into explicit, geometrically consistent 3D trajectories for both objects and cameras, enabling cinematic, physically coherent video generation.

2. Methodology: LAMP Framework

LAMP (Language-Assisted Motion Planning) reframes motion control as a language-to-program synthesis problem. Instead of predicting raw coordinates, it uses a Large Language Model (LLM) to generate symbolic motion programs in a Domain-Specific Language (DSL), which are then deterministically converted into 3D trajectories.

A. Motion Domain-Specific Language (DSL)

Inspired by cinematography conventions and the CameraBench taxonomy, LAMP defines a structured DSL to represent motion.

Primitives: The DSL includes four canonical motion primitives:
1. Free-form: Unconstrained 6-DoF motion.
2. Orbit Track: Camera revolves around a target.
3. Tail Track: Camera follows an object with temporal offsets.
4. Rotation Track: Camera rotates in place to frame a subject.
Modifiers: Each primitive is parameterized by key-value pairs (e.g., speed, ease, yaw, pitch, offset) allowing for fine-grained control over speed, direction, and style.
Structure: A motion program is an ordered list of tags (e.g., orbit_track deg_90 dir_cw spiral_out_0.5). This symbolic representation ensures interpretability and composability.

B. Procedural Dataset Construction

To train the motion planner, the authors constructed a large-scale dataset (400K text-motion pairs) without relying on existing video annotations:

Generation: Motion primitives and modifiers are sampled procedurally to create symbolic DSL programs.
Trajectory Synthesis: A deterministic converter maps DSL programs to 3D trajectories (object and camera) in a shared world coordinate system.
Text Generation: An auxiliary LLM paraphrases canonical descriptions (e.g., "camera pans right") to create diverse natural language prompts paired with the DSL and trajectories.
Scope: The dataset covers both free-form motions and object-relative motions, ensuring coverage of common and long-tail cinematic behaviors.

C. LLM-Based Motion Planning

Architecture: A vision-language LLM is fine-tuned on the procedural dataset.
Inference: Given a natural language prompt, the LLM decomposes the intent into object-centric ( $t_{obj}$ ) and camera-centric ( $t_{cam}$ ) components. It generates symbolic DSL programs for both, conditioned on the hierarchical structure of cinematography (subject motion defines dynamics; camera adjusts to maintain framing).
Iterative Refinement: Because the output is symbolic code, users can iteratively refine the motion (e.g., "move the camera lower") by editing the DSL before generating the final video, avoiding costly re-synthesis.

D. Trajectory-Conditioned Video Generation

The generated DSL programs are converted into 3D trajectories. These trajectories are rendered as a "control video" (containing 2D projections of 3D bounding boxes and horizon lines) which conditions a pre-trained video diffusion model (e.g., VACE) to synthesize the final video.

3. Key Contributions

Cinematography-Inspired DSL: The first motion language that encodes both object and camera behaviors as symbolic programs, deterministically mapped to 3D trajectories.
Large-Scale Procedural Dataset: A 400K sample dataset pairing text, DSL programs, and 3D trajectories, enabling scalable training of language-driven motion planners.
Unified Object-Camera Planning: A framework that jointly plans object and camera motion in a shared 3D space, addressing the entanglement problem that previous methods (focusing only on camera or static objects) failed to solve.
Iterative Control: A decoupled planning stage allowing users to refine motion intent via text before committing to video synthesis.

4. Experimental Results

The authors evaluated LAMP against state-of-the-art baselines (e.g., ET, GenDoP, Director3D, CCD) on camera trajectory alignment and object motion prediction.

Camera Trajectory Alignment:
- On the DataDoP and ET benchmarks, LAMP significantly outperformed baselines in F1-Score (semantic accuracy of motion tags) and CLaTr (text-trajectory alignment).
- Notably, LAMP trained only on the procedural dataset achieved superior performance to models explicitly trained on the DataDoP dataset, demonstrating strong generalization.
Object Motion Prediction:
- Achieved near-perfect F1 scores (0.9983 for coarse translation, 0.9293 for fine translation) on the procedural test set.
Ablation Studies:
- Comparing DSL vs. Direct Trajectory Regression: The DSL approach yielded significantly higher directional accuracy and lower rotation error, proving that symbolic reasoning is more effective than direct coordinate regression for complex motion.
User Study:
- In a study with 16 participants, LAMP was preferred over baselines (including GPT-5 prompted with DSL) in camera adherence, object adherence, and overall video quality (approx. 50-67% preference rate).
Qualitative Results: Generated videos showed coherent camera-object coordination, physically consistent dynamics, and faithful adherence to complex cinematic descriptions (e.g., "camera orbits 90 degrees while spiraling out").

5. Significance and Impact

Paradigm Shift: LAMP moves video generation from "text-to-pixel" to "text-to-logic-to-pixel," introducing a layer of interpretable, editable control.
Director-Level Control: It bridges the gap between casual text prompts and professional cinematography, allowing users to specify complex, multi-stage camera moves and object choreography without technical expertise in 3D modeling.
Model Agnostic: The DSL-to-trajectory converter acts as a universal interface, compatible with various pre-trained video generators (T2V, I2V, V2V).
Future Directions: The work lays the foundation for "Language-Driven Cinematography," where high-level intent directly governs motion design, potentially evolving to support multi-object interactions and dynamic world editing.

In summary, LAMP successfully demonstrates that leveraging LLMs for symbolic motion planning, grounded in a cinematography-inspired DSL, solves the controllability bottleneck in video generation, producing results that are both physically consistent and narratively expressive.