💻 computer science

CinemaTraj: Composing Atomic Camera Trajectories for 3D Scenes with LLM Agents

CinemaTraj is a novel framework that leverages LLM agents and structured 3D scene graphs to decompose natural language prompts into atomic, collision-free camera movements, thereby generating high-quality, narrated cinematic videos from real-world 3D environments.

Original authors: Qianru Li, Xuyang Chen, Erkin Türköz, Lu Liu, Xuqin Wang, Liqiu Meng, Tao Wu, Yanfeng Zhang

Published 2026-07-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qianru Li, Xuyang Chen, Erkin Türköz, Lu Liu, Xuqin Wang, Liqiu Meng, Tao Wu, Yanfeng Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are holding a magic camera that can fly through any room, but instead of you holding it, you just have to talk to it. "Fly around the sofa, then zoom in on the vase," you say. In the world of computer science, this is a dream called camera trajectory generation. It's the art of teaching computers to move a virtual camera through a 3D space in a way that looks like a movie, not just a random walk.

For a long time, computers have been great at two separate things: understanding language (like reading a story) and understanding 3D shapes (like knowing where a wall is). But getting them to work together has been tricky. Some old methods tried to guess the path based only on flat pictures, which is like trying to navigate a maze while wearing blinders—they often crashed into invisible walls. Other methods were very good at avoiding walls but had no idea how to make the camera move in a "cinematic" way, resulting in shaky, boring footage that looked like a security camera, not a movie. The big question was: Could we teach a computer to understand both the words of a movie director and the physics of a 3D room at the same time?

Enter CinemaTraj, a new framework that acts like a super-smart film director's assistant. The researchers behind it realized that to make a great movie path, you need a "brain" that understands language and a "map" that understands 3D space. They built a system where a Large Language Model (LLM)—a type of AI that is very good at understanding human speech—gets a special "3D scene graph." Think of this graph as a detailed, digital blueprint of the room that tells the AI exactly where the sofa is, if it's touching the wall, and how far the ceiling is.

With this blueprint in hand, the AI doesn't just guess; it plans. When you type, "Give me a cinematic tour of this living room," the AI breaks that request down into tiny, professional movie moves. It decides to "orbit" the sofa (circle around it), "crane" up toward the TV (move vertically), and "zoom" in on a vase. These aren't random movements; they are specific, named camera techniques that real directors use. The system then builds a smooth path for these moves, but here's the clever part: it uses a mathematical "safety net" called a Signed Distance Field (SDF). Imagine this as an invisible force field that pushes the camera away from furniture if it gets too close, ensuring the camera never crashes into the coffee table or gets stuck behind a chair.

The paper shows that this approach works incredibly well. In tests using real-world 3D scans of rooms (from a dataset called ScanNet++), CinemaTraj created paths that were much smoother and safer than previous methods. It followed the user's instructions perfectly, avoided collisions almost all the time, and even added synchronized voiceovers and subtitles to tell a story as the camera moved. The researchers found that their method was far better at making the camera move like a human director would, rather than just calculating a geometric line.

However, the paper is careful to note what it doesn't do yet. It currently works in static rooms where nothing moves. If a person walked through the scene while the camera was filming, the system wouldn't know to dodge them. It also focuses on single, continuous shots rather than editing together a whole movie with cuts and scene changes. But for now, CinemaTraj proves that by giving an AI a good map and teaching it the language of movies, we can finally generate 3D camera paths that are both safe and truly cinematic.

Technical Summary: CinemaTraj

Problem Statement

The paper addresses the challenge of automatically generating cinematically expressive and physically plausible camera trajectories through 3D scenes based on natural language descriptions. Existing approaches suffer from two primary limitations:

Lack of 3D Spatial Awareness: Language-guided methods relying on 2D image priors or feedforward networks often fail to perceive solid surfaces, resulting in trajectories that intersect with scene geometry (collisions) or fail to achieve object-level framing.
Disconnection from Cinematographic Semantics: 3D-aware methods that treat trajectory generation as a purely geometric optimization problem can navigate around obstacles but cannot interpret high-level cinematographic intent (e.g., specific shot types like dolly, orbit, or crane) from natural language.

There is a critical gap in jointly reasoning about 3D scene structure and cinematographic intent to produce collision-free, narratively coherent camera paths.

Methodology

CinemaTraj reframes camera trajectory planning as a language-grounded spatial reasoning problem. The framework consists of four sequential modules:

1. Scene Graph Extractor

The system processes RGB-D input images to construct a hierarchical 3D scene graph ( $G = (N, E)$ ).

Nodes: Represent rooms, objects, and structures, each annotated with semantic labels and Oriented Bounding Boxes (OBBs).
Edges: Encode specific relations relevant to camera movement:
- Room-Object Containment: Determines which spatial region the camera must navigate.
- Object-Structure Relations: Labels objects as "wall-adjacent" or "ceiling-attached." These labels directly constrain feasible movements (e.g., wall-adjacent objects forbid full orbits; ceiling objects forbid crane shots).
Geometry: Structural bounding boxes are extended to form a watertight enclosure, ensuring the Signed Distance Field (SDF) used later has no leakage.

2. User Prompt Translator (LLM Agent)

An LLM agent is equipped with the scene graph and a predefined "Cinematographic Toolset" to decompose free-form text prompts into a structured sequence of atomic commands.

Toolset:
- Anchor Selector ( $A$ ): Selects collision-free initial viewpoints based on scoring metrics (visibility, distance, elevation, openness).
- Object-level Trajectories ( $T_o$ ): Includes specific shot types (orbit, dolly, crane, pan, tilt, zoom, static).
- Transitional Trajectories ( $T_t$ ): Parameterized arcs connecting objects.
Output: A sequence of $3N-1$ atomic steps (Anchor $\to$ Object Trajectory $\to$ Transition $\to$ Anchor...) that decouples intent understanding from trajectory generation.

3. Parametric Atomic Trajectory Builder

This module instantiates the atomic commands into continuous camera paths using a parametric representation.

Representation: Each trajectory $\tau(t; \theta_{fix}, \theta_{free})$ is defined by fixed parameters (anchors, geometry) and free parameters (optimizable).
Motion Profiles: The parametric design preserves characteristic cinematic motion profiles (e.g., smooth orbital arcs, gradual dolly approaches) rather than generating unconstrained 6-DoF poses.
Types:
- Orbit/Dolly/Crane: Defined by radius, height, sweep angles, and elevation.
- Pan/Tilt/Zoom: Defined by interpolation of orientation or focal length.
- Transition: Modeled as a parameterized arc with a curvature angle $\alpha$ .

4. Collision and Occlusion-Free Trajectory Optimizer

To prevent collisions with cluttered geometry and ensure target visibility, the system refines the free parameters ( $\theta_{free}$ ) via differentiable optimization.

SDF Construction: A Signed Distance Field is computed from the OBB mesh, providing smooth, geometry-faithful gradients (superior to noisy 3D Gaussian Splatting density fields).
Cost Function: Minimizes a combination of:
- Collision Cost ( $C_{sdf}$ ): Penalizes points penetrating or approaching scene geometry.
- Occlusion Cost ( $C_{occl}$ ): Uses ray marching and transmittance weighting to penalize blocked views of target objects.
- Regularization ( $C_{reg}$ ): Keeps the optimized trajectory close to the initial parametric shape.
Two-Pass Optimization:
1. Optimizes object-level segments with full costs (including occlusion).
2. Rebuilds and optimizes transitional arcs without occlusion costs, splitting cross-room paths through door waypoints.

5. Subtitle and Voiceover Generator

(Optional) A Vision-Language Model (VLM) analyzes keyframes to generate synchronized subtitles and voiceover narration, enhancing the cinematic output.

Key Contributions

Framework Formalization: An end-to-end framework for language-guided cinematic video generation from 3D scenes, treating trajectory planning as a spatial reasoning problem.
Structured 3D Priors: The use of 3D scene graphs as structured priors to ground LLM reasoning in accurate geometric and semantic knowledge, combined with a parametric trajectory representation based on professional camera idioms.
Differentiable Optimization: A novel occlusion and collision-free optimization method specifically designed for cinematographic shots, utilizing SDFs to preserve shot style while ensuring physical safety.
Comprehensive Evaluation: Extensive experiments demonstrating the method's superiority in prompt alignment, trajectory quality, and safety metrics.

Experimental Results

The method was evaluated on 50 scenes from the ScanNet++ dataset using three levels of prompt specificity (fully-specified, partially-specified, and open-ended).

Baselines: Compared against ChatCam + GenDoP (text-conditioned diffusion without 3D geometry) and CCTG (geometry-aware but lacking explicit cinematographic constraints).
Quantitative Performance:
- Motion Fidelity: CinemaTraj achieved the lowest Motion MSE (1.741), significantly outperforming baselines (ChatCam: 7.741, CCTG: 9.143).
- Safety: While the "w/o Anchor" ablation had a slightly lower collision rate, CinemaTraj maintained a very low collision rate (0.056) compared to ChatCam (0.209).
- Semantic Alignment: Achieved the highest CLaTr Score (28.982), indicating strong alignment between the generated trajectory and the text prompt.
- Coverage: Achieved 100% object coverage in fully-specified prompts.
User Study: In a study with 40 participants, CinemaTraj was strongly preferred over all baselines and ablation variants across Prompt Alignment, Collision/Occlusion Avoidance, and Cinematographic Quality.

Significance and Claims

The paper claims that CinemaTraj successfully bridges the gap between natural language instructions and physically valid, professionally styled camera movements in 3D environments. By grounding an LLM agent in a structured 3D scene graph and utilizing a parametric representation optimized via SDFs, the system produces trajectories that are:

Prompt-Faithful: Accurately executing specific cinematographic instructions (e.g., "orbit," "crane").
Collision-Free: Navigating complex, cluttered indoor environments without intersecting geometry.
Cinematically Grounded: Preserving the smooth motion profiles and visual semantics of professional cinematography, unlike geometric-only or text-only approaches.

The authors note that the current work focuses on static 3D scenes and foundational movement types, leaving dynamic scenes, subject-relative shots (tracking, over-the-shoulder), and multi-shot editing grammar (e.g., 180-degree rule) as directions for future research.