💻 computer science

CT-1: Vision-Language-Camera Models Transfer Spatial Reasoning Knowledge to Camera-Controllable Video Generation

The paper introduces CT-1, a novel Vision-Language-Camera model that leverages a specialized data pipeline and wavelet-based regularization to transfer spatial reasoning knowledge into camera-controllable video generation, achieving significantly improved trajectory accuracy and physically plausible motion synthesis.

Original authors: Haoyu Zhao, Zihao Zhang, Jiaxi Gu, Haoran Chen, Qingping Zheng, Pin Tang, Yeyin Jin, Yuang Zhang, Junqi Cheng, Zenghui Lu, Peng Shu, Zuxuan Wu, Yu-Gang Jiang

Published 2026-04-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haoyu Zhao, Zihao Zhang, Jiaxi Gu, Haoran Chen, Qingping Zheng, Pin Tang, Yeyin Jin, Yuang Zhang, Junqi Cheng, Zenghui Lu, Peng Shu, Zuxuan Wu, Yu-Gang Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director filming a movie, but instead of holding a physical camera, you are talking to a robot that generates the video for you. You want to say, "The camera should slowly zoom in on the cat's eyes," or "Fly the camera backward while turning right to show the whole room."

In the past, telling a computer this was like trying to give directions to a blindfolded person who only speaks a different language. You either had to be incredibly vague (which led to the robot guessing wrong) or you had to manually type out complex math coordinates (which is boring and hard to do).

Enter CT-1, a new "super-intelligent camera operator" created by researchers. Here is how it works, explained simply:

1. The Problem: The "Blind" Camera

Existing video generators are like talented painters who can draw amazing scenes, but they don't understand how to move the "camera" around the scene.

If you say "Zoom in," they might just make the picture bigger (like a digital zoom) rather than actually moving the viewpoint forward.
If you say "Pan right," they might just slide the image sideways, breaking the 3D illusion.
The old way: You had to manually calculate the exact math for every second of the video. It was like trying to drive a car by manually calculating the rotation of every wheel.

2. The Solution: CT-1 (The "Brain" of the Operation)

The researchers built a new model called CT-1 (Camera Transformer 1). Think of CT-1 as a smart translator that sits between your brain and the video generator.

It has two eyes and a brain: It looks at a reference image (the scene) and reads your text instructions (the director's notes).
It does the math for you: Instead of you typing coordinates, CT-1 figures out exactly how the camera needs to move through 3D space to match your words. It predicts a smooth, realistic path, like a drone pilot flying a route.
It understands context: If you say "Zoom in on the cat," CT-1 knows the cat is in the center of the image, so it moves the camera forward toward the cat, not just randomly.

3. The Secret Sauce: The "Wavelet" Filter

One of the coolest parts of this paper is how they taught the model to move smoothly.
Imagine you are drawing a line on a piece of paper.

The "Jitter" Problem: Sometimes, when computers try to draw a line, they shake a little bit (like a shaky hand). This makes the video look glitchy.
The Wavelet Trick: The researchers used a mathematical tool called a Wavelet Transform. Think of this like a noise-canceling headphone for movement.
- It separates the "big picture" movement (the smooth path you want) from the "tiny shakes" (the jitter).
- It tells the model: "Keep the smooth, big movements, but throw away the tiny, jittery shakes."
- This ensures the camera glides like a professional drone operator, not a nervous tourist.

4. The Training Data: The "Camera Gym"

To teach CT-1 how to be this good, the researchers couldn't just use existing videos because they didn't have the "camera movement notes" attached.

They built a massive new dataset called CT-200K.
Imagine a giant gym where millions of videos are being watched by AI. These AIs are trained to look at a video and write a script describing exactly how the camera moved, frame by frame.
They created 47 million frames of training data. It's like giving the model a library of every possible camera move in the world so it can learn the rules of physics and perspective.

5. The Result: A Magic Director

When you put CT-1 together with a video generator:

You upload a picture and type: "The camera flies over the city, then dives down into the street."
CT-1 instantly calculates the perfect flight path.
It hands this path to the video generator.
The result is a video where the camera actually flies and dives in a physically realistic way, matching your description perfectly.

Why This Matters

Before this, making a video with specific camera moves was like trying to sculpt a statue with a sledgehammer—imprecise and messy.
CT-1 is like giving you a laser-guided chisel. It bridges the gap between your imagination (what you want to see) and the computer's execution (how the video is made). It makes video generation feel less like magic and more like a conversation with a skilled cameraman who knows exactly what you mean.

In short: CT-1 is the smart assistant that translates your "Director's Vision" into a perfect, smooth, 3D camera flight path, so you can finally get the video you imagined without needing a degree in mathematics.

1. Problem Statement

Camera-controllable video generation aims to synthesize videos with flexible, physically plausible camera movements. However, existing approaches face two primary limitations:

Imprecise Semantic Control: High-level text-to-video methods (e.g., Wan2.2, CogVideoX) often fail to follow specific camera motion instructions, resulting in static shots or unintended movements.
Labor-Intensive Parameterization: Explicit parameterized control methods (e.g., CameraCtrl, MotionCtrl) require manually specifying camera trajectory parameters (extrinsics), which is not scalable for automated scenarios and lacks semantic alignment with user intent.

The core challenge is to bridge the gap between high-level user intentions (text) and low-level camera trajectories (SE(3) poses) while ensuring the generated motion is consistent with the visual scene context.

2. Methodology: CT-1 Framework

The authors propose CT-1 (Camera Transformer 1), a novel Vision-Language-Camera (VLC) model. Instead of directly generating video pixels, CT-1 acts as a "spatial reasoning engine" that infers camera trajectories from a reference image and a text prompt, which are then fed into a video diffusion model.

A. Architecture Overview

The framework consists of three main modules:

Vision-Language Module:
- Vision Encoder: Uses a dual-branch architecture (DINOv2 and SigLIP) to extract fine-grained details and high-level semantics from the input image.
- Language Encoder: Utilizes LLaMA-2 to process text instructions.
- Fusion: Visual tokens, linguistic tokens, and a learnable <CAM> (Camera-context) token are concatenated. The <CAM> token aggregates visual and linguistic information to serve as a conditioning signal for trajectory prediction.
Camera Transformer (Diffusion Transformer - DiT):
- Unlike regression models that predict a single deterministic trajectory, CT-1 models the distribution of plausible camera trajectories using a Diffusion Transformer.
- It takes the <CAM> token and diffusion timestep embeddings to denoise a trajectory sequence $K_{1:T} \in \mathbb{R}^{T \times C}$ (where $C$ represents SE(3) pose parameters).
Controllable Video Generation:
- The predicted trajectory is decoupled from the generation process. It is converted into a control signal and injected into a downstream video diffusion model (e.g., CameraNoise, CameraCtrl) to synthesize the final video.

B. Key Technical Innovations

Wavelet-based Regularization Loss (WavReg):
- Motivation: Camera trajectories are multi-scale structures. Low-frequency components represent global, smooth motion, while high-frequency components capture rapid, local perturbations (jitter).
- Mechanism: The authors apply a 1D Discrete Wavelet Transform (DWT) to the predicted trajectory. They introduce a loss function that penalizes errors in both low-frequency (global trend) and high-frequency (detail) components, with higher weights assigned to low-frequency components to ensure physical stability.
- Formula: $L = L_{diff} + \beta L_{wav}$ , where $L_{wav}$ enforces smoothness and structural fidelity across frequency scales.
Non-Uniqueness Handling: The model acknowledges that a single text prompt can correspond to multiple valid trajectories (one-to-many mapping). By using a diffusion process, CT-1 learns a distribution over these valid motions rather than a single fixed point.

3. Dataset: CT-200K

To train CT-1, the authors constructed CT-200K, a large-scale dataset containing over 47 million frames across 200K+ video samples.

Data Curation Pipeline:
- General Scenarios: Uses Video-VLMs and Image-VLMs to generate camera motion descriptions and content descriptions. An LLM filters out hallucinated motion cues to ensure alignment between the text and the visual content.
- Reasoning Scenarios: Focuses on egocentric videos where objects are manipulated. The pipeline generates instructions requiring spatial reasoning (e.g., "move the toolbox to the right of the drill") to force the model to infer camera motion based on object relationships.
- Pose Estimation: Utilizes the VGGT model to estimate ground-truth camera poses for unlabeled videos, providing the necessary trajectory supervision.

4. Experimental Results

The framework was evaluated on CameraBench100 (a subset of CameraBench) and various out-of-distribution (OOD) scenarios.

Quantitative Performance:
- Success Rate: CT-1 achieved an average success rate of 81.6%, outperforming the best prompt-input baseline (Wan2.2) by 25.7% and trajectory-input baselines by over 170%.
- Video Quality: Using VBench, the generated videos maintained high aesthetic and imaging quality (0.709), comparable to state-of-the-art video generators, proving that the trajectory injection does not degrade visual fidelity.
Qualitative Analysis:
- CT-1 successfully handles complex motions (e.g., "truck right while tracking backward") where baseline models often fail or produce static shots.
- It demonstrates strong spatial reasoning, correctly inferring camera movement based on implicit cues (e.g., focusing on a specific object mentioned in the text).
Ablation Studies:
- Scaling: Performance improves consistently with model size (Base $\to$ Large $\to$ Huge), validating the scalability of the DiT architecture.
- WavReg: The wavelet loss significantly improves motion smoothness and reduces jitter compared to standard velocity or acceleration regularization.
- Token Design: The dedicated <CAM> token is crucial; removing it or using pooled features significantly degrades performance.

5. Key Contributions

VLC Model Paradigm: Introduced a new model class that jointly reasons over visual observations and semantic instructions to infer spatially aware, temporally coherent camera trajectories.
CT-1 Architecture: Proposed a Transformer-based instantiation that injects camera-context tokens into a Diffusion Transformer, enabling distributional modeling of camera poses.
Wavelet Regularization: Developed a novel frequency-domain loss (WavReg) that effectively balances global motion trends with local stability, addressing the jitter issues common in trajectory generation.
CT-200K Dataset: Created a large-scale, curated dataset with over 47M frames, specifically designed to support training for both general camera control and complex spatial reasoning scenarios.
Modular Integration: Demonstrated that CT-1 can be decoupled and integrated with various existing video diffusion models (CameraNoise, CameraCtrl, MotionCtrl) to enhance their controllability without modifying their core architectures.

6. Significance

This work represents a significant step forward in controllable video generation. By shifting the focus from direct text-to-video generation to text-to-trajectory-to-video, CT-1 effectively transfers spatial reasoning knowledge from multimodal models to generative models. This approach solves the "hallucination" problem in camera control, allowing for precise, physically plausible, and semantically aligned camera movements. The introduction of the CT-200K dataset and the wavelet-based regularization technique provides a robust foundation for future research in 4D world modeling and automated video production.