Geometry-Guided Camera Motion Understanding in VideoLLMs

This paper addresses the inability of current VideoLLMs to explicitly recognize camera motion by introducing a comprehensive framework that includes a new dataset and benchmark, a diagnostic analysis of visual encoder weaknesses, and a lightweight, training-free pipeline that injects geometric camera cues from 3D foundation models to significantly improve motion-aware reasoning.

The Gist

Imagine you are watching a movie. You see the actors, the explosions, and the dialogue. But have you ever noticed how the camera moves? Is it zooming in to show fear? Is it panning left to reveal a surprise? Is it tilting up to show a giant monster?

This movement is the "grammar" of filmmaking. It tells you how to feel and where to look.

The problem, according to this paper, is that modern AI video models (called VideoLLMs) are great at describing what is happening (e.g., "a man is running"), but they are terrible at describing how the camera is moving. They often get confused, mixing up the actor's movement with the camera's movement, or just guessing randomly.

Here is a simple breakdown of how the authors fixed this, using some creative analogies.

1. The Problem: The "Blind" AI

Think of a VideoLLM as a very smart student who has read millions of books and seen thousands of movies. But, this student has a specific blind spot: they don't understand the camera.

If you show them a video of a car driving past a tree, they might say, "The tree is moving backward." They don't realize the camera is moving forward while the tree stays still. They lack the "geometric sense" to tell the difference between the world moving and the camera moving.

2. The Solution: The "Camera Translator"

The authors didn't want to retrain the whole AI student (which would be like sending them back to school for 10 years). Instead, they built a specialized translator that sits next to the student.

Here is how their new system works, step-by-step:

Step A: The "3D GPS" (The Teacher)

First, they used a powerful, pre-trained AI called VGGT. Think of VGGT as a 3D GPS navigator that knows exactly where the camera is in space at every single second. It calculates the camera's position, rotation, and speed with mathematical precision.

• Analogy: If the VideoLLM is a tourist looking out the window, VGGT is the pilot in the cockpit who knows the exact flight path.

Step B: The "Motion Dictionary" (The Classifier)

The GPS data is too complex for the student to understand directly. So, they built a small, lightweight "translator" (a classifier). This translator takes the GPS data and turns it into simple, human-readable labels like "Pan Left," "Zoom In," or "Tilt Up."

• Analogy: The translator converts the pilot's complex coordinates into a simple note: "We are turning left."

Step C: The "Cheat Sheet" (Structured Prompting)

This is the magic trick. Instead of forcing the student to learn from scratch, the authors simply hand the student a cheat sheet before they answer a question.
They take the "Pan Left" label and paste it into the prompt: "Here is a video. By the way, the camera is panning left. Now, describe the video."

• Analogy: It's like giving a student a hint during a test. The student doesn't need to learn the math; they just need to use the hint to write a better answer.

3. The Results: From "Vague" to "Cinematic"

The authors tested this on a new dataset they created (like a practice exam for camera moves).

• Without the cheat sheet: The AI said things like, "The camera moves quickly." (Vague and often wrong).
• With the cheat sheet: The AI said, "The camera pans left to reveal the drummer, then tilts up to show the lights." (Precise and cinematic).

4. Why This Matters

The paper shows that current AI models are "geometry-blind." They see the content but miss the structure. By adding this external "3D GPS" and feeding the information as a hint, they made the AI much smarter at understanding movies without needing to retrain the whole system.

In summary:
The authors realized AI movies were missing the "camera language." They didn't try to teach the AI to be a cinematographer from scratch. Instead, they hired a 3D GPS expert to tell the AI exactly what the camera is doing, and then simply whispered those instructions to the AI as it described the scene. The result? The AI suddenly started speaking like a professional filmmaker.

Technical Summary

1. Problem Statement

Current Video Large Language Models (VideoLLMs) excel at high-level semantic understanding (objects, actions, narratives) but fail significantly at recognizing fine-grained camera motion primitives (e.g., pan, tilt, dolly, truck).

• The Gap: Camera motion is a fundamental geometric signal crucial for cinematic grammar, spatial reasoning, and authorial intent. However, VideoLLMs often confuse camera movement with object motion, fail to distinguish directions (e.g., pan-left vs. pan-right), and hallucinate motion due to a lack of explicit geometric supervision in their training data.
• Root Cause: The authors hypothesize that camera motion cues are weakly represented in the intermediate features of VideoLLM vision encoders. As visual tokens are compressed through deep network layers (e.g., in ViT blocks), geometric and temporal cues necessary for distinguishing camera dynamics are attenuated or lost.
• Challenge: Existing datasets lack precise, geometry-grounded labels for short segments, and retraining large VideoLLMs to learn these signals is computationally expensive and data-intensive.

2. Methodology

The authors propose a lightweight, model-agnostic pipeline that injects geometric camera cues into frozen VideoLLMs via structured prompting, avoiding costly fine-tuning. The approach consists of three main components:

A. Data and Benchmark Construction

• CameraMotionDataset: A large-scale synthetic dataset (12k segments) derived from the MultiCamVideo dataset (ReCamMaster). It consists of non-overlapping 1-second within-shot segments rendered in Unreal Engine 5 with known camera extrinsics (pose and intrinsics).
  • Labels: Annotated with 15 atomic motion primitives (e.g., pan-left, dolly-in, arc-clockwise) based on a taxonomy from CameraBench.
  • Constraints: Enforces physical incompatibility (e.g., a camera cannot pan left and right simultaneously) and limits label cardinality (max 3 primitives per segment).
• CameraMotionVQA: A multiple-choice benchmark where models must select the correct set of motion primitives from four candidates (one ground truth, three distractors) for a given 1-second clip.

B. Geometry-Guided Cue Extraction & Classification

Instead of relying on the VideoLLM's internal features, the system extracts camera motion from an external 3D Foundation Model (3DFM):

1. Teacher Model: Uses a frozen VGGT (Visual Geometry Grounded Transformer), which predicts camera parameters and pose tokens from video frames.
2. Temporal Classifier: A lightweight Transformer classifier processes the sequence of camera tokens from VGGT.
   • It employs a constraint-regularized loss function that includes:
     • Binary Cross-Entropy (standard classification).
     • Incompatibility Regularization: Penalizes predictions that violate physical constraints (e.g., predicting both pan-left and pan-right).
     • Cardinality Regularization: Ensures the number of predicted primitives stays within a valid range (1–3).
3. Distillation (Efficiency): To reduce the high inference cost of the 1.2B parameter VGGT, the authors distill its camera perception into a lightweight Q-Former student model that reuses the frozen VideoLLM's visual features.

C. Structured Prompting Injection

The predicted motion primitives are injected into the VideoLLM inference pipeline via structured prompting:

• The system generates a per-second motion list (e.g., [static, pan-left, dolly-in]).
• This list is prepended to the user prompt as an explicit temporal scaffold.
• The prompt instructs the VideoLLM to describe the video using "filmmaker's language," explicitly grounding the description in the provided motion sequence.
• Key Advantage: This requires no weight updates to the VideoLLM, making it a plug-and-play solution.

D. Diagnosis via Probing

The authors conducted probing experiments on Qwen2.5-VL using Q-Former-style query tokens to analyze where motion information is lost.

• Finding: Camera motion cues are strongly recoverable in shallow layers but degrade significantly in deeper ViT blocks, confirming that token compression attenuates geometric signals.

3. Key Contributions

1. CameraMotionDataset & CameraMotionVQA: The first large-scale, shot-consistent dataset with fine-grained, constraint-aware camera motion labels and a standardized VQA benchmark for evaluating VideoLLMs.
2. Geometry-Guided Pipeline: A novel framework that extracts camera cues from frozen 3DFMs, predicts constrained motion primitives, and injects them into VideoLLMs via structured prompting without retraining.
3. Diagnostic Insights: Empirical evidence demonstrating that current VideoLLMs suffer from a "representation gap" regarding camera motion, which is recoverable only through external geometric priors.
4. Distillation Strategy: A method to distill heavy 3D foundation model cues into efficient student models, achieving a favorable trade-off between accuracy and inference latency.

4. Experimental Results

• Baseline Performance: Off-the-shelf VideoLLMs (including Qwen2.5-VL and InternVL) perform near random guessing (~25% accuracy) on CameraMotionVQA, often confusing geometrically similar primitives.
• Proposed Method Performance:
  • The VGGT-based classifier achieves 73.8% instance accuracy and 0.87 Macro-F1 on the dataset, significantly outperforming all baseline VideoLLMs.
  • The Distilled VGGT–Q-Former model achieves 63.8% accuracy (only ~8% drop from the teacher) while reducing peak GPU memory usage by 61% and increasing throughput by 5.3x.
• Qualitative Improvements: With structured prompting, VideoLLMs generate descriptions that are:
  • Temporally Grounded: Correctly sequence events (e.g., "then pan-left, then dolly-in").
  • Geometrically Accurate: Distinguish between camera moves and object moves.
  • Cinematographically Aware: Use correct terminology (e.g., "medium close-up," "whip pan") and maintain spatial coherence.

5. Significance and Conclusion

This work addresses a critical blind spot in VideoLLMs: the inability to understand how a video is filmed, not just what is in it.

• Practical Impact: The proposed plug-and-play pipeline enables camera-aware applications such as descriptive video services (DVS), film-style retrieval, and spatial reasoning without the prohibitive cost of fine-tuning massive models.
• Theoretical Insight: It establishes that geometric priors from 3D foundation models are essential for recovering motion signals lost in semantic-focused vision encoders.
• Future Directions: The authors suggest extending this to intrinsic camera changes (e.g., zoom), exploring broader 3DFM backbones, and investigating how these cues transfer to video generation tasks.

In summary, the paper demonstrates that geometry-driven cue extraction combined with structured prompting is a highly effective, efficient, and scalable strategy to bridge the gap between current VideoLLMs and the nuanced requirements of cinematic understanding.