💻 computer science

Unified Camera Positional Encoding for Controlled Video Generation

This paper introduces UCPE (Unified Camera Positional Encoding), a geometry-consistent representation that unifies camera poses, intrinsics, and lens distortions via Relative and Absolute Orientation Encodings, enabling state-of-the-art camera controllability in video generation with minimal parameter overhead.

Original authors: Cheng Zhang, Boying Li, Meng Wei, Yan-Pei Cao, Camilo Cruz Gambardella, Dinh Phung, Jianfei Cai

Published 2026-03-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Cheng Zhang, Boying Li, Meng Wei, Yan-Pei Cao, Camilo Cruz Gambardella, Dinh Phung, Jianfei Cai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are directing a movie, but instead of hiring a human cameraman, you are talking to a super-smart AI robot. You want to tell the robot exactly how to move: "Fly low over the grass, tilt the camera up to see the sky, and use a fisheye lens so everything looks curved and wide."

The problem is, most AI video generators today are like robots that only understand one type of camera: a standard, flat, "pinhole" camera (like a basic human eye). If you ask them to use a fisheye lens or a drone view, they get confused, the video looks warped, or the camera moves in weird, jerky ways. They don't truly understand the physics of how a camera sees the world.

This paper introduces a new system called UCPE (Unified Camera Positional Encoding). Think of it as giving the AI robot a universal translator for camera lenses.

Here is how it works, broken down into simple concepts:

1. The Problem: The "One-Size-Fits-All" Lens

Imagine you have a toy robot that only knows how to walk on flat ground. If you ask it to walk up a steep hill or swim in a pool, it falls over.

Old AI: Only understands "Pinhole" cameras (standard, flat vision). If you ask for a fisheye lens (like a security camera) or a wide-angle drone shot, the AI tries to force the flat rules onto a curved reality. The result? The video looks broken, the camera drifts, or the distortion looks fake.
The Goal: We need an AI that understands that a fisheye lens bends light differently than a telephoto lens, and that a drone sees the world from a different angle than a person walking on the street.

2. The Solution: "Relative Ray Encoding" (The Laser Pointer Analogy)

The authors realized that instead of describing the camera (the box holding the lens), we should describe the rays of light entering the camera.

The Old Way: Telling the AI, "I am holding a camera at this specific spot in the world." This is like giving someone a map with a "You Are Here" pin. It works if the map is perfect, but if the map changes (different lens), the pin is wrong.
The New Way (UCPE): Imagine every single pixel in the video is a tiny laser pointer shooting a beam of light out into the world.
- UCPE teaches the AI to track these laser beams.
- It doesn't matter if the lens is wide, narrow, or curved (fisheye). The AI just looks at where each laser beam is pointing relative to its neighbors.
- Analogy: Instead of telling a dancer, "Stand at the corner of the room," you tell them, "Keep your arm pointing at that specific tree." No matter how the room is distorted or how the camera moves, the relationship between the dancer and the tree stays true. This allows the AI to handle any lens type perfectly.

3. The "Gravity Compass" (Absolute Orientation)

There was another problem: If you tell an AI to "roll the camera" (tilt it sideways), it often gets lost. It doesn't know which way is "up" because it's just looking at pixels.

The Fix: The authors added a "Gravity Compass" (called Absolute Orientation Encoding).
How it works: They teach the AI to recognize the "Up" direction, just like a human knows the sky is up and the ground is down. They use a special map (a "Lat-Up map") that tells the AI, "This part of the image is the sky, and this part is the ground."
Result: Now, when you say "Roll the camera 45 degrees," the AI knows exactly what that means physically. It won't accidentally make the sky turn into the floor.

4. The Magic Trick: The "Lightweight Adapter"

Usually, to teach an AI new things, you have to retrain its entire brain, which takes months and massive computers.

The Innovation: The authors built a tiny "plug-in" (an adapter) that sits between the AI's brain and the camera instructions.
Analogy: Imagine the AI is a master chef who knows how to cook anything. You don't need to retrain the chef on how to use a new spice; you just give them a new recipe card (the adapter).
Benefit: This adapter is incredibly small (less than 1% of the AI's total size). It lets the existing, powerful video AI instantly understand complex camera movements and lenses without needing a massive overhaul.

Why This Matters

This technology is a game-changer for several fields:

Movie Makers: You can type, "Drone shot diving through a canyon with a fisheye lens," and get a realistic video, not a glitchy mess.
Self-Driving Cars: Cars use weird, wide-angle cameras to see everything around them. This AI can simulate how a car sees the world, helping to train self-driving software in a virtual world.
Robotics: Robots need to understand their 360-degree surroundings. This helps them "imagine" what the world looks like from different angles.

Summary

In short, UCPE is like giving a video-generating AI a universal remote control for reality. It understands that a fisheye lens bends the world, that a drone sees from above, and that "up" is always towards the sky. By teaching the AI to track the "laser beams" of light rather than just the camera box, it can generate stunning, realistic videos with any camera angle or lens you can imagine.

1. Problem Statement

While Transformers have become the backbone for 3D perception, video generation, and world models, existing methods for camera-controlled video generation suffer from significant limitations:

Simplified Geometric Assumptions: Most current approaches rely on pinhole camera models, ignoring the non-linear projections and strong lens distortions (e.g., fisheye, wide-angle) common in real-world applications like autonomous driving, robotics, and 360° panoramas.
Inadequate Encoding: Existing camera encodings fall into two categories:
- Absolute Encodings: Feed raw parameters or Plücker coordinates directly. These are sensitive to world-frame choices and fail to generalize across different scenes or coordinate systems.
- Relative Encodings: Model pairwise transformations (e.g., GTA, PRoPE) but typically assume pinhole projections and cannot handle spatially varying distortions within a single view.
Ambiguous Orientation: Current Text-to-Video (T2V) methods often define camera poses relative to the first frame, leaving the absolute orientation (specifically pitch and roll) ambiguous. This makes it impossible to specify a precise global "up" direction or gravity alignment.

2. Methodology: UCPE

The authors propose UCPE (Unified Camera Positional Encoding), a framework that unifies 6-DoF poses, intrinsics, and lens distortions into a geometry-consistent representation for Transformers.

A. Core Components

Relative Ray Encoding (RRE):
- Instead of treating an entire image as a single rigid entity (camera-level encoding), RRE reformulates the encoding at the ray level.
- Each image token is associated with a specific viewing ray defined by an origin and direction in world space.
- It constructs a local ray-to-world transformation matrix ( $T^{rw}_t$ ) for each token, anchored by the camera center and the ray direction.
- This allows the attention mechanism to reason in ray space, capturing fine-grained geometric variations caused by lens distortions and wide fields of view (FoV), making it compatible with arbitrary camera models (e.g., Unified Camera Model - UCM).
Absolute Orientation Encoding (AOE):
- To resolve the ambiguity of global rotation, the authors introduce a Latitude-Up (Lat-Up) map.
- This encoding anchors the camera to a gravity-aligned "up" direction.
- It consists of two parts:
  - Latitude Map: Encodes the angular elevation of each ray relative to the horizontal plane.
  - Up Map: Computes the 2D displacement on the image plane resulting from a small rotation of the ray toward the world "up" vector.
- This provides explicit control over pitch and roll, enabling the generation of videos with precise absolute orientations.

B. Integration: Spatial Attention Adapter

UCPE is integrated into pre-trained Diffusion Transformers (specifically Wan2.1) via a lightweight spatial attention adapter.
Mechanism: The adapter runs in parallel with the original self-attention layers. It injects the hybrid encoding (Relative Ray + Lat-Up) into the Query, Key, and Value projections.
Efficiency: The adapter uses a token projection compression strategy (e.g., reducing dimension to 1/8) and zero-initialized linear layers. This adds less than 1% trainable parameters (approx. 35.5M parameters on a 7.3B model) while preserving the visual fidelity of the pre-trained model.

3. Key Contributions

Unified Representation (UCPE): A model-agnostic framework that jointly encodes 6-DoF poses, intrinsics, and lens distortions, enabling consistent reasoning across heterogeneous camera geometries (pinhole, wide-angle, fisheye).
Absolute Orientation Control: The introduction of the Lat-Up map allows for precise, repeatable control over the initial camera's pitch and roll, solving the global orientation ambiguity in T2V generation.
Efficient Adapter Design: A lightweight attention injection method that enables camera-aware generation with minimal parameter overhead (<1%), avoiding the need for full model retraining.
Large-Scale Benchmark: Construction of a diverse dataset (~48k clips) synthesized from 360° videos using the Unified Camera Model (UCM), covering a wide range of FoVs, distortion levels, and motion trajectories.

4. Experimental Results

The authors evaluated UCPE on their synthesized dataset and the RealEstate10K dataset, comparing against baselines like ReCamMaster, Wan CameraCtrl, CameraCtrl, and AC3D.

Quantitative Performance:
- Pose Control: UCPE achieved the lowest Rotation Error (RotErr) and Translation Error (TransErr), significantly outperforming baselines (e.g., ~4.12° RotErr vs. ~9.23° for ReCamMaster with absolute control).
- Lens Control: It demonstrated superior accuracy in controlling FoV and distortion coefficients ( $k_1, k_2$ ), faithfully reproducing fisheye and wide-angle effects.
- Video Quality: Maintained high visual fidelity (FVD, FID, CLIP scores) comparable to or better than baselines, despite adding minimal parameters.
Qualitative Results:
- UCPE faithfully followed complex camera trajectories and produced consistent lens distortions aligned with the Lat-Up map.
- Baselines often exhibited motion deviations, failed to reproduce intended distortions, or produced artifacts under pinhole settings.
Generalization:
- UCPE generalized well to the RealEstate10K dataset (which uses pinhole cameras) without fine-tuning, outperforming models trained specifically on that data.
- It showed robustness to text-camera conflicts (e.g., generating a fisheye view even if the text prompt suggested a telephoto view).
- It could generalize to unseen camera models (e.g., Brown-Conrady) at inference time by simply swapping the ray-mapping function.

5. Significance

Bridging Physics and Generative AI: UCPE provides a unified bridge between physical camera geometries (including complex distortions) and the attention mechanisms of Transformers.
Enabling Real-World Applications: By supporting diverse lenses and absolute orientation, UCPE is directly applicable to autonomous driving (fisheye dashcams), embodied AI (robotic perception), and world models that require 360° or wide-angle understanding.
Efficiency: The approach demonstrates that high-fidelity, geometry-aware video generation does not require massive parameter increases, making it feasible to upgrade existing large-scale video models with advanced camera control.
Future Potential: The ray-space formulation suggests a path toward generalizable 3D representations that are not tied to specific projection models, potentially benefiting novel view synthesis and 3D reconstruction tasks.