🤖 AI

CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation

The paper proposes CRePE, a novel positional encoding method that leverages curved ray expectations and a geometric attention adapter to enable unified, stable camera control and external geometry conditioning in video generation, effectively addressing the limitations of existing pinhole-based encodings for wide-angle and fisheye lenses.

Original authors: Seonghyun Jin, Youngmin Kim, Sunwoo Park, Jong Chul Ye

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Seonghyun Jin, Youngmin Kim, Sunwoo Park, Jong Chul Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot artist how to paint a moving video scene. You want to tell the robot exactly how the camera should move (pan left, zoom in, spin around) and what kind of "lens" to use (like a standard camera, a wide-angle lens that stretches the edges, or a fisheye lens that makes everything look like a bubble).

The problem with previous robot artists is that they only understood the direction the camera was looking, but not how far away things were in that direction. It's like giving someone a compass that tells them "North" but doesn't tell them if a mountain is 1 mile away or 100 miles away. When the camera moves, the robot gets confused about where objects should be, especially with weird lenses like fisheye that bend the world.

This paper introduces a new tool called CRePE (Curved Ray Expectation Positional Encoding) to fix this. Here is how it works, using simple analogies:

1. The "Fuzzy Flashlight" vs. The "Laser Pointer"

Old methods acted like a laser pointer. They drew a straight line from the camera to a specific spot in the image and said, "This pixel is right here." This works okay for normal cameras, but fails with wide-angle or fisheye lenses because those lenses bend light like a funhouse mirror. The straight line doesn't match the bent reality.

CRePE acts like a fuzzy flashlight. Instead of saying, "This pixel is exactly at point X," it says, "This pixel is somewhere along this curved path, likely between these distances." It creates a "probability cloud" of where the object might be along the line of sight. This allows the robot to understand that with a fisheye lens, the path to an object isn't a straight line; it's a curve.

2. The "Smart GPS" for Video Tokens

In AI video generation, the image is broken into tiny puzzle pieces called "tokens."

The Old Way: The robot knew which direction each puzzle piece was facing, but not how deep it was in the scene.
The CRePE Way: CRePE gives every puzzle piece a "Smart GPS" tag. It predicts a distance and a range of uncertainty for that piece. It tells the robot, "This piece of the street is likely 5 meters away, give or take 1 meter."

3. The "Middle-Manager" Strategy

The paper tested where to put this new "Smart GPS" inside the robot's brain (which is a massive neural network).

They found that putting it in the very beginning or the very end of the process didn't work well.
The Sweet Spot: They found that the middle layers of the brain are the best place for this information. It's like having a middle manager who knows the details of the project (the geometry) and can organize the workers (the tokens) perfectly before the final product is finished.

4. The "Training Wheels" (Pseudo-Supervision)

To teach the robot this new skill, the researchers used a "training wheel" system. They used a separate, pre-existing AI (a "geometry foundation model") to guess the distances in the training videos.

They didn't force the robot to copy these guesses perfectly.
Instead, they used the guesses as a safety net. If the robot's guess was wildly wrong, the safety net nudged it back toward reality. If the safety net's guess was bad (like looking at a blurry sky), the robot was allowed to trust its own judgment.

5. The Result: Better Movies with Bending Lenses

When they tested CRePE:

Better Camera Control: The videos followed the requested camera movements much more accurately, especially with wide-angle and fisheye lenses.
Better Geometry: The objects in the video stayed in the right place as the camera moved, without warping or floating weirdly.
Bonus Feature: Because the robot now understands "distance" so well, you can actually swap in a distance map from a different video. This lets you take the movement from one video and the shape of the world from another, creating new scenes that didn't exist before.

In Summary:
CRePE is a new way of teaching AI video generators to understand depth and curved lenses. Instead of just knowing which way to look, the AI now knows how far away things are, allowing it to create smooth, realistic videos even when using crazy, distorted camera lenses.

Technical Summary: CRePE for Unified-Camera-Controlled Video Generation

Problem Statement

Camera-conditioned video generation requires models to synthesize videos that adhere to specific camera trajectories and lens configurations. While recent approaches have moved toward attention-level positional encoding to model relative token relationships under view changes, existing methods face significant limitations:

Ray-Only Limitations: Methods like UCPE encode the viewing ray direction but do not specify the scene content's position along that ray. This is insufficient for modeling token correspondences when the camera moves, as correspondences depend on both ray direction and scene depth.
Pinhole Bias: Many existing encodings rely on pinhole camera geometry, which fails to capture the nonlinear projection geometry inherent in wide-angle and fisheye lenses (Unified Camera Model or UCM).
Lack of Depth Awareness: Current positional encodings do not effectively integrate scene geometry, limiting the model's ability to handle diverse lens models and refine token relationships during view changes.

Methodology: CRePE

The authors propose Curved Ray Expectation Positional Encoding (CRePE), a Unified Camera Model (UCM)-compatible positional encoding that represents each image token not just by its viewing ray, but as a depth-aware positional distribution along that ray.

Core Components

Curved-Ray Expected RoPE:
- Instead of assuming a single endpoint, CRePE models the token's position as a uniform distribution in log radial distance ( $z = \log r$ ) defined by a predicted center ( $\mu$ ) and interval width ( $\sigma$ ).
- This distribution is discretized into $K$ breakpoints. Each breakpoint is lifted along the source UCM ray, transformed into the query camera frame, and forward-projected.
- Because UCM projection induces a curved path in the query view for a source ray interval, CRePE analytically integrates the Rotary Positional Encoding (RoPE) phasor over this piecewise-linear curved path. This yields an expected modulation coefficient that captures ray-position uncertainty under non-pinhole projection.
- The default setting uses $K=5$ breakpoints to balance geometric accuracy and computational efficiency.
Geometric Attention Adapter:
- CRePE is implemented via a Geometric Attention Adapter added to a frozen video Diffusion Transformer (DiT).
- A shared geometry head ( $g_\phi$ ) predicts the log radial-distance parameters ( $\mu, \sigma$ ) from key-side token features.
- These predictions are converted into expected RoPE modulation coefficients, which modulate the key features in selected middle layers of the DiT.
- The design preserves the stable "ray-only" camera signal in early and late layers (using standard UCPE) while injecting depth-aware projected-path modulation in the middle layers where radial-distance information is most recoverable.
Radial-Distance Supervision & Radial MixForcing:
- To stabilize the geometry prediction, the model is supervised with pseudo radial-distance maps generated by a monocular geometry foundation model (UniK3D).
- A validity filter excludes unreliable predictions (e.g., non-finite values, distances $>20$ m).
- Radial MixForcing: A training-time teacher-substitution algorithm allows the model to accept externally supplied radial maps during inference. If an external map is valid, it replaces the predicted distribution; otherwise, the model falls back to its own prediction. This enables external geometry control without retraining.

Key Contributions

Depth-Aware Positional Encoding: CRePE introduces a positional encoding that considers not only ray direction but also possible scene positions along the ray, enabling depth-aware encoding for curved projected-path geometry under the Unified Camera Model.
Adapter-Based Implementation: The authors apply CRePE to frozen video DiTs via a Geometric Attention Adapter, incorporating token-wise scene distance into attention mechanisms and stabilizing predictions with pseudo radial-distance supervision.
External Geometry Control: By extending the positional-encoding pathway, CRePE supports external radial-map control for scene-geometry-conditioned generation and source-video motion transfer via Radial MixForcing.

Experimental Results

The authors evaluated CRePE against baselines including ReCamMaster (external trajectory control) and UCPE (unified-camera ray encoding) on the PanShot dataset, which covers pinhole, wide-angle, and fisheye settings.

Camera Control & Geometry: CRePE significantly outperforms baselines in camera controllability metrics (CamMC, rotation error, translation error) and lens distortion metrics ( $k_1, k_2$ ), particularly under wide-angle and fisheye conditions where pinhole assumptions fail.
Ablation Studies: Comparing CRePE against a RayRoPE-style endpoint PE baseline (which approximates the ray interval with only two endpoints), CRePE achieves a better overall average rank. This confirms that integrating over the curved projected path (via $K>2$ breakpoints) provides benefits beyond simple endpoint approximations.
Video Quality: While CRePE improves geometry-aware and perceptual metrics (VBench, aesthetic quality), it remains competitive on standard video-quality metrics (FVD, FID). In some full-video quality metrics (e.g., FVD, Inception Score), the ray-only baseline (UCPE) remains slightly stronger, suggesting a trade-off between strict geometric adherence and distributional video quality.
Layer Placement: Experiments show that applying CRePE to the middle 10 blocks of the DiT provides the best trade-off between geometric control and generation quality, outperforming applying it to all layers or only early/late layers.

Significance and Claims

The paper claims that CRePE represents a significant step toward unified camera control that is robust across diverse lens configurations, including non-pinhole lenses. By moving from ray-only encoding to curved-ray expectation, the method better captures the nonlinear projection geometry of wide-angle and fisheye cameras.

The authors position CRePE as a building block for geometry-aware, camera-controllable video generation. They emphasize that the method improves the model's ability to follow camera trajectories and maintain scene consistency under view changes without requiring source videos or reference images at inference. Furthermore, the extension to Radial MixForcing demonstrates a pathway toward unifying camera control and external scene-geometry control within a single adapter framework, enabling tasks like motion transfer and scene-layout manipulation via external radial maps.

The work is presented as a preprint, and the authors acknowledge limitations, such as the reliance on the quality of pseudo-supervision from monocular geometry models and the current focus on central-camera models (UCM) rather than rolling-shutter or non-central models.