💻 computer science

SCOPE: Scale-Consistent One-Pass Estimation of 3D Geometry

SCOPE is a novel one-pass method for estimating 3D geometry from extended monocular video sequences that introduces viewpoint-invariant alignment, appearance-invariant learning, and frequency-modulated positioning to achieve significant improvements in geometric accuracy and temporal consistency compared to state-of-the-art approaches.

Original authors: Zheng Zhang, Lihe Yang, Tianyu Yang, Chaohui Yu, Yixing Lao, Xiaoyang Guo, Biao Gong, Fan Wang, Hengshuang Zhao

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zheng Zhang, Lihe Yang, Tianyu Yang, Chaohui Yu, Yixing Lao, Xiaoyang Guo, Biao Gong, Fan Wang, Hengshuang Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect 3D model of a room just by watching a video of someone walking through it. The challenge isn't just seeing the walls; it's making sure that the wall you see in the first second of the video is the exact same size and shape as the wall you see 100 seconds later, even if the camera spins around, the lights change, or the person walks in and out of the frame.

Most current AI tools struggle with this. They are like a person trying to draw a map while walking: they get the details right for the spot they are standing on, but as they walk further, they start to get lost, forget how big the room really is, or accidentally stretch the walls. This is called "scale drift."

SCOPE (Scale-Consistent One-Pass Estimation of 3D Geometry) is a new AI method designed to solve this problem. Here is how it works, using simple analogies:

1. The "One-Pass" Superpower

Most video AI tools look at a video in small chunks, like watching a movie one scene at a time. If the scene changes, they might get confused about how big things are compared to the previous scene.

SCOPE is different. It watches the entire video in one single glance (a "single forward pass"). Think of it like a conductor who hears the entire symphony at once, rather than listening to one instrument at a time. This allows it to understand the whole story of the video instantly, ensuring that the size of a car in the first frame matches the size of that same car in the last frame, no matter how long the video is.

2. The "Shared Ruler" (Scale Consistency)

Imagine you are measuring a room with a tape measure. If you use a different tape measure for every wall, your final map will be a mess.

Old methods: They measure every frame with a slightly different, invisible ruler. By the end of the video, the "ruler" has stretched or shrunk, causing the 3D model to look warped or broken.
SCOPE: It uses one shared ruler for the entire video. It forces every single frame to agree on the same scale and position. This means if a table is 1 meter wide in the first second, it remains 1 meter wide in the 100th second, preventing the "drift" that ruins 3D models.

3. The "Time-Traveling Teacher" (Long-Range Consistency)

Usually, AI only checks if Frame 10 looks like Frame 11. If it makes a tiny mistake, that mistake gets bigger by Frame 20, and huge by Frame 100.
SCOPE uses a clever trick called hierarchical supervision. It's like a teacher who doesn't just check your homework for today, but also checks how your work from last week compares to today.

It looks at the video at different time speeds: comparing frames that are 1 second apart, 2 seconds apart, 4 seconds apart, and even 8 seconds apart.
This ensures the AI understands the "big picture" of how the scene moves over time, not just the immediate next step.

4. The "Stretchy Training" (Handling Long Videos)

Training an AI to watch a 10-minute video is hard because computers run out of memory. Usually, AI is trained on short clips (like 24 seconds) and then asked to guess what happens in a 10-minute video. This is like asking a student who only studied for 10 minutes to write a thesis.
SCOPE uses a technique called frequency-modulated positioning.

The Analogy: Imagine you are teaching a student to run a marathon, but you only have a 100-meter track. Instead of just running the 100 meters, you teach them to imagine the track is "stretched" out. You tell them, "If you run this 100 meters, imagine it represents 1,000 meters."
By simulating these "stretched" sequences during training, SCOPE learns how to handle videos that are much longer than the videos it was actually trained on, without getting confused or running out of memory.

5. The "Blindfold Test" (Appearance Invariance)

Sometimes the lighting changes, or a shadow moves, and the AI gets confused, thinking the object itself has changed.
SCOPE is trained with appearance-invariant learning. It's like training a student to recognize a friend even if they are wearing sunglasses, a hat, or standing in the dark. The AI is taught to ignore the changing colors and lights and focus only on the shape and structure of the objects. This keeps the 3D model stable even when the video gets dark or the camera spins wildly.

The Result

When the researchers tested SCOPE, they found it could build 3D models from video sequences that were hundreds of frames long with almost no "drift" or warping.

It reduced errors in 3D shape prediction by about 24%.
It reduced errors in keeping the video consistent over time by about 35%.
It did this much faster than other methods, processing 300 frames in less than 8 seconds.

In short, SCOPE is a new way for computers to watch a video and build a perfect, unbroken 3D map of the world inside it, ensuring that what they see at the beginning is exactly consistent with what they see at the end.

Technical Summary: SCOPE (Scale-Consistent One-Pass Estimation of 3D Geometry)

1. Problem Statement

Estimating 3D geometry from monocular video sequences presents a fundamental challenge in computer vision: balancing high geometric accuracy with long-range temporal consistency. Existing methods typically excel in one area at the expense of the other. Single-image approaches (e.g., MoGe) capture detailed geometry but produce inconsistent results when applied frame-by-frame, leading to scale drift. Conversely, video-specific methods often provide short-term consistency but lack geometric precision or fail to maintain global scale over hundreds of frames, particularly under complex camera trajectories and lighting variations. Furthermore, processing lengthy sequences simultaneously is often infeasible due to memory constraints, creating a tension between the need for precise temporal position encoding and the ability to extrapolate to sequences far longer than training examples.

2. Methodology

SCOPE proposes a novel approach to generate affine-invariant 3D point maps from RGB videos in a single forward pass. The method ensures that all frames in a sequence share the same scale and shift parameters, enabling consistent scale-invariant representations.

2.1 Architecture and Core Components

The model builds upon the MoGe architecture, utilizing a DINOv2-L visual encoder and a transformer-based decoder. To handle video, the authors insert temporal attention modules after each of the four decoder feature levels. The key innovations include:

Viewpoint-Invariant Geometry: To enforce geometric consistency, the method transforms points from individual camera coordinates into a common reference frame using camera poses (available during training). A multi-scale geometric loss is applied in this unified coordinate system, comparing structures across different viewpoints. This includes global alignment and local grid-based alignment (using grid sizes of 4 and 16) to capture both coarse structure and fine details.
Appearance-Invariant Learning: To ensure robustness against transient visual conditions (e.g., lighting changes, blur), the training process employs frame-specific augmentations with independently sampled color transformations and blur patterns. This forces the model to focus on invariant geometric features rather than transient appearance cues.
Frequency-Modulated Positional Extrapolation: To handle sequences significantly longer than the training context (e.g., 300+ frames vs. 24-frame training), SCOPE employs a specialized Rotary Position Encoding (RoPE) with Neural Tangent Kernel (NTK) adaptation.
- Dynamic NTK Scaling: The frequency components are scaled based on the ratio of inference sequence length to training length. This preserves the model's ability to capture fine-grained local details (high-frequency components) while amplifying adjustments for long-range dependencies (low-frequency components).
- Train-Time Sequence Stretching: During training, the model is exposed to "virtual" extended sequences. Position encodings are computed for a longer virtual sequence and sampled at intervals to match the actual input length. This simulates extrapolation, teaching the model to handle sequences orders of magnitude longer than its training data.

2.2 Loss Functions and Temporal Consistency

The training objective integrates per-frame geometric losses with cross-frame constraints:

Cross-Frame Geometric Loss ( $L_{cross}$ ): Enforces consistency by aligning point clouds from different frames into a shared reference frame, minimizing projection errors across global and local scales.
Hierarchical Temporal Supervision ( $L_{temp}$ ): Instead of relying solely on adjacent frame constraints (which lead to error accumulation), SCOPE introduces a derivative supervision framework. It penalizes the temporal derivative of depth values across exponentially increasing time intervals ( $\delta = 1, 2, 4, 8$ ). This structure-preserving constraint directly penalizes scale drift and ensures smooth, consistent predictions over extended durations.

2.3 Inference Strategy

SCOPE operates in a single-pass manner. Unlike sliding window approaches that process overlapping segments independently (leading to potential drift at boundaries), SCOPE processes the entire sequence at once using the frequency-modulated position encoding. This eliminates computational redundancy and maintains global context throughout the video.

3. Key Contributions

Scale-Consistent One-Pass Estimation: A unified framework that generates 3D point maps with shared scale and shift parameters across entire video sequences, eliminating the scale drift common in frame-by-frame or sliding-window methods.
Three Key Innovations:
- Viewpoint-invariant geometry alignment via multi-scale cross-frame losses.
- Appearance-invariant learning to isolate structural features from transient visual conditions.
- Frequency-modulated positioning (NTK-adapted RoPE with train-time stretching) enabling robust extrapolation to sequences vastly exceeding training lengths.
Efficient Long-Range Processing: The ability to process hundreds of frames in a single forward pass with minimal drift, achieving high computational efficiency compared to video diffusion or sliding-window baselines.

4. Experimental Results

Experiments were conducted on diverse datasets including Sintel, ScanNet, Bonn, KITTI, and DDAD, covering synthetic, indoor, and outdoor driving scenarios.

Quantitative Performance: SCOPE significantly outperforms state-of-the-art methods. On ScanNet, it reduces relative point map error by 24.2% and temporal alignment error (TAE) by 34.9% compared to existing approaches. It achieves the lowest average rank across all evaluated datasets.
Comparison with Metrics:
- Compared to MoGe-2, SCOPE achieves comparable point-map accuracy but significantly better temporal alignment.
- Compared to VGGT and $\pi^3$ , SCOPE offers superior temporal consistency and is more parameter-efficient (322M vs. 959M parameters for $\pi^3$ ).
- Compared to DepthCrafter and Video Depth Anything, SCOPE is 6–42× faster (39.1 FPS vs. <7 FPS) while maintaining competitive or superior accuracy.
Downstream Impact: When used as input for 4D reconstruction (via MegaSAM), SCOPE produces coherent, non-fragmented results. On the Sintel dataset, it improved Absolute Trajectory Error (ATE) by 55–60% and Relative Trajectory Error (RTE) by 58–63% compared to MoGe and MonST3R.
Ablation Studies:
- Extrapolation: The combination of NTK-adapted RoPE and sequence stretching (RoPE+) yields the best trade-off, significantly reducing TAE on long sequences (270 frames) compared to standard encodings.
- Loss Components: Both cross-frame geometric constraints and hierarchical temporal supervision are critical; their combination yields the best temporal consistency.
- Inference: Single-pass processing consistently outperforms sliding window approaches, reducing error by nearly 10% on KITTI.

5. Significance and Claims

The paper claims that SCOPE addresses the fundamental tension between geometric accuracy and temporal consistency in monocular video depth estimation. By enabling the processing of extended sequences in a single pass with minimal drift, SCOPE facilitates high-quality 3D and 4D reconstruction in challenging scenarios with complex camera motions and lighting variations.

The authors position SCOPE as a significant step forward for applications requiring long-range consistency, such as autonomous navigation, virtual reality, and novel view synthesis. While the current implementation relies on external tools (MegaSAM) for camera pose estimation during the evaluation pipeline, the paper notes that future work aims to estimate camera trajectories directly within the network. The method is presented as a robust, efficient, and scalable solution that bridges the gap between single-image precision and video-level coherence.