💻 computer science

Scaling Sequence-to-Sequence Generative Neural Rendering

Kaleido is a unified, decoder-only rectified flow transformer that treats 3D rendering as a sequence-to-sequence video synthesis task, enabling state-of-the-art, explicit-3D-free view synthesis with strong zero-shot performance by leveraging large-scale video pre-training.

Original authors: Shikun Liu, Kam Woh Ng, Wonbong Jang, Jiadong Guo, Junlin Han, Haozhe Liu, Yiannis Douratsos, Juan C. Pérez, Zijian Zhou, Chi Phung, Tao Xiang, Juan-Manuel Pérez-Rúa

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shikun Liu, Kam Woh Ng, Wonbong Jang, Jiadong Guo, Junlin Han, Haozhe Liu, Yiannis Douratsos, Juan C. Pérez, Zijian Zhou, Chi Phung, Tao Xiang, Juan-Manuel Pérez-Rúa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a single photo of a cat sitting on a windowsill. If you wanted to see what the cat looks like from behind, or from the side, a normal computer program would struggle because it only has that one flat picture. It doesn't "know" the cat has a 3D body; it only knows the pixels in that one image.

This paper introduces Kaleido, a new type of AI that solves this problem by changing how we think about 3D vision. Instead of trying to build a perfect, mathematical 3D model of the world (like a digital clay sculpture), Kaleido treats 3D vision as a video game.

Here is the simple breakdown of how it works, using everyday analogies:

1. The Big Idea: 3D is Just a Special Kind of Video

The authors argue that we don't need to teach computers complex geometry rules to understand 3D space. Instead, they realized that 3D is just a special type of video.

The Analogy: Think of a video as a sequence of frames playing over time. Kaleido treats a 3D scene as a sequence of images playing over space.
How it works: If you walk around a statue, the images you see change. Kaleido learns this pattern by watching millions of hours of regular videos. It learns that "if I move the camera left, the object shifts right." It doesn't need to know the math of a sphere; it just learns the visual pattern of how things look when you move around them.

2. The "Any-to-Any" Magic

Older AI models were like rigid robots: "I can only take 1 photo and make 5 new ones," or "I can only take 5 photos and make 1 new one."

Kaleido is like a chameleon. It can take any number of photos you give it (1, 5, or 50) and generate any number of new views you want, from any angle you choose.

The Metaphor: Imagine you have a magic sketchbook. If you show it one drawing of a house, it can instantly draw the back, the side, and the inside. If you show it ten drawings of the house from different angles, it can draw an even more perfect, detailed version of the house from a new angle. It doesn't matter how many photos you start with; it just adapts.

3. Learning from the "World's Library"

To get really good at this, Kaleido didn't just study 3D models (which are rare and hard to make). It read the entire library of the internet's video data.

The Analogy: Think of it like a child learning to speak. They don't study a grammar book first; they listen to thousands of hours of people talking. Kaleido "listened" to millions of hours of video. Because videos naturally show how objects look from different angles as the camera moves, Kaleido learned "visual common sense." It learned that a cup has a bottom, a top, and sides, just by seeing it move in videos.

4. The "Register" Fix (Stabilizing the Chaos)

When the researchers tried to make the AI bigger and smarter, it started to get "crazy." The numbers inside the computer's brain would get so huge they caused errors (like a calculator overflowing).

The Analogy: Imagine a crowded party where everyone is shouting. The noise gets so loud it breaks the speakers. The researchers found a few "mute buttons" (called registers) they could add to the system. These buttons didn't stop the conversation, but they absorbed the extra noise, keeping the party calm and the AI stable. This allowed them to build a much larger, more powerful model without it crashing.

5. What Can It Actually Do?

The paper shows that Kaleido is currently the best at its specific job:

It beats the experts: In tests where it had to guess what a scene looks like from a new angle, Kaleido performed better than other AI models, even when it was given very few starting photos.
It rivals the "slow" methods: Usually, to get a perfect 3D view, you have to spend hours tweaking a specific scene on a computer. Kaleido does this instantly, without any tweaking, and matches that high quality.
It builds 3D models: If you feed Kaleido a few photos of an object, it can generate so many perfect views that you can use them to build a real, rotatable 3D model of that object.

What It Can't Do Yet (The Limits)

The authors are honest about what Kaleido cannot do right now:

It's not a real-time video game engine yet: It takes time to generate the images, so you can't use it for instant, live-action gaming just yet.
It gets confused by extreme angles: If you ask it to look at something from a very weird, impossible angle, the result might look a bit "dream-like" or slightly wrong.
It doesn't handle zooming: It can move the camera around, but it can't currently simulate the camera lens zooming in and out (a "dolly zoom") while moving.

In summary: Kaleido is a new way of teaching computers to see in 3D. Instead of building a rigid 3D map, it learns to "imagine" new views by treating space like a video, using massive amounts of video data to learn how the world looks from every angle.

Technical Summary: Scaling Sequence-to-Sequence Generative Neural Rendering (Kaleido)

Problem Statement

The paper identifies two primary bottlenecks hindering the scalability of 3D vision and general-purpose rendering:

Fragmented 3D Representations: The field lacks a consensus on a unified 3D representation, with methods split between explicit structures (voxels, point clouds) and implicit fields (NeRFs). This fragmentation prevents the collective scaling of architectures for a single representation.
High Cost of 3D Data: Creating 3D datasets is slow and expensive due to the requirement for strict 3D consistency, often necessitating manual mesh creation or complex multi-view camera labeling.

The authors argue that 3D perception is fundamentally a form of "visual common sense" derived from extensive observation rather than a strict geometric problem. Consequently, they propose that an ideal rendering system should learn implicit representations by capturing statistical patterns from large-scale visual data, rather than explicitly modeling perfect geometric consistency.

Methodology: The Kaleido Framework

Core Hypothesis:
The paper posits that 3D can be treated as a specialized sub-domain of video. Just as large language models acquire coding capabilities from general text corpora, a general-purpose rendering model can be created by treating 3D as a structured sub-domain of video data.

Unified Sequence-to-Sequence Formulation:
Kaleido reformulates rendering as a sequence-to-sequence image synthesis task. It models the conditional distribution of target views ( $X_T$ ) given reference views ( $X_R$ ) and their corresponding 6-DoF camera poses ( $P_R, P_T$ ). This "any-to-any" view prediction is handled via a masked autoregressive framework.

Architectural Innovations:

Decoder-Only Rectified Flow Transformer: Kaleido adopts a scalable transformer architecture inspired by Diffusion Transformers (DiT) and Llama-3, operating within a rectified flow objective. It uses a masked autoregressive framework to generate image tokens.
Unified Positional Encoding (GTA): A key innovation is a parameter-free encoding scheme extending Rotary Positional Encoding (RoPE) and Geometric Transformation Attention (GTA). This design creates a unified representation for:
- 2D spatial positions (pixel coordinates).
- 3D camera poses (6-DoF extrinsics).
- Temporal positions (frame indices).
  This allows the same architecture to process both multi-view 3D data and video data without modification.
Windowed Cross-View Attention: To improve feature exchange between views while maintaining efficiency, the model employs a temporal window attention mechanism where query tokens attend to a local $K \times K$ window across frames, rather than a single token.
Auxiliary Visual Features: The model integrates features from pre-trained DINOv2 encoders to enhance depth estimation and 3D perception, particularly for in-the-wild images.

Training Strategies and Stability:

Two-Stage Training: The model is first pre-trained on large-scale, unstructured video data (34M clips) to build a robust world representation. It is then fine-tuned on structured multi-view 3D datasets (Objaverse, uCO3D, RealEstate10K, DL3DV) with progressive resolution scaling (256px $\to$ 1024px).
Handling Massive Activations: The authors observed severe training instability in rectified flow transformers due to "massive activations" (values reaching 15k–24k). They resolve this by appending learnable "register" tokens to the keys and values in attention layers, which stabilizes activation magnitudes without degrading performance.
Noise-Biased Sampling: Standard logit-normal sampling is found suboptimal for rendering. The authors introduce a "Mode" sampler with a shift factor that biases training and inference toward high-noise timesteps ( $t \approx 1$ ), where initial scene structure is formed. This is crucial for the highly constrained solution space of image-to-3D rendering.
View Sampling: An exponential distribution is used to sample the number of reference views, prioritizing challenging, low-view scenarios during training to improve generalization.

Key Contributions

Kaleido Family of Spatial Generative Models (SGMs): A unified model capable of performing object- and scene-level view synthesis from any number of reference views to any number of target views with full 6-DoF camera control.
Unified Architecture: A decoder-only rectified flow transformer that seamlessly processes 3D and video data using a single positional encoding design, eliminating the need for 3D-specific architectural modifications (e.g., Plücker ray encodings).
Scaling Recipe: A comprehensive scaling strategy involving architectural improvements (SwiGLU, GQA), stability techniques (register tokens), and tailored sampling strategies (noise-biased SNR samplers).
Video Pre-training for 3D: Demonstrating that pre-training on large-scale video data significantly improves the efficiency and performance of subsequent 3D fine-tuning, reducing reliance on scarce, camera-labeled 3D datasets.

Results

Novel View Synthesis (NVS):

Kaleido achieves state-of-the-art zero-shot performance across object-level (OmniObject3D, GSO-30, RTMV) and scene-level (LLFF, Mip-NeRF 360, Tanks and Temples) benchmarks.
Few-View Settings: Kaleido-Small (0.6B parameters) matches or outperforms larger baselines (SV3D, EscherNet) despite having fewer parameters.
Many-View Settings: Kaleido is the first zero-shot generative model to match the rendering quality of per-scene optimization methods (e.g., Instant-NGP, 3D Gaussian Splatting) when provided with many reference views. On LLFF and Mip-NeRF 360, it surpasses Instant-NGP.

Camera-Conditioned Video Generation:

Evaluated on DL3DV-140 and Tanks and Temples, Kaleido significantly outperforms specialized camera-conditioned video models (Wonderland, VD3D, ViewCrafter) in camera guidance precision (Rotation and Translation error), demonstrating superior geometric consistency while maintaining visual quality.

3D Reconstruction:

When used as a view synthesizer for off-the-shelf reconstruction frameworks (NeuS2), Kaleido achieves state-of-the-art results on the GSO-30 dataset.
It demonstrates high view efficiency, achieving better reconstruction quality with just 2 reference views than EscherNet achieves with 10 views. High-resolution (1024px) generations yield meshes with intricate, sharp textures.

Significance and Claims

The paper claims that Kaleido represents a significant step toward a universal, general-purpose rendering engine. By unifying 3D and video modeling under a single sequence-to-sequence framework, it overcomes the fragmentation of 3D representations and the scarcity of 3D data.

Key claims regarding significance include:

Paradigm Shift: Treating 3D as a specialized sub-domain of video allows the model to inherit spatial perception and visual common sense from massive video datasets, similar to how LLMs learn from text corpora.
Zero-Shot Capability: For the first time, a generative model matches the quality of per-scene optimization methods in many-view settings without any per-scene training or architectural changes.
Scalability: The model exhibits strong positive scaling properties, where performance consistently improves with increased model size and data.

Limitations Acknowledged:
The authors note that Kaleido currently does not model camera intrinsics (preventing dolly-zoom effects), can exhibit texture flickering or sticking in extreme low-view/high-frequency scenarios, and may lack semantic plausibility under extreme viewpoint changes. Additionally, inference speed is not yet real-time. Future work is proposed to address these via intrinsic parameterization, integration of text-to-image priors, and inference acceleration techniques.