💻 computer science

Camera Control for Text-to-Image Generation via Learning Viewpoint Tokens

This paper introduces a framework for precise camera control in text-to-image generation by learning parametric viewpoint tokens on a hybrid dataset of 3D-rendered and photorealistic images, achieving state-of-the-art geometric accuracy and generalization to unseen object categories while preserving image quality and prompt fidelity.

Original authors: Xinxuan Lu, Charless Fowlkes, Alexander C. Berg

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinxuan Lu, Charless Fowlkes, Alexander C. Berg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director trying to tell a magical AI painter exactly how to paint a picture. You say, "Paint a red sports car." The AI paints a beautiful car, but it's always facing you, head-on, like a mugshot.

You try to be more specific: "Paint it from the side, maybe looking down from a hill." But the AI gets confused. It might paint the car from the front again, or it might paint a car that looks like it's floating in space with no ground, or it might just ignore your angle instructions entirely.

This is the problem the paper "Camera Control for Text-to-Image Generation via Learning Viewpoint Tokens" tries to solve.

Here is the simple breakdown of how they fixed it, using some everyday analogies.

1. The Problem: The "Vague Artist"

Current AI image generators are like incredibly talented artists who have seen millions of photos, but they are terrible at following geometric directions.

The Issue: If you ask for a "30-degree left view," the AI doesn't really understand what "30 degrees" means in 3D space. It just guesses based on what it thinks looks "cool" or "normal."
The Result: You get a picture of a car, but it's facing the wrong way, or the background doesn't match the angle of the car.

2. The Solution: The "GPS Token"

The authors realized that natural language (words) is too fuzzy for precise angles. You can't say "rotate 42.5 degrees" easily in a sentence without the AI getting lost.

So, they invented a "Viewpoint Token."

The Analogy: Imagine the AI's brain is a giant library of images. When you type a prompt, you usually just hand the librarian a note saying "Red Car."
The Fix: The authors added a special GPS coordinate tag (the token) to that note. It's like handing the librarian a note that says: "Red Car, but please look at it from the North-West corner, 20 feet away, and tilt your head down slightly."
How it works: They turned math numbers (like angle, distance, and tilt) into a special code (a "token") that the AI can read just like a word. This code tells the AI exactly where the "camera" is standing.

3. The Training: The "Two-Part Diet"

To teach the AI to understand these GPS tokens, they had to feed it a very specific diet. If they only used one type of data, the AI would get "sick" (it would forget how to paint realistic scenes).

They created a Two-Part Dataset:

Part 1: The "Blueprint" (3D Renderings):
- Think of this as training the AI with perfect, computer-generated 3D models. These are like architectural blueprints. They are perfect at showing angles and geometry, but they look a bit fake and plastic.
- Why it helps: It teaches the AI the math of angles. "Okay, if the number says 90 degrees, the car must be sideways."
Part 2: The "Real World" (Photorealistic Augmentation):
- This is the tricky part. They took those 3D models and used another AI (like a photo editor) to paste them into real, messy, beautiful backgrounds (like a street in Venice or a forest).
- Why it helps: This teaches the AI realism. It learns that a car on a street has shadows, reflections, and a horizon line that matches the angle.

By mixing the "Blueprints" (for math) with the "Real World" (for beauty), the AI learns to be both geometrically accurate and visually stunning.

4. The Magic Trick: "Factorized" Learning

Previous methods tried to memorize specific objects. If you trained them on lions, they could show you a lion from the side, but if you asked for a "side view of a toaster," they would fail because they hadn't memorized toasters.

This new method is smarter. It learned general rules.

The Analogy: Instead of memorizing "How a Lion looks from the left," the AI learned "How any object looks from the left."
It figured out that "Left" means the same thing for a lion, a car, a unicorn, or a toaster. This allows it to generate views of things it has never seen before (like a mythical Phoenix or a specific type of robot) and get the angle right.

5. The Results

When they tested this new system:

Accuracy: It followed the camera angles much better than previous methods. If you asked for a view from above, it actually looked from above.
Consistency: The background matched the object. If the camera was looking down, the ground looked right.
Flexibility: It worked on things it had never seen in training (like a "Santa Claus" or a "Dolphin"), proving it learned the concept of a camera, not just the concept of specific animals.

Summary

Think of this paper as giving the AI a virtual camera rig that it can control with a remote. Before, the AI was like a painter who could only paint what was directly in front of them. Now, thanks to these "Viewpoint Tokens," the AI can spin around the object, zoom in, look down from a bird's eye view, or peek from behind, all while keeping the picture looking realistic and following your instructions perfectly.

1. Problem Statement

Current state-of-the-art text-to-image (T2I) models struggle to interpret natural language instructions regarding camera viewpoints. While models excel at semantic fidelity (e.g., "a red car"), they fail at geometric precision (e.g., "30° left-side view" or "45° top-down").

Ambiguity: Natural language is discrete and ambiguous for specifying continuous 3D camera parameters.
Failure Modes: Models often hallucinate incorrect poses, collapse to biased canonical angles (usually front-facing), or produce inconsistent geometry across trials.
Limitations of Prior Work:
- 3D-aware models (e.g., NeRF-based) often require additional inputs (depth maps, reference images) beyond text, limiting usability.
- Token-based methods (e.g., Compass Control) often overfit to specific object appearances or fail to generalize to unseen object categories because they learn object-specific correlations rather than factorized geometric representations.

2. Methodology

The authors propose a framework that augments text prompts with learnable parametric viewpoint tokens, enabling precise camera control without requiring additional geometric inputs (like depth maps) at inference.

A. Viewpoint Parameterization

The camera is parameterized relative to the object using a 5-factorized representation ( $\theta$ ):

Azimuth ( $\theta_{az}$ ): Horizontal rotation around the object.
Elevation ( $\theta_{el}$ ): Vertical angle relative to the horizon.
Radius ( $r$ ): Distance from the object (normalized by object diameter).
Pitch ( $\theta_{pitch}$ ): Camera tilt up/down.
Yaw ( $\theta_{yaw}$ ): Camera tilt left/right.

Coordinate System: The object is fixed at the origin with its front facing the positive x-axis, ensuring consistent "left/right" and "front/back" definitions across all object categories.

B. Viewpoint Token Encoding

Encoding: The 5 parameters are encoded into a vector. Azimuth is encoded using sine/cosine to handle periodicity; radius is normalized; pitch/yaw/elevation are used as radians.
MLP Encoder: A lightweight 3-layer Multi-Layer Perceptron (MLP) maps these encoded parameters to a viewpoint token embedding ( $\mathbf{e}_{view}$ ).
Integration: This token is concatenated with the text embeddings and fed into the T2I backbone (e.g., Harmon, Stable Diffusion). The model is fine-tuned to jointly process semantic text and explicit geometric tokens.

C. Dataset Construction (Two-Part Strategy)

To prevent overfitting to 3D renderings while maintaining geometric supervision, the authors curate a hybrid dataset:

Large Rendered Dataset (Geometric Supervision):
- 3,111 objects from TexVerse (animals, vehicles, people, furniture).
- Rendered at 120 random viewpoints each (~373k images).
- Provides dense, perfect geometric ground truth.
Photorealistic Augmented Dataset (Appearance Diversity):
- 800 high-quality objects rendered at 20 viewpoints.
- Processed using Nano Banana (Gemini 2.5 Flash Image) to replace the 3D background and object texture with photorealistic scenes while preserving the exact camera pose.
- Includes diverse object descriptions and background prompts.
- Result: ~6.6k images.

Training Strategy: The model is trained by sampling equally from both datasets. This forces the model to learn the relationship between the token and the view, rather than memorizing the specific 3D mesh appearance.

3. Key Contributions

Parametric Viewpoint Tokens: Introduces a method to embed explicit 3D camera parameters directly into the text embedding space, achieving state-of-the-art accuracy in azimuth, elevation, radius, pitch, and yaw.
Factorized Generalization: Unlike prior methods that overfit to specific object shapes, the learned tokens represent factorized geometric information. This allows the model to generalize to unseen object categories (e.g., mythical creatures, specific vehicle types) without retraining.
Global Scene Understanding: By training on full scenes (foreground + background) rather than isolated object cutouts, the method learns consistent foreground-background relationships and global scene geometry.
Two-Part Dataset Design: A novel training regime combining high-volume 3D renders (for geometry) and low-volume photorealistic augmentations (for realism) to balance geometric precision with visual fidelity.

4. Experimental Results

The method was evaluated on the Harmon backbone (and tested on SD2.1/SD3.5) against baselines like ControlNet-Depth, Stable-Virtual-Camera, and Compass Control.

Quantitative Performance:
- Viewpoint Accuracy: Achieved the lowest mean angular errors across all 5 parameters. For example, Azimuth error was 18.11° (vs. 31.07° for Compass Control and 54.89° for SV-Camera).
- Generalization: On "diverse" objects (unseen during training), the method maintained low error (19.06°), whereas Compass Control's error jumped significantly (37.29°), indicating severe overfitting.
- Prompt Fidelity: Achieved higher CLIP similarity scores and GenEval benchmarks than Compass Control, proving that adding viewpoint tokens does not degrade the model's ability to follow text prompts.
Qualitative Results:
- Successfully generated complex views (e.g., high elevation, back views) where other methods failed or produced distorted shapes.
- Demonstrated robustness on non-existent objects (e.g., "car made of vines," "origami elephant") while maintaining correct viewpoint.
Ablation Studies:
- Removing the rendered dataset caused a significant drop in accuracy, confirming the necessity of geometric supervision.
- Fine-tuning the backbone (LLM/MAR) was crucial; freezing it led to poor performance, suggesting the pre-trained text space lacks 3D geometry awareness.
- The specific factorized encoding outperformed Plücker rays and 12D matrices.

5. Significance and Limitations

Significance:

Bridging the Gap: The work bridges the gap between 3D-aware generation (which usually needs reference images) and text-to-image generation (which lacks geometric control).
Scalability: The approach is backbone-agnostic and works across different T2I models.
Future Pathway: It demonstrates that text-vision latent spaces can be endowed with explicit 3D structure, paving the way for "geometrically-aware" prompts.

Limitations:

Prior Bias: The model still inherits a bias from T2I backbones toward eye-level, centered horizons, particularly for famous landmarks (e.g., Taj Mahal).
Extreme Viewpoints: The current dataset covers elevation up to 45° and excludes roll rotations. Performance degrades at extreme elevations (>75°) or with roll, partly due to the difficulty of synthesizing reliable photorealistic data for these rare views using current image editing tools.
Fine Details: Occasional failures in human faces and fine structural details under extreme angles.

In conclusion, this paper presents a robust, generalizable solution for precise camera control in T2I generation by learning factorized viewpoint tokens, moving beyond object-specific correlations to achieve true geometric understanding.