💻 computer science

Lighting-grounded Video Generation with Renderer-based Agent Reasoning

The paper introduces LiVER, a diffusion-based framework that achieves state-of-the-art, scene-controllable video generation by disentangling 3D properties like layout, lighting, and camera trajectory through a renderer-based agent and a novel large-scale dataset.

Original authors: Ziqi Cai, Taoyu Yang, Zheng Chang, Si Li, Han Jiang, Shuchen Weng, Boxin Shi

Published 2026-04-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ziqi Cai, Taoyu Yang, Zheng Chang, Si Li, Han Jiang, Shuchen Weng, Boxin Shi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a movie director. You want to film a scene where a glass skyscraper reflects a golden sunset, and the camera slowly circles around it.

In the world of old video AI, you would just type: "A glass skyscraper at sunset." The AI would guess the rest. Sometimes it gets it right, but often the sun might be in the wrong place, the reflections might look like plastic, or the camera might suddenly teleport. It's like asking a painter who has never seen the sun to guess what a sunset looks like.

LiVER (Lighting-grounded Video Generation) is like giving that AI a virtual film set, a physics textbook, and a smart assistant all in one.

Here is how it works, broken down into simple steps:

1. The "Smart Assistant" (The Renderer-based Agent)

Instead of just guessing, LiVER has a "Scene Agent." Think of this agent as a digital production designer.

You tell it your idea: "A modern building with a camera circling it."
The Agent doesn't just write a script; it actually builds a rough 3D model of the scene. It places the building, sets up the lights, and plans the camera's path.
It's like the agent is saying, "Okay, I've built a rough cardboard model of your scene and set up a lamp to mimic the sunset. Now, let's make it look real."

2. The "Physics Cheat Sheet" (The Scene Proxy)

This is the paper's secret sauce. Most AI just looks at pictures. LiVER looks at how light actually behaves.

The agent takes that 3D model and runs it through a "physics engine" (like the software used to make video games).
It creates a special "cheat sheet" called a Scene Proxy. This isn't just a video; it's a stack of layers showing:
- Diffuse: The basic color (like a flat painting).
- Rough: How matte or shiny the surface is (like sandpaper vs. glass).
- Glossy: The sharp, bright reflections (like a mirror).
Think of this as giving the AI a mathematical map of the light. It knows exactly where the shadow should fall and how the glass should reflect the sun, because it calculated it using real physics, not just by guessing.

3. The "Magic Artist" (The Video Diffusion Model)

Now comes the artist. This is the part of the AI that actually draws the pretty pictures.

Usually, this artist is very good at making things look real but is bad at following strict rules.
LiVER hands the artist the "Physics Cheat Sheet" (the Scene Proxy) and says, "Paint this, but make sure the light hits the glass exactly like this map says."
Because the artist has the cheat sheet, the final video looks photorealistic. The shadows move correctly, the reflections shimmer, and the camera moves smoothly around the building.

Why is this a big deal?

No More "Glitchy" Light: In other AI videos, if you move the camera, the shadows might stay stuck in one place or the reflection might disappear. In LiVER, the light behaves like it does in the real world. If you move the camera, the light moves with it.
Total Control: You can change the lighting after the scene is set. Want to turn the sunset into a stormy night? You just swap the "lighting map," and the AI re-renders the video with the new mood, keeping the building and camera movement exactly the same.
The "Editable" Set: Because the AI builds a 3D model first, you can actually go in and move a chair or change a wall color before the video is made, just like in a real movie studio.

The "Training" (How it learned)

To teach this AI, the researchers didn't just show it movies. They created a giant library of two types of videos:

Real Life: They took real videos and reverse-engineered them to figure out the 3D shapes and lighting.
Synthetic Life: They built thousands of fake scenes in a computer with crazy, perfect lighting conditions (like a sun spinning around a room) to teach the AI how light should behave.

They trained the AI in three stages, like learning to ride a bike:

Training Wheels: First, they taught the AI how to read the "Physics Cheat Sheet."
Balancing: Then, they let it practice drawing the video while holding the sheet.
The Race: Finally, they let it run wild with both real and fake data to make sure it can handle any lighting situation.

The Bottom Line

LiVER is like upgrading from a magic 8-ball (which gives you random, often wrong answers) to a professional film studio (where you have control over the set, the lights, and the camera). It allows creators to make videos that aren't just "pretty," but are physically accurate and perfectly controllable.

1. Problem Statement

While diffusion models have achieved significant progress in video generation, they suffer from limited controllability over critical physical scene factors. Specifically:

Entanglement: Key elements like layout, lighting, and camera trajectories are often entangled or weakly modeled.
Physical Inaccuracy: Existing methods struggle to simulate complex physical interactions, particularly physically accurate lighting (e.g., shadows, reflections, ambient occlusion) on realistic materials (skin, metal, glass).
Lack of Explicit Control: Current approaches often rely on text prompts or 2D proxies, which fail to provide the explicit 3D geometric and lighting constraints required for filmmaking and virtual production.

2. Methodology

The authors propose LiVER, a framework that generates photorealistic videos by conditioning a diffusion model on explicit 3D scene properties derived from a renderer-based agent.

A. Core Framework Components

Renderer-based Agent:
- Function: Translates high-level user text instructions into structured 3D control signals.
- Process:
  - Scene Building: Parses text to create a scene graph (objects, spatial relations) and retrieves 3D assets.
  - Lighting Setup: Selects or generates HDR environment maps based on mood descriptions (e.g., "overcast," "warm").
  - Camera Planning: Infers camera trajectories (e.g., "orbit," "dolly zoom") and generates smooth paths.
- Output: A unified 3D representation comprising geometry, lighting, and camera trajectory.
Lighting-grounded Scene Proxy:
- Instead of feeding raw 3D meshes directly, the framework uses a 3D engine (Blender) to render a stack of 2D image sequences (passes) that serve as the conditioning input.
- The Proxy ( $y$ ): Consists of three channels stacked together:
  - Diffuse: Captures low-frequency ambient lighting.
  - Rough GGX: Captures medium-frequency broad reflections.
  - Glossy GGX: Captures high-frequency specular highlights.
- This proxy preserves physically realistic lighting cues while providing layout information.
Video Generation Backbone & Integration:
- Base Model: Built upon Wan2.2-5B, a state-of-the-art text-to-video diffusion model.
- Proxy Encoder: A lightweight 2D convolutional network encodes the scene proxy into a compact feature representation.
- Adapter: A conditioning module injects these features into the video latent space. It learns a spatial residual added to the latent code ( $z' = z + \alpha \cdot z_y$ ), ensuring the proxy guides the generation without disrupting the base model's priors.

B. Dataset: LiVERSet

To train and evaluate the model, the authors constructed LiVERSet, a large-scale dataset (11K videos) with dense annotations:

LiVER-Real: Real-world videos annotated with reconstructed 3D geometry, HDR environment maps, and camera poses (using tools like VGGT, Grounding-DINO, and DiffusionLight-Turbo).
LiVER-Syn: Synthetic videos rendered with diverse, procedurally generated lighting (rotating HDR maps) and dynamic camera movements to ensure broad lighting variability.
Annotations: Includes scene geometry, environment maps, camera poses, and text captions.

C. Training Strategy

A three-stage training scheme is employed to ensure stable convergence:

Conditional Pathway Training: Freezes the video backbone; trains only the proxy encoder and adapter to learn the mapping from proxy to control signals.
Joint LoRA Fine-tuning: Unfreezes LoRA layers in the backbone; jointly fine-tunes the backbone and conditioning modules to refine semantic alignment.
Lighting Diversity Expansion: Mixes real and synthetic data (1:1 ratio) to enhance the model's ability to handle diverse lighting phenomena.

3. Key Contributions

LiVER Framework: A novel approach for lighting-grounded video generation that disentangles layout, lighting, and camera control using a unified 3D representation.
Renderer-based Agent: An automated system that translates text prompts into structured 3D scene graphs, lighting setups, and camera trajectories, bridging the gap between user intent and physical simulation.
Lighting-aware Dataset (LiVERSet): A comprehensive dataset containing over 11K videos with dense physical annotations (HDR maps, geometry, camera poses) for both real and synthetic data.
Physically Accurate Rendering: The use of PBR (Physically Based Rendering) passes (Diffuse, GGX) as conditioning signals allows the model to generate realistic material responses (shadows, reflections) that previous methods could not achieve.

4. Experimental Results

The method was evaluated against state-of-the-art baselines (CameraCtrl, MotionCtrl, VideoFrom3D) on the LiVER-Real test set.

Quantitative Performance:
- Quality: Achieved the lowest FVD (32.56) and FID (129.56), indicating superior video quality and realism.
- Control Fidelity: Demonstrated the lowest errors in Camera Pose (ATE, RPE) and Lighting estimation (SI-MSE), and the highest mIoU (0.87) for layout preservation.
- CLIP Score: Highest semantic alignment with text prompts (30.97).
Qualitative Evaluation:
- Visual comparisons showed LiVER produces more realistic lighting effects (shadows, reflections) and adheres more strictly to camera trajectories and object layouts than baselines.
User Study:
- In a study with 25 participants, LiVER was preferred over all baselines in Video Quality (83.4%), Scene Control (83.3%), Camera Control (72.1%), and Lighting Control (59.3%).
Ablation Studies:
- Removing synthetic data resulted in incorrect, flat illumination, proving the necessity of diverse synthetic lighting for training.
- Removing the staged training led to degraded performance and static outputs, highlighting the importance of progressive learning.

5. Significance and Impact

Bridging Generative AI and Virtual Production: LiVER moves video generation from "black-box" text-to-video to a controllable, physics-aware pipeline. It enables workflows similar to traditional CGI where lighting and camera can be explicitly edited.
Disentanglement: It successfully separates lighting from geometry and camera motion, allowing users to change lighting conditions (e.g., time of day) without altering the scene structure.
Usability: The inclusion of an agent that automates the creation of 3D proxies lowers the barrier to entry, allowing users to generate complex, physically grounded videos via simple text prompts while retaining the ability to manually edit the 3D scene in standard software.
Future Applications: The framework sets a new standard for applications in filmmaking, advertising, and immersive content creation where precise control over physical realism is essential.