🤖 AI

LumiVideo: An Intelligent Agentic System for Video Color Grading

LumiVideo is an intelligent agentic system that mimics professional colorists' cognitive workflows to autonomously generate cinematic color grades from raw log footage by synthesizing LLM reasoning with retrieval-augmented generation, outputting industry-standard ASC-CDL and 3D LUT configurations for interpretable, temporally consistent, and iteratively controllable results.

Original authors: Yuchen Guo, Junli Gong, Hongmin Cai, Yiu-ming Cheung, Weifeng Su

Published 2026-04-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuchen Guo, Junli Gong, Hongmin Cai, Yiu-ming Cheung, Weifeng Su

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a raw, uncooked meal. It's technically edible, but it looks gray, flat, and unappetizing. Video color grading is the art of cooking that meal: adding spices, adjusting the heat, and plating it so it looks delicious and tells a story.

For decades, this "cooking" has been done by human chefs (colorists) using complex, expensive software. But now, a new system called LumiVideo has arrived. It's not just a robot that tries to guess what the final picture should look like; it's a smart, thinking assistant that learns the rules of cooking and gives you the exact recipe to follow.

Here is how LumiVideo works, broken down into simple steps with some fun analogies:

1. The Problem: The "Black Box" Mistake

Most current AI tools try to fix video by redrawing the pixels (like a painter trying to repaint a photo from scratch).

The Analogy: Imagine trying to fix a blurry photo by asking an artist to draw a new one based on a description. The result often looks weird, flickers like a strobe light, and you can't change the artist's mind later without starting over. It's a "black box"—you put a video in, and a weird video comes out, but you have no idea how it was made or how to tweak it.

2. The LumiVideo Solution: The "Smart Sous-Chef"

LumiVideo doesn't redraw the picture. Instead, it acts like a master chef's sous-chef who understands the ingredients and the recipe. It doesn't touch the pixels directly; it calculates the perfect settings (the recipe) to apply to the video.

It works in four stages, just like a human professional:

Step 1: Perception (The "Tasting")

Before cooking, you have to taste the raw ingredients.

What it does: LumiVideo looks at the flat, gray "Log" video (the raw footage) and asks two questions:
1. Physical Stream: "How bright is it? Is it dark like a cave or bright like noon?" (It measures the light scientifically).
2. Semantic Stream: "What is in the scene? Is it a sunset? Are there people's faces? Is it a blue ocean?" (It uses AI to understand the story).
The Analogy: It's like a sommelier tasting a wine to know if it's too acidic or too sweet before deciding what food to pair it with.

Step 2: Reasoning (The "Menu Planning")

This is the brain of the operation. Instead of guessing, LumiVideo uses a Tree of Thoughts.

What it does: It thinks, "If I make the shadows darker, will the faces look too dark? If I make the sunset warmer, will the sky turn orange?" It branches out into different possibilities, checks them against a library of professional rules (RAG), and picks the best path.
The Analogy: Imagine a chess player looking three moves ahead. "If I move here, the opponent will do that. If I move there, I win." LumiVideo tries out three different "recipes" in its head, picks the winner, and discards the bad ones.

Step 3: Execution (The "Recipe Card")

Once it decides on the best look, it doesn't paint the picture. It writes a recipe card.

What it does: It outputs standard industry settings (called ASC-CDL and 3D LUTs). These are mathematical instructions like "Make the shadows 10% bluer" or "Make the highlights 5% warmer."
The Analogy: Instead of the AI painting the cake, it hands you a perfect recipe card. You can take that card to any professional kitchen (like DaVinci Resolve software), and it will work perfectly. Because it's just math, the video never flickers or glitches.

Step 4: Reflection (The "Taste Test & Tweak")

This is the magic part. In the old days, if you didn't like the AI's result, you had to delete it and start over.

What it does: You can talk to LumiVideo. If you say, "The shadows feel too muddy, make them cooler," it understands exactly what to change. It adjusts only the shadow settings and leaves everything else (like the sky or the skin tones) exactly as they were.
The Analogy: Imagine you are at a restaurant. You tell the waiter, "The soup is a bit salty." A normal AI would throw the soup away and make a new bowl. LumiVideo is like a chef who just adds a pinch of water to fix the saltiness, keeping the rest of the soup perfect.

Why is this a Big Deal?

It's Transparent: You aren't stuck with a weird AI result. You get a recipe you can edit, tweak, and understand.
It's Consistent: Because it uses math instead of "drawing," the video looks smooth and stable, frame after frame.
It's a New Benchmark: The authors also created LumiGrade, the first "test kitchen" full of raw, ungraded videos to see how well AI can cook.

In a nutshell: LumiVideo is the first AI that doesn't just "guess" what a movie should look like. It acts like a professional colorist: it tastes the scene, plans the recipe, calculates the math, and lets you tweak the seasoning until it's perfect.

1. Problem Statement

Video color grading is a critical, labor-intensive post-production process that transforms flat, log-encoded raw footage into emotionally resonant cinematic visuals. Current automated approaches suffer from three fundamental flaws:

Black-Box Nature: Existing methods (e.g., diffusion models, pixel-to-pixel translation) output raw pixels directly, lacking mathematical interpretability and preventing professional editors from understanding or modifying the underlying logic.
Temporal Inconsistency: Frame-by-frame generation often introduces flickering, spatial artifacts, and structural degradation, rendering the output unusable for video.
Lack of Iterative Control: Static executors cannot support the iterative, state-aware refinement workflow required by professional colorists (e.g., "make shadows cooler but keep highlights warm").

Furthermore, there is a lack of public benchmarks for automated grading, as existing datasets consist of already graded sRGB clips rather than raw log footage.

2. Methodology: The LumiVideo Framework

LumiVideo reframes color grading from a pixel-generation task to a transparent, parameter-driven reasoning process. It operates as an autonomous agent mimicking the cognitive workflow of human colorists through four distinct stages: Perception, Reasoning, Execution, and Reflection.

A. Parameter Space Selection

Instead of generating RGB arrays, LumiVideo outputs industry-standard ASC-CDL (American Society of Cinematographers Color Decision List) parameters:

Lift, Gamma, Gain (L, G, G): RGB channel adjustments.
Saturation, Contrast, Pivot (S, C, V): Global adjustments.
These parameters are compiled into a globally consistent 3D Look-Up Table (LUT), ensuring zero structural degradation and seamless integration with Non-Linear Editors (NLEs) like DaVinci Resolve.

B. Stage 1: Perception (Dual-Stream Module)

To handle the flat, desaturated nature of log-encoded video (which often causes visual hallucinations in VLMs), the system uses a dual-stream approach:

Physical Stream: Applies deterministic Color Space Transforms (CST) to map log footage to Rec.709. It calculates objective exposure profiles (black/white points, mid-gray) and luminance percentiles.
Semantic Stream: A Vision-Language Model (VLM) analyzes the normalized frame to extract high-level context: lighting conditions (e.g., "golden hour"), narrative genre, scene subjects, and protected tone regions (e.g., skin tones, sky hues) defined as hard constraints.

C. Stage 2: Reasoning (Tree of Thoughts + RAG)

The core decision-making engine utilizes a Large Language Model (LLM) synergized with Retrieval-Augmented Generation (RAG) and a Tree of Thoughts (ToT) search strategy:

RAG: Retrieves professional heuristics (e.g., "Golden Hour Warmth" rules) from a curated database based on the perceived scene state.
Tree of Thoughts: The LLM explores multiple grading hypotheses (branches) simultaneously. It generates candidate parameter sets, evaluates them using a VLM Critic (which scores cinematic quality and constraint adherence), and prunes suboptimal paths via Beam Search.
Outcome: This systematic exploration navigates the non-linear ASC-CDL space to deduce the optimal base grade without user input.

D. Stage 3: Execution

The selected parameters are compiled into a mathematically rigorous 3D LUT.

Adaptive Lift: Introduces a modified lift formulation to prevent highlight clipping in HDR content.
Exponential Roll-off: Emulates analog film shoulder response to preserve texture in bright regions.
The final output is a .cube file compatible with professional NLEs.

E. Stage 4: Reflection (State-Aware Iteration)

To support human-in-the-loop refinement, the system accepts natural language feedback (e.g., "push shadows toward teal").

Intent Parsing: The LLM parses the directive to identify specific parameters to adjust (e.g., Lift) and the magnitude of change (e.g., "slightly" $\rightarrow$ $\Delta \approx 0.01$ ).
Structural Locking: Unmentioned parameters are strictly locked to their previous values, ensuring that refinements are incremental and predictable without regenerating the video.

3. Key Contributions

LumiVideo System: The first agentic system capable of producing professional-quality cinematic base grades from raw log footage autonomously, outputting editable ASC-CDL parameters rather than pixels.
Structured Cinematic Reasoning: A novel architecture combining LLMs, RAG, and Tree of Thoughts to solve the non-linear optimization problem of color grading, ensuring both aesthetic quality and constraint adherence.
Industry-Standard Output: Guarantees temporal consistency and interoperability with professional workflows (DaVinci Resolve) via 3D LUT generation.
LumiGrade Benchmark: The first publicly available benchmark for automated video color grading, comprising over 100 professionally shot log-encoded clips with rich metadata and expert-graded references.

4. Experimental Results

The system was evaluated on the LumiGrade benchmark against general-purpose VLMs (GPT-5, Gemini), industry baselines (Camera Official LUTs), and learning-based methods (Diffusion + CST).

Quantitative Performance: LumiVideo achieved state-of-the-art results across technical (MANIQA, DeQA), aesthetic (Q-Align, CLIP-AS), and grading-specific (LLM-Judge) metrics. It outperformed the Human Expert in most automated metrics and surpassed all other automated baselines significantly.
User Studies:
- General Users: LumiVideo achieved a 38.2% win rate in preference studies, closely trailing the Human Expert (43.6%) and far exceeding generative models (<3.1%).
- Professional Colorists: Rated the system's usability at 4.4/5.0, noting its ability to provide a reliable "starting point" that respects highlight integrity and skin tones, behaving like a "junior colorist."
Ablation Studies: Confirmed that removing the Tree of Thoughts, RAG, or Protected Tones significantly degraded performance, validating the necessity of the multi-stage agentic approach.

5. Significance

LumiVideo represents a paradigm shift in automated visual editing. By moving away from "black-box" pixel generation toward interpretable, parameter-driven reasoning, it bridges the gap between generative AI and professional cinematic pipelines.

Practical Impact: It enables non-experts to achieve professional-grade results while providing human colorists with a powerful, iterative tool that respects industry standards.
Scientific Contribution: It establishes a new methodology for handling continuous, non-linear visual parameter spaces using LLM agents and introduces a critical benchmark (LumiGrade) to drive future research in automated cinematography.

Limitations: The current system applies a global LUT and cannot yet perform spatially varying adjustments (e.g., power windows or secondary qualifiers), which remains a direction for future work.